Statistical Prediction Unit 2 ReviewRegression: Linear and Polynomial Models

What's This All About?

• Regression analysis predicts the relationship between a dependent variable and one or more independent variables
• Helps understand how the value of the dependent variable changes when any one of the independent variables is varied, while the other independent variables are held fixed
• Commonly used for forecasting and finding cause and effect relationships between variables
• Regression models can be used for prediction, inference, hypothesis testing, and modeling of causal relationships
• Dependent variable is the main factor you're trying to understand or predict
• Independent variables are the factors you suspect have an impact on your dependent variable
• Regression analysis helps you understand which factors matter most, which factors can be ignored, and how these factors influence each other

Key Concepts and Definitions

• Regression coefficient measures the strength of the relationship between a dependent variable and an independent variable
• Intercept is the expected mean value of the dependent variable when all independent variables are equal to zero
• Residuals are the differences between the observed values of the dependent variable and the predicted values
• R-squared ($R^2$) is a statistical measure of how close the data are to the fitted regression line
  • It's also known as the coefficient of determination
  • $R^2$ values range from 0 to 1, with higher values indicating a better fit
• Adjusted R-squared adjusts the $R^2$ based on the number of independent variables in the model
• P-value is used to determine the statistical significance of the regression coefficients
  • A low p-value (typically ≤ 0.05) indicates that the relationship between the variables is statistically significant

Types of Regression Models

• Simple linear regression involves only one independent variable
  • Relationship between the dependent variable and independent variable is assumed to be linear
  • Equation: $y = \beta_0 + \beta_1x + \epsilon$, where $y$ is the dependent variable, $x$ is the independent variable, $\beta_0$ is the intercept, $\beta_1$ is the slope, and $\epsilon$ is the error term
• Multiple linear regression involves two or more independent variables
  • Equation: $y = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_nx_n + \epsilon$, where $y$ is the dependent variable, $x_1, x_2, ..., x_n$ are the independent variables, $\beta_0$ is the intercept, $\beta_1, \beta_2, ..., \beta_n$ are the slopes, and $\epsilon$ is the error term
• Polynomial regression models the relationship between the dependent variable and independent variables as an nth degree polynomial
  • Useful when the relationship between the variables is curvilinear
• Stepwise regression is a method of fitting regression models in which the choice of predictive variables is carried out by an automatic procedure (forward selection, backward elimination, or bidirectional elimination)
• Ridge regression is a technique for analyzing multiple regression data that suffer from multicollinearity, which occurs when independent variables in a regression model are correlated

Building Linear Regression Models

• Collect data on the dependent variable and independent variable(s)
• Create a scatter plot to visually inspect the relationship between the variables
• Use the least squares method to estimate the regression coefficients
  • Least squares method minimizes the sum of the squared residuals
• Assess the model's goodness of fit using measures like $R^2$ and adjusted $R^2$
• Test the statistical significance of the regression coefficients using t-tests or F-tests
• Validate the assumptions of linear regression (linearity, homoscedasticity, independence, and normality)
  • Linearity assumes a linear relationship between the dependent variable and independent variable(s)
  • Homoscedasticity assumes constant variance of the residuals across all levels of the independent variable(s)
  • Independence assumes that the residuals are not correlated with each other
  • Normality assumes that the residuals are normally distributed

Polynomial Regression: When Lines Aren't Enough

• Polynomial regression is used when the relationship between the dependent variable and independent variable is not linear
• Polynomial regression equation: $y = \beta_0 + \beta_1x + \beta_2x^2 + ... + \beta_nx^n + \epsilon$, where $y$ is the dependent variable, $x$ is the independent variable, $\beta_0$ is the intercept, $\beta_1, \beta_2, ..., \beta_n$ are the coefficients, and $\epsilon$ is the error term
• The degree of the polynomial (n) determines the number of bends in the fitted line
  • A second-degree polynomial (quadratic) has one bend
  • A third-degree polynomial (cubic) has two bends
• Higher degree polynomials can fit more complex relationships but are also more prone to overfitting
• Polynomial features can be created from the original independent variable(s) using techniques like feature engineering

Model Evaluation and Selection

• Split the data into training and testing sets to evaluate the model's performance on unseen data
• Use cross-validation techniques (k-fold, leave-one-out) to assess the model's performance and stability
• Compare the performance of different models using metrics like mean squared error (MSE), root mean squared error (RMSE), and mean absolute error (MAE)
  • MSE is the average of the squared differences between the predicted and actual values
  • RMSE is the square root of the MSE and provides an interpretable measure of the model's accuracy
  • MAE is the average of the absolute differences between the predicted and actual values
• Use the Akaike Information Criterion (AIC) or Bayesian Information Criterion (BIC) to compare models with different numbers of parameters
  • AIC and BIC balance the model's goodness of fit with its complexity, favoring simpler models
• Select the model that performs best on the testing set and has the lowest complexity

Real-World Applications

• Predicting house prices based on features like square footage, number of bedrooms, and location
• Forecasting sales based on advertising expenditure, promotions, and economic indicators
• Analyzing the relationship between a patient's characteristics (age, weight, blood pressure) and the likelihood of developing a disease
• Predicting stock prices based on historical data, market trends, and company performance
• Estimating the impact of various factors (education, experience, industry) on an individual's salary

Common Pitfalls and How to Avoid Them

• Overfitting occurs when a model is too complex and fits the noise in the data rather than the underlying pattern
  • Regularization techniques (L1 and L2) can help prevent overfitting by adding a penalty term to the loss function
  • Use cross-validation to detect overfitting and select the appropriate model complexity
• Multicollinearity occurs when independent variables are highly correlated with each other
  • Multicollinearity can lead to unstable and unreliable estimates of the regression coefficients
  • Use correlation matrices and variance inflation factors (VIF) to detect multicollinearity
  • Consider removing one of the correlated variables or using dimensionality reduction techniques (PCA)
• Outliers can have a significant impact on the regression results
  • Identify outliers using scatter plots, residual plots, and Cook's distance
  • Consider removing outliers if they are due to measurement errors or data entry mistakes
  • Use robust regression techniques (M-estimators, RANSAC) that are less sensitive to outliers
• Non-normality of residuals can invalidate the assumptions of hypothesis tests and confidence intervals
  • Use residual plots and normality tests (Shapiro-Wilk, Kolmogorov-Smirnov) to check for non-normality
  • Consider transforming the dependent variable or using generalized linear models (GLMs) if the residuals are not normally distributed