🥖Linear Modeling Theory Unit 2 – Least Squares Estimation & Model Fit

Key Concepts

• Least squares estimation minimizes the sum of squared residuals to find the best-fitting model parameters
• Model fit assesses how well a linear model explains the variability in the response variable
• Residuals represent the differences between observed and predicted values of the response variable
• Coefficient of determination (R-squared) measures the proportion of variance in the response variable explained by the model
  • Ranges from 0 to 1, with higher values indicating better model fit
• Adjusted R-squared accounts for the number of predictors in the model and penalizes overfitting
• F-statistic tests the overall significance of the model by comparing the explained variance to the unexplained variance
• t-statistics and p-values assess the significance of individual model coefficients

Theoretical Foundation

• Linear modeling assumes a linear relationship between the response variable and one or more predictor variables
• The goal is to find the best-fitting line or hyperplane that minimizes the sum of squared residuals
• Ordinary least squares (OLS) estimation is the most common method for estimating model parameters
• Gauss-Markov theorem states that OLS estimators are the best linear unbiased estimators (BLUE) under certain assumptions
  • Assumptions include linearity, independence, homoscedasticity, and normality of errors
• Maximum likelihood estimation (MLE) is an alternative method that estimates parameters by maximizing the likelihood function
• Bayesian estimation incorporates prior knowledge about the parameters and updates them based on observed data
• Regularization techniques (ridge regression, lasso) can be used to address multicollinearity and improve model stability

Mathematical Framework

• Linear model: $y = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_px_p + \epsilon$
  • $y$ is the response variable, $x_1, x_2, ..., x_p$ are predictor variables, $\beta_0, \beta_1, ..., \beta_p$ are model coefficients, and $\epsilon$ is the error term
• Residuals: $e_i = y_i - \hat{y}_i$, where $y_i$ is the observed value and $\hat{y}_i$ is the predicted value for the $i$-th observation
• Sum of squared residuals: $SSR = \sum_{i=1}^{n} e_i^2$
• Coefficient of determination: $R^2 = 1 - \frac{SSR}{SST}$, where $SST$ is the total sum of squares
• Adjusted R-squared: $R^2_{adj} = 1 - \frac{SSR/(n-p-1)}{SST/(n-1)}$, where $n$ is the number of observations and $p$ is the number of predictors
• F-statistic: $F = \frac{MSR}{MSE}$, where $MSR$ is the mean squared regression and $MSE$ is the mean squared error
• t-statistic for coefficient $\beta_j$: $t_j = \frac{\hat{\beta}_j}{SE(\hat{\beta}_j)}$, where $SE(\hat{\beta}_j)$ is the standard error of the estimated coefficient

Least Squares Method

• Least squares estimation finds the model coefficients that minimize the sum of squared residuals
• The objective function is $\min_{\beta} SSR = \min_{\beta} \sum_{i=1}^{n} (y_i - \beta_0 - \beta_1x_{i1} - ... - \beta_px_{ip})^2$
• The normal equations are derived by setting the partial derivatives of the objective function with respect to each coefficient equal to zero
  • $\frac{\partial SSR}{\partial \beta_j} = -2 \sum_{i=1}^{n} (y_i - \beta_0 - \beta_1x_{i1} - ... - \beta_px_{ip})x_{ij} = 0$ for $j = 0, 1, ..., p$
• The normal equations can be expressed in matrix form as $X^TX\beta = X^Ty$, where $X$ is the design matrix and $y$ is the response vector
• The least squares estimator is given by $\hat{\beta} = (X^TX)^{-1}X^Ty$, assuming $X^TX$ is invertible
• The fitted values are calculated as $\hat{y} = X\hat{\beta}$, and the residuals are $e = y - \hat{y}$

Model Assumptions

• Linearity assumes a linear relationship between the response variable and predictor variables
  • Violations can be detected by plotting residuals against fitted values or predictor variables
• Independence assumes that the errors are uncorrelated and observations are independent
  • Violations can be detected using the Durbin-Watson test or by examining residual plots for patterns
• Homoscedasticity assumes that the variance of the errors is constant across all levels of the predictors
  • Violations (heteroscedasticity) can be detected by plotting residuals against fitted values or predictor variables
• Normality assumes that the errors follow a normal distribution with mean zero
  • Violations can be detected using normal probability plots (Q-Q plots) or formal tests like the Shapiro-Wilk test
• No multicollinearity assumes that the predictor variables are not highly correlated with each other
  • Violations can be detected using correlation matrices, variance inflation factors (VIF), or condition indices
• Outliers and influential observations can have a significant impact on the least squares estimates
  • Outliers can be identified using residual plots or standardized residuals
  • Influential observations can be identified using leverage values, Cook's distance, or DFFITS

Estimating Parameters

• The least squares estimator $\hat{\beta} = (X^TX)^{-1}X^Ty$ provides point estimates for the model coefficients
• The standard errors of the estimated coefficients are given by the square roots of the diagonal elements of the covariance matrix $\hat{\sigma}^2(X^TX)^{-1}$
  • $\hat{\sigma}^2 = \frac{SSR}{n-p-1}$ is an unbiased estimator of the error variance
• Confidence intervals for the coefficients can be constructed using the t-distribution with $n-p-1$ degrees of freedom
  • A $(1-\alpha)100\%$ confidence interval for $\beta_j$ is $\hat{\beta}_j \pm t_{\alpha/2, n-p-1}SE(\hat{\beta}_j)$
• Hypothesis tests for individual coefficients can be performed using the t-statistic and comparing it to the critical value from the t-distribution
  • The null hypothesis is $H_0: \beta_j = 0$, and the alternative hypothesis is $H_1: \beta_j \neq 0$
• The F-test for overall model significance compares the explained variance to the unexplained variance
  • The null hypothesis is $H_0: \beta_1 = \beta_2 = ... = \beta_p = 0$, and the alternative hypothesis is that at least one coefficient is non-zero

Assessing Model Fit

• The coefficient of determination (R-squared) measures the proportion of variance in the response variable explained by the model
  • Higher values indicate better model fit, but R-squared can be misleading when comparing models with different numbers of predictors
• Adjusted R-squared penalizes the addition of unnecessary predictors and is more suitable for model comparison
  • The model with the highest adjusted R-squared is preferred when comparing models with different numbers of predictors
• The F-statistic tests the overall significance of the model by comparing the explained variance to the unexplained variance
  • A significant F-statistic (p-value < $\alpha$) indicates that the model explains a significant portion of the variability in the response variable
• Residual plots (residuals vs. fitted values, residuals vs. predictor variables) can reveal violations of model assumptions
  • Patterns in the residual plots suggest that the model assumptions may not be met
• Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) are information-theoretic measures for model selection
  • Lower values of AIC and BIC indicate better model fit, penalizing model complexity
• Cross-validation techniques (k-fold, leave-one-out) assess the model's predictive performance on unseen data
  • The model with the lowest cross-validation error is preferred

Practical Applications

• Linear regression is widely used in various fields, including economics, finance, social sciences, and engineering
• Predicting housing prices based on features like square footage, number of bedrooms, and location
  • The model coefficients represent the marginal effect of each feature on the housing price
• Analyzing the relationship between advertising expenditure and sales revenue for a company
  • The model can help determine the effectiveness of advertising campaigns and optimize budget allocation
• Investigating the factors influencing student performance in standardized tests
  • The model can identify the most important predictors of student success and inform educational policies
• Estimating the impact of socioeconomic factors on life expectancy across different countries
  • The model can provide insights into the determinants of health outcomes and guide public health interventions
• Forecasting energy consumption based on historical data and weather variables
  • The model can help energy companies plan their production and distribution more efficiently
• Developing credit scoring models to assess the creditworthiness of loan applicants
  • The model can help financial institutions make informed lending decisions and manage credit risk
• Analyzing the relationship between customer demographics and purchasing behavior for targeted marketing campaigns
  • The model can help businesses identify the most promising customer segments and tailor their marketing strategies accordingly