🥖Linear Modeling Theory Unit 5 – Matrix Approach to Linear Regression

Key Concepts and Definitions

• Linear regression models the relationship between a dependent variable and one or more independent variables using a linear equation
• Matrix notation represents the linear regression model in a compact and efficient way using matrices and vectors
• Least squares estimation method estimates the regression coefficients by minimizing the sum of squared residuals
• Gauss-Markov theorem states that under certain assumptions, the least squares estimators are the best linear unbiased estimators (BLUE)
• Hypothesis testing assesses the significance of the regression coefficients and the overall model fit using t-tests and F-tests
• Residual analysis examines the differences between the observed and predicted values to assess model assumptions and fit
• Multicollinearity occurs when independent variables are highly correlated, which can affect the interpretation of regression coefficients
• Heteroscedasticity refers to the situation where the variance of the residuals is not constant across the range of the independent variables

Matrix Notation and Basics

• Matrix notation represents the linear regression model as $y = Xβ + ε$, where $y$ is the vector of observations, $X$ is the design matrix, $β$ is the vector of regression coefficients, and $ε$ is the vector of residuals
• The design matrix $X$ contains the values of the independent variables, with each row representing an observation and each column representing a variable
• The vector of regression coefficients $β$ contains the intercept and the slopes associated with each independent variable
• The vector of residuals $ε$ represents the differences between the observed and predicted values of the dependent variable
• Matrix multiplication is used to compute the predicted values of the dependent variable: $ŷ = Xβ$
• The transpose of a matrix $X$, denoted as $X'$, is obtained by interchanging its rows and columns
• The inverse of a square matrix $X$, denoted as $X^{-1}$, satisfies the property $XX^{-1} = I$, where $I$ is the identity matrix

Linear Regression Model Setup

• The linear regression model assumes a linear relationship between the dependent variable and the independent variables
• The model can be written as $y_i = β_0 + β_1x_{i1} + β_2x_{i2} + ... + β_px_{ip} + ε_i$, where $y_i$ is the $i$-th observation of the dependent variable, $x_{ij}$ is the $i$-th observation of the $j$-th independent variable, and $ε_i$ is the $i$-th residual
• In matrix notation, the model is represented as $y = Xβ + ε$, where $y$ is an $n × 1$ vector, $X$ is an $n × (p+1)$ matrix, $β$ is a $(p+1) × 1$ vector, and $ε$ is an $n × 1$ vector
  • The first column of the design matrix $X$ is typically a column of ones, representing the intercept term
  • The remaining columns of $X$ contain the values of the independent variables
• The assumptions of the linear regression model include linearity, independence, homoscedasticity, and normality of residuals
• Violations of these assumptions can lead to biased or inefficient estimates of the regression coefficients and affect the validity of hypothesis tests and confidence intervals

Least Squares Estimation

• The least squares estimation method estimates the regression coefficients by minimizing the sum of squared residuals (SSR)
• The SSR is given by $SSR = ∑_{i=1}^n (y_i - ŷ_i)^2 = (y - Xβ)'(y - Xβ)$, where $y_i$ is the $i$-th observed value, $ŷ_i$ is the $i$-th predicted value, and $n$ is the sample size
• The least squares estimator of $β$, denoted as $β̂$, is obtained by solving the normal equations: $X'Xβ̂ = X'y$
• The solution to the normal equations is given by $β̂ = (X'X)^{-1}X'y$, provided that $(X'X)$ is invertible
  • The matrix $(X'X)$ is called the Gram matrix or the cross-product matrix
  • The invertibility of $(X'X)$ requires that the columns of $X$ are linearly independent (no perfect multicollinearity)
• The fitted values (predicted values) of the dependent variable are computed as $ŷ = Xβ̂$
• The residuals are computed as $e = y - ŷ = y - Xβ̂$

Properties of Matrix Estimators

• Under the assumptions of the linear regression model, the least squares estimator $β̂$ has several desirable properties
• $β̂$ is an unbiased estimator of $β$, meaning that $E(β̂) = β$
• The variance-covariance matrix of $β̂$ is given by $Var(β̂) = σ^2(X'X)^{-1}$, where $σ^2$ is the variance of the residuals
  • The diagonal elements of $Var(β̂)$ are the variances of the individual regression coefficients
  • The off-diagonal elements are the covariances between the regression coefficients
• The Gauss-Markov theorem states that among all linear unbiased estimators, the least squares estimator $β̂$ has the smallest variance (BLUE)
• The estimated variance of the residuals, denoted as $s^2$, is an unbiased estimator of $σ^2$ and is given by $s^2 = SSR / (n - p - 1)$, where $SSR$ is the sum of squared residuals, $n$ is the sample size, and $p$ is the number of independent variables
• The standard errors of the regression coefficients are the square roots of the diagonal elements of $s^2(X'X)^{-1}$
• The fitted values $ŷ$ and the residuals $e$ are orthogonal, meaning that $ŷ'e = 0$

Hypothesis Testing and Inference

• Hypothesis testing is used to assess the significance of the regression coefficients and the overall model fit
• The null hypothesis for a single regression coefficient $β_j$ is $H_0: β_j = 0$, which implies that the $j$-th independent variable has no effect on the dependent variable
• The alternative hypothesis can be two-sided ($H_1: β_j ≠ 0$) or one-sided ($H_1: β_j > 0$ or $H_1: β_j < 0$)
• The test statistic for a single regression coefficient is the t-statistic, given by $t_j = (β̂_j - 0) / SE(β̂_j)$, where $β̂_j$ is the estimated coefficient and $SE(β̂_j)$ is its standard error
• The t-statistic follows a t-distribution with $(n - p - 1)$ degrees of freedom under the null hypothesis
• The p-value is the probability of observing a t-statistic as extreme as or more extreme than the observed value, assuming the null hypothesis is true
• Confidence intervals for the regression coefficients can be constructed using the t-distribution and the standard errors: $β̂_j ± t_{α/2, n-p-1} × SE(β̂_j)$, where $α$ is the significance level
• The F-test is used to assess the overall significance of the regression model, testing the null hypothesis that all regression coefficients (except the intercept) are simultaneously zero
• The F-statistic is given by $F = (SSR_R - SSR_F) / (p_F - p_R) / (SSR_F / (n - p_F - 1))$, where $SSR_R$ and $SSR_F$ are the sum of squared residuals for the reduced and full models, and $p_R$ and $p_F$ are the number of parameters in the reduced and full models

Model Diagnostics and Assumptions

• Model diagnostics are used to assess the validity of the linear regression assumptions and the adequacy of the model fit
• Residual plots (residuals vs. fitted values, residuals vs. independent variables) can reveal patterns that indicate violations of linearity, homoscedasticity, or independence assumptions
• Normal probability plots (Q-Q plots) of the residuals can assess the normality assumption
• The Durbin-Watson statistic tests for the presence of autocorrelation in the residuals
• The variance inflation factor (VIF) measures the degree of multicollinearity among the independent variables
  • VIF values greater than 5 or 10 indicate potential multicollinearity issues
• Influential observations (outliers or high-leverage points) can be identified using measures such as Cook's distance, leverage values, or studentized residuals
• Partial residual plots (component-plus-residual plots) can help assess the linearity assumption for individual independent variables
• The coefficient of determination ($R^2$) measures the proportion of variance in the dependent variable explained by the independent variables
  • Adjusted $R^2$ accounts for the number of independent variables in the model and is more suitable for comparing models with different numbers of variables
• The standard error of the regression ($SER$) measures the average distance between the observed values and the predicted values

Applications and Examples

• Linear regression is widely used in various fields, such as economics, finance, social sciences, and engineering, to model and analyze relationships between variables
• Example: A real estate company wants to predict housing prices based on factors such as square footage, number of bedrooms, and location
  • The dependent variable is the housing price, and the independent variables are square footage, number of bedrooms, and dummy variables for location
  • The linear regression model can estimate the effect of each factor on housing prices and predict prices for new properties
• Example: A marketing firm wants to analyze the impact of advertising expenditure on sales
  • The dependent variable is sales, and the independent variable is advertising expenditure
  • The linear regression model can estimate the marginal effect of advertising on sales and help determine the optimal advertising budget
• Example: A public health researcher wants to investigate the relationship between body mass index (BMI) and various health indicators, such as blood pressure and cholesterol levels
  • The dependent variables are blood pressure and cholesterol levels, and the independent variable is BMI
  • Separate linear regression models can be fitted for each health indicator to assess the impact of BMI on these outcomes
• Example: An environmental scientist wants to study the effect of temperature and precipitation on crop yields
  • The dependent variable is crop yield, and the independent variables are temperature and precipitation
  • The linear regression model can estimate the sensitivity of crop yields to changes in temperature and precipitation, which can inform agricultural practices and policy decisions