🎲Intro to Statistics Unit 12 – Linear Regression and Correlation

What's Linear Regression?

• Statistical method used to model and analyze the linear relationship between a dependent variable and one or more independent variables
• Aims to find the best-fitting straight line through the data points by minimizing the sum of the squared residuals (least squares method)
• Equation of the line is in the form $y = mx + b$, where $m$ is the slope and $b$ is the y-intercept
• Helps predict the value of the dependent variable based on the value(s) of the independent variable(s)
• Can be used for both simple linear regression (one independent variable) and multiple linear regression (two or more independent variables)
• Assumes a linear relationship exists between the variables, and that the residuals are normally distributed with constant variance
• Provides a measure of how well the model fits the data using the coefficient of determination ($R^2$)

Key Concepts and Terms

• Dependent variable (response variable): The variable being predicted or explained by the independent variable(s)
• Independent variable (predictor variable): The variable(s) used to predict or explain the dependent variable
• Slope ($m$): The change in the dependent variable for a one-unit change in the independent variable
• Y-intercept ($b$): The value of the dependent variable when the independent variable is zero
• Residuals: The differences between the observed values and the predicted values from the regression line
• Least squares method: A method used to find the best-fitting line by minimizing the sum of the squared residuals
• Coefficient of determination ($R^2$): A measure of how well the regression line fits the data, ranging from 0 to 1
  • $R^2 = 1$ indicates a perfect fit, while $R^2 = 0$ indicates no linear relationship
• P-value: The probability of obtaining the observed results if the null hypothesis (no linear relationship) is true
  • A small p-value (typically < 0.05) suggests that the linear relationship is statistically significant

The Math Behind It

• The least squares method is used to find the best-fitting line by minimizing the sum of the squared residuals
• Residuals are calculated as: $e_i = y_i - \hat{y}_i$, where $y_i$ is the observed value and $\hat{y}_i$ is the predicted value
• The sum of the squared residuals (SSR) is given by: $SSR = \sum_{i=1}^{n} e_i^2 = \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$
• The slope ($m$) and y-intercept ($b$) of the best-fitting line are calculated using the following formulas:
  • $m = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n} (x_i - \bar{x})^2}$
  • $b = \bar{y} - m\bar{x}$
• The coefficient of determination ($R^2$) is calculated as: $R^2 = 1 - \frac{SSR}{SST}$, where $SST$ is the total sum of squares
• The standard error of the estimate ($s_e$) measures the average distance between the observed values and the regression line: $s_e = \sqrt{\frac{SSR}{n-2}}$

Correlation vs. Regression

• Correlation measures the strength and direction of the linear relationship between two variables
  • Pearson's correlation coefficient ($r$) ranges from -1 to 1, with -1 indicating a perfect negative correlation, 1 indicating a perfect positive correlation, and 0 indicating no linear correlation
• Regression goes a step further by providing a model to predict the value of the dependent variable based on the independent variable(s)
• Correlation does not imply causation, while regression can suggest a causal relationship if certain assumptions are met (e.g., no confounding variables, temporal precedence)
• The square of the correlation coefficient ($r^2$) is equal to the coefficient of determination ($R^2$) in simple linear regression
• Both correlation and regression assume a linear relationship between the variables and are sensitive to outliers

How to Do It: Step-by-Step

1. Identify the dependent and independent variables
2. Collect data on both variables for a sample of observations
3. Create a scatterplot of the data to visually assess the linearity of the relationship
4. Calculate the slope ($m$) and y-intercept ($b$) using the least squares method formulas
5. Write the equation of the best-fitting line in the form $y = mx + b$
6. Calculate the coefficient of determination ($R^2$) to assess the goodness of fit
7. Interpret the results, including the slope, y-intercept, and $R^2$
8. Use the regression equation to make predictions for new values of the independent variable(s)
9. Assess the assumptions of linear regression (linearity, normality of residuals, constant variance) and address any violations if necessary

Interpreting Results

• The slope ($m$) represents the change in the dependent variable for a one-unit change in the independent variable
  • A positive slope indicates a positive linear relationship, while a negative slope indicates a negative linear relationship
• The y-intercept ($b$) is the value of the dependent variable when the independent variable is zero
  • In some cases, the y-intercept may not have a meaningful interpretation (e.g., if the independent variable cannot be zero)
• The coefficient of determination ($R^2$) measures the proportion of the variance in the dependent variable that is predictable from the independent variable(s)
  • $R^2$ ranges from 0 to 1, with higher values indicating a better fit of the model to the data
• The p-value for the slope indicates whether the linear relationship is statistically significant
  • A small p-value (typically < 0.05) suggests that the slope is significantly different from zero and that a linear relationship exists
• Confidence intervals for the slope and y-intercept provide a range of plausible values for these parameters based on the sample data

Real-World Applications

• Predicting sales based on advertising expenditure in marketing research
• Estimating the relationship between years of education and income in labor economics
• Modeling the effect of temperature on crop yields in agricultural studies
• Assessing the impact of a drug dosage on patient outcomes in medical research
• Forecasting stock prices based on various economic indicators in finance
• Analyzing the relationship between air pollution levels and respiratory illness rates in environmental studies
• Predicting customer satisfaction based on service quality metrics in business management

Common Pitfalls and Limitations

• Assuming causation based on correlation or regression results without considering other factors (confounding variables, reverse causality)
• Extrapolating beyond the range of the observed data (e.g., predicting values for independent variables outside the sample range)
• Failing to assess and address violations of the assumptions of linear regression (linearity, normality of residuals, constant variance)
• Overfitting the model by including too many independent variables, leading to reduced generalizability
• Ignoring the presence of outliers or influential observations that can significantly affect the regression results
• Misinterpreting the y-intercept when it does not have a meaningful interpretation in the context of the problem
• Relying solely on $R^2$ to assess the model's goodness of fit without considering other factors (e.g., practical significance, subject matter knowledge)