11.3 Least Squares Method and Regression Equation

Simple linear regression is a powerful tool for understanding relationships between variables. It helps us predict one variable based on another, like guessing someone's blood pressure from their age. The method uses a straight line to represent this relationship.

The least squares method finds the best-fitting line by minimizing differences between observed and predicted values. We can use this line to make predictions, but it's important to remember that these predictions aren't perfect and come with some uncertainty.

Simple Linear Regression

Concept of simple linear regression

• Statistical method models relationship between two variables
  • Independent (explanatory) variable: $x$ (age, income)
  • Dependent (response) variable: $y$ (blood pressure, spending)
• Determines strength and direction of linear relationship between variables
• Predicts value of dependent variable based on independent variable
• Relationship represented by straight line called regression line
  • Equation: $y = \beta_0 + \beta_1x + \epsilon$
    • $\beta_0$: y-intercept, value of $y$ when $x = 0$
    • $\beta_1$: slope, change in $y$ for one-unit change in $x$
    • $\epsilon$: random error term, deviation of observed values from regression line

Least squares method application

• Estimates parameters $\beta_0$ and $\beta_1$ of regression line
  • Minimizes sum of squared differences between observed and predicted $y$ values
• Estimate slope $\beta_1$ using formula: $\hat{\beta_1} = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n} (x_i - \bar{x})^2}$
  • $\hat{\beta_1}$: estimated slope
  • $x_i$, $y_i$: observed values of independent and dependent variables
  • $\bar{x}$, $\bar{y}$: sample means of independent and dependent variables
  • $n$: number of observations
• Estimate y-intercept $\beta_0$ using formula: $\hat{\beta_0} = \bar{y} - \hat{\beta_1}\bar{x}$
  • $\hat{\beta_0}$: estimated y-intercept

Derivation of regression equation

• Substitute estimated values of $\beta_0$ and $\beta_1$ into general equation: $y = \beta_0 + \beta_1x + \epsilon$
• Resulting regression equation: $\hat{y} = \hat{\beta_0} + \hat{\beta_1}x$
  • $\hat{y}$: predicted value of dependent variable for given value of independent variable $x$

Interpretation of regression components

• Slope $\hat{\beta_1}$ represents change in dependent variable $y$ for one-unit change in independent variable $x$
  • Positive slope indicates positive linear relationship (weight and height)
  • Negative slope indicates negative linear relationship (age and reaction time)
• Y-intercept $\hat{\beta_0}$ represents value of dependent variable $y$ when independent variable $x$ equals zero
  • May not have meaningful interpretation if range of $x$ values does not include zero (negative age)

Predictions using regression equation

• Substitute given value of independent variable $x$ into equation: $\hat{y} = \hat{\beta_0} + \hat{\beta_1}x$
• Resulting value $\hat{y}$ is predicted value of dependent variable for given value of independent variable
• Predictions subject to uncertainty
  • Accuracy depends on strength of linear relationship and quality of model fit (correlation coefficient, residual analysis)