Essential Linear Regression Coefficients

Why This Matters

Linear regression coefficients are the building blocks of every model you'll construct and interpret in this course. You're not just being tested on definitions: exams expect you to understand how coefficients work together to describe relationships, quantify uncertainty, and evaluate model quality. These concepts connect directly to hypothesis testing, model comparison, diagnostics, and prediction, making them foundational for everything from simple bivariate analysis to complex multiple regression.

When you encounter a regression output, read it like a story: the coefficients tell you what's happening, the standard errors and confidence intervals tell you how certain you can be, and the fit statistics tell you how well the model captures reality. Don't just memorize formulas. Know what each coefficient reveals about the underlying data and when to use each metric to answer different analytical questions.

---

Model Parameters: The Core Relationship

These coefficients define the actual regression line and tell you what the model predicts. They're the heart of your equation: $\hat{y} = \beta_0 + \beta_1 x$.

Intercept ($\beta_0$)

• Baseline value: the expected value of $y$ when all independent variables equal zero
• Anchors the regression line by setting its vertical position on the coordinate plane
• Interpretation caveat: only meaningful if $x = 0$ falls within the realistic range of your data. If your predictor is something like "years of experience" and nobody in your sample has zero years, the intercept is just a mathematical anchor, not a substantive finding.

Slope ($\beta_1$)

• Rate of change: quantifies how much $y$ changes for each one-unit increase in $x$, holding all other predictors constant in multiple regression
• Sign indicates direction: positive slopes show direct relationships; negative slopes show inverse relationships
• Magnitude matters for comparing effect sizes, but only across standardized variables. Comparing raw slopes across predictors measured on different scales is misleading.

---

Uncertainty Quantification: How Precise Are Your Estimates?

These metrics tell you how much your coefficient estimates might vary from sample to sample. They're essential for distinguishing real effects from statistical noise.

Standard Error of Coefficients

• Precision measure: quantifies the variability of coefficient estimates across repeated sampling
• Smaller is better: low standard errors indicate your estimates are stable and reliable
• Foundation for inference: used to construct confidence intervals and calculate t-statistics

The standard error depends on several things: sample size, the variance of the residuals, and the spread of your predictor variable. More data and less noise both shrink it.

Confidence Intervals

A confidence interval gives you a range of plausible values for the true population parameter. A 95% CI means that if you repeated the sampling process many times, about 95% of the intervals constructed this way would contain the true parameter.

• Width indicates precision: narrow intervals suggest reliable estimates; wide intervals signal uncertainty
• Zero-exclusion test: if a 95% CI for $\beta_1$ doesn't contain zero, the coefficient is statistically significant at $\alpha = 0.05$
• Formula: $\hat{\beta} \pm t^* \times \text{SE}(\hat{\beta})$, where $t^*$ is the critical value for your chosen confidence level and degrees of freedom

---

Hypothesis Testing: Is the Effect Real?

These statistics help you determine whether your coefficients reflect genuine relationships or could have occurred by chance. The logic follows: estimate → standardize → evaluate probability.

t-Statistic

• Standardized coefficient: calculated as $t = \frac{\hat{\beta}}{\text{SE}(\hat{\beta})}$, measuring how many standard errors the coefficient is from zero
• Larger absolute values indicate stronger evidence against the null hypothesis ($H_0: \beta = 0$)
• Degrees of freedom matter: in simple linear regression, $df = n - 2$. Critical values depend on this, especially in small samples where the t-distribution has heavier tails than the normal distribution.

p-Value

• Probability of extremity: the probability of observing a test statistic as extreme as (or more extreme than) yours, assuming the null hypothesis is true
• Decision threshold: typically reject $H_0$ when $p < 0.05$, indicating statistical significance
• Not effect size: a tiny p-value doesn't mean a large or important effect. With a huge sample, even trivially small effects become "significant." Always pair p-values with effect size measures.

---

Model Fit: How Well Does the Model Work?

These statistics evaluate whether your model captures meaningful variation in the data. They answer: "Is this model actually useful?"

R-Squared ($R^2$)

• Proportion of variance explained: ranges from 0 to 1, with higher values indicating better explanatory power
• Interpretation: an $R^2 = 0.75$ means 75% of the variation in $y$ is accounted for by the model
• Limitation: $R^2$ always increases (or stays the same) when you add predictors, even useless ones. This makes it unreliable for comparing models of different sizes.

In simple linear regression, $R^2$ equals the square of the Pearson correlation coefficient $r$. That connection breaks down in multiple regression, where $R^2$ reflects the joint explanatory power of all predictors together.

Adjusted R-Squared

• Penalized fit measure: adjusts $R^2$ downward based on the number of predictors relative to sample size
• Model comparison tool: use this instead of $R^2$ when comparing models with different numbers of variables
• Can decrease if a new predictor doesn't improve fit enough to justify its inclusion, which is exactly the behavior you want for model selection

F-Statistic

• Overall model test: evaluates whether the regression model explains significantly more variance than a model with no predictors (the intercept-only model)
• Calculated as the ratio of mean explained variance to mean unexplained variance: $F = \frac{\text{MSR}}{\text{MSE}}$, where MSR is the mean square due to regression and MSE is the mean square error
• Complements individual t-tests: the F-test evaluates the model as a whole, while t-statistics test each coefficient separately. In simple linear regression with one predictor, $F = t^2$.

---

Diagnostics: Is Something Wrong?

Diagnostic statistics help you identify problems that could invalidate your model's assumptions or distort your results.

Variance Inflation Factor (VIF)

Multicollinearity occurs when predictor variables are highly correlated with each other. The VIF measures how much the variance of a coefficient estimate is inflated because of this correlation.

• Rule of thumb: VIF > 10 signals problematic multicollinearity; some use VIF > 5 as a warning threshold. A VIF of 1 means no collinearity at all for that predictor.
• Consequences of ignoring: inflated standard errors, unstable coefficients that swing wildly between samples, and unreliable hypothesis tests
• Calculated for each predictor by regressing that predictor on all the others: $\text{VIF}_j = \frac{1}{1 - R^2_j}$, where $R^2_j$ is the R-squared from regressing predictor $j$ on the remaining predictors

---

Quick Reference Table

---

Self-Check Questions

1. If a 95% confidence interval for $\beta_1$ is $[0.23, 0.89]$, what can you conclude about the coefficient's statistical significance at $\alpha = 0.05$? Why?

2. Compare and contrast $R^2$ and adjusted $R^2$: when would these two statistics lead you to different conclusions about model quality?

3. A regression output shows a slope of 2.5 with a standard error of 0.5. Calculate the t-statistic and explain what it tells you about the relationship.

4. Which two statistics would you examine first if you suspected multicollinearity was inflating your standard errors? What values would concern you?

5. You're asked to "interpret the regression equation $\hat{y} = 12.4 + 3.2x$ in context." What specific information must you include for both the intercept and slope to earn full credit?