3.1 Hypothesis Testing for Regression Coefficients

Hypothesis Testing for Regression Coefficients

Hypothesis testing for regression coefficients lets you determine whether a predictor variable has a statistically significant linear relationship with the response variable, or whether the observed association could plausibly be due to chance. This is the foundation of inference in simple linear regression: without it, you have estimates but no way to judge whether those estimates reflect real patterns in the population.

Formulating Null and Alternative Hypotheses

The null hypothesis claims that the true population coefficient is zero, meaning the predictor has no linear effect on the response:

$H_0: \beta_i = 0$

where $\beta_i$ is the population coefficient for the $i^{th}$ predictor. If this is true, knowing the predictor's value tells you nothing (linearly) about the response.

The alternative hypothesis claims the coefficient is not zero, meaning a linear relationship does exist:

$H_a: \beta_i \neq 0$

This is a two-sided test. Sometimes you have a directional prediction, which calls for a one-sided alternative:

• $H_a: \beta_i > 0$ if you expect a positive relationship
• $H_a: \beta_i < 0$ if you expect a negative relationship

For example, if a researcher believes that increased advertising expenditure leads to higher sales, the appropriate alternative would be $H_a: \beta_{\text{advertising}} > 0$. One-sided tests are more powerful in the hypothesized direction but can't detect effects in the opposite direction, so only use them when you have a strong prior reason.

Conducting Hypothesis Tests Using t-Tests

The t-test for a regression coefficient compares the estimated coefficient to the variability you'd expect from sampling alone. Here's the process:

1. Compute the test statistic. The t-statistic measures how many standard errors the estimate is from zero:

$t = \frac{\hat{\beta}_i - 0}{SE(\hat{\beta}_i)} = \frac{\hat{\beta}_i}{SE(\hat{\beta}_i)}$

where $\hat{\beta}_i$ is the estimated coefficient and $SE(\hat{\beta}_i)$ is its standard error.

1. Understand the standard error. The standard error of $\hat{\beta}_i$ captures how much the estimate would vary across repeated samples. It depends on the variance of the residuals and the spread of the predictor values. Larger residual variance inflates the standard error; greater spread in the predictor shrinks it.

2. Determine the degrees of freedom. For the t-distribution, degrees of freedom equal $n - p - 1$, where $n$ is the number of observations and $p$ is the number of predictors. In simple linear regression, $p = 1$, so degrees of freedom are $n - 2$.

3. Compare to the critical value or compute a p-value. If the absolute value of the t-statistic exceeds the critical value for your chosen $\alpha$ and degrees of freedom, you reject $H_0$. For instance, with $\alpha = 0.05$, $n = 50$, and $p = 1$, the degrees of freedom are $50 - 1 - 1 = 48$, and the two-tailed critical value is approximately $\pm 2.011$.

Interpreting Hypothesis Test Results

Rejecting or Failing to Reject the Null Hypothesis

If you reject $H_0$: There is sufficient evidence that $\beta_i \neq 0$. The predictor has a statistically significant linear relationship with the response. Changes in the predictor are associated with changes in the response beyond what you'd expect from random variation alone.

If you fail to reject $H_0$: The data don't provide strong enough evidence to conclude that $\beta_i \neq 0$. This does not mean the predictor is unrelated to the response. It means the signal wasn't strong enough relative to the noise in your data. A small sample size, high variability, or a genuinely weak effect could all produce this result.

Understanding the Coefficient's Sign and Magnitude

The sign of $\hat{\beta}_i$ tells you the direction of the linear relationship:

• Positive coefficient: as the predictor increases, the response tends to increase (direct relationship)
• Negative coefficient: as the predictor increases, the response tends to decrease (inverse relationship)

The magnitude tells you the size of the effect. Specifically, $\hat{\beta}_i$ represents the expected change in the response for a one-unit increase in the predictor, holding all else constant. If the coefficient for "age" is 0.5, then each additional year of age is associated with a 0.5-unit increase in the response.

Keep in mind that a coefficient can be large in magnitude but not statistically significant (if the standard error is also large), or small in magnitude but highly significant (if estimated very precisely). Statistical significance and practical importance are separate questions.

Significance of Regression Coefficients

Using P-Values to Determine Significance

The p-value is the probability of observing a t-statistic as extreme as (or more extreme than) the one you calculated, assuming the null hypothesis is true. It quantifies how surprising your data are under $H_0$.

• A small p-value (less than $\alpha$) means the observed result would be unlikely if $\beta_i$ were truly zero. You reject $H_0$ and conclude the coefficient is statistically significant. For example, if $\alpha = 0.05$ and the p-value is 0.02, you reject $H_0$.
• A large p-value (greater than $\alpha$) means the observed result is reasonably consistent with $\beta_i = 0$. You fail to reject $H_0$. For example, if $\alpha = 0.05$ and the p-value is 0.15, you do not reject $H_0$.

The p-value does not tell you the probability that $H_0$ is true. It tells you how likely the observed data (or something more extreme) would be if $H_0$ were true. That distinction matters.

Choosing an Appropriate Significance Level

The significance level $\alpha$ is the threshold you set before testing. It represents the maximum probability of a Type I error, which is rejecting $H_0$ when it's actually true (a false positive).

Common choices:

• $\alpha = 0.01$: Stringent. Requires very strong evidence to reject $H_0$. Used when false positives carry serious consequences, such as in medical research or regulatory decisions.
• $\alpha = 0.05$: The most common default across many fields. Balances the risk of false positives and false negatives.
• $\alpha = 0.10$: More lenient. Useful in exploratory analyses or when missing a real effect (Type II error) is more costly than a false alarm.

There's a tradeoff: lowering $\alpha$ reduces your Type I error rate but increases your Type II error rate (failing to detect a real effect). The right choice depends on the stakes involved and the goals of your analysis.