Hypothesis Testing Methods

Why This Matters

Hypothesis testing is how you move from "I think there's a pattern here" to "I can confidently claim this effect is real." Every time a study claims a new drug works, a policy made a difference, or two groups behave differently, hypothesis testing is doing the heavy lifting behind the scenes. You're being tested on your ability to choose the right test for the right situation, interpret results correctly, and understand what "statistically significant" actually means.

The methods in this guide aren't just formulas to memorize. They represent different tools for different jobs. Some compare means, others compare variances or distributions. Some require your data to be normally distributed; others don't care. The key concepts to master include parametric vs. non-parametric approaches, comparing means vs. variances vs. distributions, and when assumptions matter. Don't just memorize which test does what. Know why you'd reach for one tool instead of another.

---

Comparing Means: The Workhorses of Hypothesis Testing

Most hypothesis tests you'll encounter ask a simple question: are these means different? The tests below handle this question under different conditions: known vs. unknown variance, one group vs. two, independent vs. paired observations.

Z-Test

• Use when population variance is known. This is rare in practice but common on exams as a foundational concept.
• Requires normality or large samples ($n > 30$) thanks to the Central Limit Theorem.
• Test statistic: $Z = \frac{\bar{x} - \mu}{\sigma / \sqrt{n}}$, where $\sigma$ is the known population standard deviation.

The numerator measures how far your sample mean is from the hypothesized population mean. The denominator is the standard error, which captures how much sampling variability you'd expect. A large $Z$ value means your sample mean is far from $\mu$ relative to what random chance would produce.

T-Test (One-Sample, Two-Sample, Paired)

• One-sample T-test compares a sample mean to a hypothesized population mean when variance is unknown. It uses the sample standard deviation $s$ instead of $\sigma$, which adds uncertainty and produces heavier tails in the distribution.
• Two-sample T-test compares means from two independent groups. You choose between the pooled version (assumes equal variances in both groups) and Welch's version (allows unequal variances). When in doubt, Welch's is the safer choice.
• Paired T-test handles dependent observations (before/after measurements, matched subjects). It analyzes the differences within pairs, not the raw scores, effectively reducing the problem to a one-sample T-test on those differences.

---

Comparing Multiple Groups: Beyond Two Means

When you have three or more groups, running multiple T-tests inflates your Type I error rate (the probability of falsely rejecting a true null hypothesis). With three groups, you'd need three pairwise T-tests, and the chance of at least one false positive climbs well above your chosen $\alpha$. ANOVA solves this by testing all groups simultaneously using variance decomposition.

ANOVA (One-Way, Two-Way)

• One-way ANOVA tests whether means differ across three or more groups based on a single factor. The null hypothesis is $H_0: \mu_1 = \mu_2 = \cdots = \mu_k$.
• Two-way ANOVA examines two factors simultaneously and can detect interaction effects, which occur when the impact of one factor depends on the level of another. For example, a drug might work differently for men and women.
• The F-statistic is the ratio of between-group variance to within-group variance: $F = \frac{MS_{between}}{MS_{within}}$. Larger values suggest the group means truly differ, because variation between groups is large relative to the noise within groups.

F-Test

• Compares two variances to test if they're significantly different: $F = \frac{s_1^2}{s_2^2}$, which follows an F-distribution under the null.
• Underlies ANOVA calculations. Every ANOVA table includes an F-statistic for testing mean equality.
• Assumes normality and independence. The F-test is sensitive to violations of normality, especially with small samples, so check your assumptions before relying on it.

---

Testing Relationships: Regression and Model Comparison

These methods go beyond "are groups different?" to ask "how are variables related?" and "which model explains the data better?" Regression quantifies relationships; likelihood ratio tests compare competing model specifications.

Regression Analysis (Simple and Multiple)

• Simple regression models the relationship between one predictor and one outcome: $Y = \beta_0 + \beta_1 X + \epsilon$. Here $\beta_0$ is the intercept, $\beta_1$ is the slope (the change in $Y$ for a one-unit change in $X$), and $\epsilon$ is the error term.
• Multiple regression includes two or more predictors, allowing you to control for confounding variables and assess each predictor's unique contribution while holding the others constant.
• Hypothesis tests on coefficients (testing $H_0: \beta_j = 0$) determine whether a predictor has a statistically significant linear relationship with the outcome. These typically use T-tests on individual coefficients and F-tests for joint significance of multiple coefficients.

Likelihood Ratio Test

• Compares nested models by examining the ratio of their maximum likelihoods: $\Lambda = \frac{L(\text{restricted})}{L(\text{unrestricted})}$. The restricted (null) model is a special case of the unrestricted (alternative) model with some parameters set to zero or constrained.
• The test statistic $-2 \ln(\Lambda)$ follows a chi-square distribution with degrees of freedom equal to the difference in the number of parameters between the two models.
• Essential for logistic regression and generalized linear models where you can't simply compare $R^2$ values or use standard F-tests.

---

Categorical Data: When Means Don't Apply

Not all data is continuous. When you're working with counts, categories, or frequencies, you need tests designed for discrete distributions. The chi-square test is your primary tool here.

Chi-Square Test

The chi-square test compares what you actually observe in your data to what you'd expect if there were no association between the variables.

• Test of independence: tests whether two categorical variables are associated by comparing observed cell frequencies to expected frequencies under independence.
• Goodness-of-fit: tests whether observed frequencies match a hypothesized distribution (e.g., are dice rolls uniformly distributed?).
• Test statistic: $\chi^2 = \sum \frac{(O - E)^2}{E}$, where $O$ = observed counts and $E$ = expected counts under the null hypothesis.
• Requires adequate expected frequencies. The rule of thumb is $E \geq 5$ in each cell. If this isn't met, consider Fisher's exact test instead.

---

Non-Parametric Alternatives: When Assumptions Fail

Parametric tests (Z, T, ANOVA) assume your data is normally distributed. When that assumption is violated due to small samples, skewed distributions, or ordinal data, non-parametric tests provide valid alternatives by working with ranks instead of raw values.

Wilcoxon Rank-Sum Test

• Non-parametric alternative to the two-sample T-test. Compares distributions of two independent groups without assuming normality.
• Works with ranks rather than actual values, making it robust to outliers and skewed data. You pool all observations, rank them from smallest to largest, then compare the sum of ranks between groups.
• Also called the Mann-Whitney U test. Same procedure, different name. Know both for exams.

Kolmogorov-Smirnov Test

• Compares entire distributions, not just central tendency. Tests whether two samples come from the same underlying population.
• Examines cumulative distribution functions (CDFs). The test statistic is the maximum vertical distance between the two empirical CDFs, making it sensitive to differences in location, spread, and shape.
• The one-sample version tests whether data follows a specific theoretical distribution (e.g., "is this sample normally distributed?").

---

Resampling Methods: Let the Data Speak

When you can't rely on theoretical distributions or your sample is unusual, bootstrap methods let you empirically estimate sampling distributions by repeatedly resampling your own data.

Bootstrap Methods

Here's how bootstrapping works:

1. Take your original sample of $n$ observations.
2. Draw a new sample of size $n$ with replacement from the original data. Some observations will appear multiple times; others won't appear at all.
3. Compute the statistic of interest (mean, median, ratio, etc.) on this resampled data.
4. Repeat steps 2-3 thousands of times (typically 1,000 to 10,000).
5. The distribution of those computed statistics is your bootstrap distribution, which approximates the true sampling distribution.

You can use this bootstrap distribution to estimate confidence intervals and standard errors without requiring normality or known formulas. This is particularly useful for complex statistics like medians, ratios, or correlation coefficients where theoretical distributions are hard to derive.

The bootstrap is assumption-light but not assumption-free. It does assume your original sample reasonably represents the population. If your sample is biased or too small to capture the population's structure, bootstrapping won't fix that.

---

Quick Reference Table

---

Self-Check Questions

1. You have two independent groups and non-normal data with several outliers. Which two tests could you use, and why might you prefer the non-parametric option?

2. A researcher wants to test whether a new teaching method improves scores by measuring the same students before and after the intervention. Which test is appropriate, and why would a two-sample T-test be incorrect here?

3. Compare one-way ANOVA and the two-sample T-test. Under what conditions does ANOVA become necessary, and what additional information does two-way ANOVA provide?

4. A problem presents count data showing how many customers preferred each of four product designs across three age groups. Which test would you use, and what assumption must you verify before proceeding?

5. Your data violates normality assumptions, and you need to construct a 95% confidence interval for the median. Which method allows you to do this without relying on theoretical distributions, and briefly describe how it works.