🎲Data Science Statistics Unit 11 – Hypothesis Testing & p-values

What's the Big Idea?

• Hypothesis testing is a statistical method used to make decisions or draw conclusions about a population based on sample data
• Involves formulating a null hypothesis ($H_0$) and an alternative hypothesis ($H_a$) about a population parameter
• Null hypothesis assumes no effect or difference, while the alternative hypothesis proposes an effect or difference exists
• Collect sample data and calculate a test statistic to determine the likelihood of observing the data under the null hypothesis
• Use the p-value, which represents the probability of observing the data or more extreme results given the null hypothesis is true, to make a decision
  • If the p-value is less than a predetermined significance level (often 0.05), reject the null hypothesis in favor of the alternative hypothesis
  • If the p-value is greater than the significance level, fail to reject the null hypothesis due to insufficient evidence
• Hypothesis testing helps researchers and data scientists make informed decisions and draw meaningful conclusions from data

Key Concepts to Know

• Null hypothesis ($H_0$): A statement assuming no effect or difference in the population parameter of interest
• Alternative hypothesis ($H_a$): A statement proposing an effect or difference in the population parameter, often the researcher's claim
• Test statistic: A value calculated from the sample data used to determine the likelihood of observing the data under the null hypothesis (e.g., z-score, t-score, $\chi^2$)
• p-value: The probability of observing the data or more extreme results, assuming the null hypothesis is true
  • Ranges from 0 to 1, with smaller values indicating stronger evidence against the null hypothesis
• Significance level ($\alpha$): A predetermined probability threshold (often 0.05) used to make decisions about rejecting or failing to reject the null hypothesis
• Type I error: Rejecting the null hypothesis when it is actually true (false positive)
  • The probability of a Type I error is equal to the significance level ($\alpha$)
• Type II error: Failing to reject the null hypothesis when the alternative hypothesis is actually true (false negative)
  • The probability of a Type II error is denoted by $\beta$
• Power: The probability of correctly rejecting the null hypothesis when the alternative hypothesis is true (1 - $\beta$)

The Hypothesis Testing Process

1. State the null hypothesis ($H_0$) and alternative hypothesis ($H_a$) based on the research question or problem
2. Determine the appropriate test statistic and distribution (e.g., z-test, t-test, $\chi^2$ test) based on the data and assumptions
3. Set the significance level ($\alpha$) for the test (often 0.05)
4. Collect sample data and calculate the test statistic using the appropriate formula
5. Calculate the p-value associated with the test statistic using the distribution and sample size
6. Compare the p-value to the significance level ($\alpha$) and make a decision:
   • If p-value < $\alpha$, reject the null hypothesis in favor of the alternative hypothesis
   • If p-value ≥ $\alpha$, fail to reject the null hypothesis due to insufficient evidence
7. Interpret the results in the context of the research question or problem, considering the practical significance and limitations of the study

Understanding p-values

• The p-value represents the probability of observing the sample data or more extreme results, assuming the null hypothesis is true
• Ranges from 0 to 1, with smaller values indicating stronger evidence against the null hypothesis
• Interpretation of p-values:
  • A small p-value (typically < 0.05) suggests that the observed data is unlikely to occur by chance alone if the null hypothesis is true, providing evidence to reject the null hypothesis
  • A large p-value (typically ≥ 0.05) suggests that the observed data is likely to occur by chance alone if the null hypothesis is true, providing insufficient evidence to reject the null hypothesis
• P-values do not provide information about the size or practical significance of an effect, only the statistical significance
• P-values are affected by sample size, with larger sample sizes more likely to yield smaller p-values for the same effect size
• It is important to consider the context, limitations, and potential biases of the study when interpreting p-values

Types of Hypothesis Tests

• One-sample tests: Compare a sample statistic to a known population parameter
  • One-sample z-test: Used when the population standard deviation is known and the sample size is large (n ≥ 30) or the population is normally distributed
  • One-sample t-test: Used when the population standard deviation is unknown and the sample size is small (n < 30) or the population is normally distributed
• Two-sample tests: Compare two independent samples to determine if there is a significant difference between their means or proportions
  • Independent samples t-test: Used when comparing the means of two independent groups, assuming equal variances and normally distributed populations
  • Welch's t-test: Used when comparing the means of two independent groups with unequal variances, assuming normally distributed populations
  • Two-proportion z-test: Used when comparing the proportions of two independent groups, assuming large sample sizes (np ≥ 10 and n(1-p) ≥ 10 for each group)
• Paired tests: Compare two related samples or repeated measures on the same individuals
  • Paired t-test: Used when comparing the means of two related samples or repeated measures, assuming normally distributed differences
  • Wilcoxon signed-rank test: A non-parametric alternative to the paired t-test when the normality assumption is violated
• Analysis of Variance (ANOVA): Compare the means of three or more independent groups
  • One-way ANOVA: Used when comparing the means of three or more independent groups, assuming equal variances and normally distributed populations
  • Two-way ANOVA: Used when examining the effects of two independent variables (factors) on a dependent variable, as well as their interaction
• Chi-square tests: Used for categorical data to test the independence of two variables or the goodness-of-fit of a distribution
  • Chi-square test of independence: Used to determine if there is a significant association between two categorical variables
  • Chi-square goodness-of-fit test: Used to determine if an observed distribution of categorical data fits an expected distribution

Common Mistakes and Pitfalls

• Misinterpreting the p-value as the probability of the null hypothesis being true or the alternative hypothesis being false
  • The p-value is the probability of observing the data or more extreme results, assuming the null hypothesis is true
• Confusing statistical significance with practical significance
  • A small p-value indicates statistical significance but does not necessarily imply a large or practically meaningful effect
• Failing to check assumptions of the hypothesis test
  • Each test has specific assumptions (e.g., normality, equal variances) that must be met for the results to be valid
  • Violating assumptions can lead to incorrect conclusions and Type I or Type II errors
• Multiple testing and the increased risk of Type I errors
  • Conducting multiple hypothesis tests on the same data increases the likelihood of obtaining a significant result by chance alone
  • Use appropriate methods to control for multiple testing, such as the Bonferroni correction or false discovery rate (FDR) control
• Overreliance on hypothesis testing and p-values
  • Hypothesis testing should be used in conjunction with other statistical methods, such as confidence intervals and effect sizes, to provide a more comprehensive understanding of the data
  • Consider the context, limitations, and potential biases of the study when interpreting results
• Insufficient sample size and power
  • Small sample sizes may not have enough power to detect a significant effect, increasing the risk of Type II errors
  • Conduct power analyses to determine the appropriate sample size for a desired level of power and effect size

Real-world Applications

• Clinical trials: Hypothesis testing is used to evaluate the effectiveness and safety of new drugs, treatments, or medical devices
  • Example: A pharmaceutical company conducts a randomized controlled trial to compare the efficacy of a new drug to a placebo in reducing blood pressure
• A/B testing in marketing and web design: Hypothesis testing is used to compare the performance of two or more versions of a website, advertisement, or email campaign
  • Example: An e-commerce company tests two different layouts of their product page to determine which one leads to higher conversion rates
• Quality control in manufacturing: Hypothesis testing is used to monitor and ensure the quality of products by comparing sample measurements to specified standards
  • Example: A manufacturing plant tests the strength of their steel beams to ensure they meet the required specifications
• Psychology and social science research: Hypothesis testing is used to investigate the relationships between variables and test theories about human behavior
  • Example: A psychologist conducts a study to determine if there is a significant difference in stress levels between two different therapy techniques
• Environmental studies: Hypothesis testing is used to assess the impact of human activities on ecosystems and test the effectiveness of conservation efforts
  • Example: An ecologist compares the biodiversity of two forest areas, one with and one without a conservation program, to evaluate the program's success

Practice Problems and Solutions

1. A researcher claims that the average weight of a certain species of fish is greater than 10 pounds. A random sample of 25 fish has a mean weight of 10.5 pounds with a standard deviation of 1.2 pounds. Conduct a one-tailed hypothesis test at the 0.05 significance level to determine if there is sufficient evidence to support the researcher's claim.

   • $H_0$: $\mu \leq 10$
   • $H_a$: $\mu > 10$
   • Test statistic: $t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}} = \frac{10.5 - 10}{1.2/\sqrt{25}} = 2.08$
   • p-value: $P(t_{24} > 2.08) = 0.024$
   • Since the p-value (0.024) is less than the significance level (0.05), we reject the null hypothesis and conclude that there is sufficient evidence to support the researcher's claim.

2. A company claims that their new battery has a mean life of more than 500 hours. A random sample of 50 batteries has a mean life of 510 hours with a standard deviation of 40 hours. Conduct a one-tailed hypothesis test at the 0.01 significance level to determine if there is sufficient evidence to support the company's claim.

   • $H_0$: $\mu \leq 500$
   • $H_a$: $\mu > 500$
   • Test statistic: $z = \frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}} = \frac{510 - 500}{40/\sqrt{50}} = 1.77$
   • p-value: $P(Z > 1.77) = 0.038$
   • Since the p-value (0.038) is greater than the significance level (0.01), we fail to reject the null hypothesis and conclude that there is insufficient evidence to support the company's claim at the 0.01 significance level.

3. A study is conducted to compare the effectiveness of two different teaching methods on student test scores. A random sample of 30 students is selected for each method, and their test scores are recorded. The sample mean and standard deviation for Method A are 85 and 6, respectively, while the sample mean and standard deviation for Method B are 82 and 5, respectively. Conduct a two-tailed hypothesis test at the 0.05 significance level to determine if there is a significant difference in the mean test scores between the two methods.

   • $H_0$: $\mu_A = \mu_B$
   • $H_a$: $\mu_A \neq \mu_B$
   • Test statistic: $t = \frac{\bar{x}_A - \bar{x}_B}{\sqrt{\frac{s_A^2}{n_A} + \frac{s_B^2}{n_B}}} = \frac{85 - 82}{\sqrt{\frac{6^2}{30} + \frac{5^2}{30}}} = 2.07$
   • p-value: $P(|t_{58}| > 2.07) = 0.043$
   • Since the p-value (0.043) is less than the significance level (0.05), we reject the null hypothesis and conclude that there is a significant difference in the mean test scores between the two teaching methods.