6.7 Potential Errors When Performing Tests

TLDR

A Type I error means you reject a true null hypothesis (a false positive), and a Type II error means you fail to reject a false null hypothesis (a false negative). The probability of a Type I error is the significance level alpha, the probability of a Type II error is beta, and power (1 minus beta) is the chance of correctly rejecting a false null. On the AP Statistics exam, you need to identify each error in context, explain its consequence, and know what increases power.

more resources to help you study

practice multiple choiceFRQ practice & scoringcheatsheetsscore calculatorkey terms

Why This Matters for the AP Statistics Exam

Significance tests can give the wrong answer just by random chance, even when your procedure is perfect. This topic builds the reasoning you need to talk about that risk clearly. On multiple-choice questions, you may be asked to match a scenario to the correct error type or to predict how a change in sample size or alpha affects power. On free-response questions, you often have to describe a Type I or Type II error in context, state a real consequence, and explain how to increase the power of a test. Getting comfortable with the link between alpha, beta, and power also helps you make smart decisions about significance levels later in inference units for means, chi-square, and slopes.

Key Takeaways

• A Type I error rejects a true null hypothesis (false positive); its probability is the significance level alpha.
• A Type II error fails to reject a false null hypothesis (false negative); its probability is beta.
• Power is the probability of correctly rejecting a false null hypothesis, and power = 1 minus beta.
• The probability of a Type II error decreases when sample size increases, alpha increases, standard error decreases, or the true parameter is farther from the null.
• Lowering alpha reduces Type I error risk but raises Type II error risk, so there is a trade-off.
• Which error is worse depends on the situation, and that consequence should guide your choice of alpha.

The Two Error Types

Every hypothesis test ends in one of two decisions: reject the null hypothesis or fail to reject it. Either decision can be wrong, and the two ways of being wrong have names.

Type I Error

A Type I error happens when the null hypothesis is actually true but you reject it. This is a false positive. It usually comes from getting an unusually rare sample that produces a small p-value, which makes the evidence look stronger than it really is. The probability of a Type I error is the significance level alpha. Common choices like 0.05 or 0.01 keep this probability small on purpose.

Type II Error

A Type II error happens when the null hypothesis is actually false but you fail to reject it. This is a false negative. Here the p-value was not small enough to reject, even though the alternative is really true. The probability of a Type II error is beta.

A simple memory trick: the probabilities of a Type I and Type II error are alpha and beta, in that order.

Power and What Affects Errors

Power is the probability that a test correctly rejects a false null hypothesis. It is the complement of beta, so power = 1 minus beta. A test with high power is good at detecting a real effect when one exists.

The probability of a Type II error decreases (and power increases) when any of the following happens, as long as the others stay the same:

1. Sample size increases.
2. Significance level alpha increases.
3. Standard error decreases.
4. The true parameter value is farther from the null.

Notice the trade-off built into alpha. Raising alpha increases power and lowers Type II error risk, but it also raises the probability of a Type I error. Lowering alpha does the reverse. A significance level of 0.05 is a common middle ground, but the right choice depends on which error is more costly. Because alpha is the probability of a Type I error, the consequences of that error should guide how you set alpha.

The most reliable way to increase power without raising your Type I error rate is to increase the sample size. A larger sample gives a smaller standard error and a sampling distribution that is tighter around the truth, so a real difference is easier to detect.

How to Use This on the AP Statistics Exam

MCQ

• Match a described mistake to the correct error type. Reject a true null is Type I; fail to reject a false null is Type II.
• Predict how a change affects power or beta. For example, increasing sample size or increasing alpha both increase power.
• Remember that alpha equals the probability of a Type I error, so changing the significance level directly changes that risk.

Free Response

• Describe a Type I or Type II error in context. Do not just give the textbook definition; state what rejecting or failing to reject means for the specific claim and population in the problem.
• State a consequence in context. Explain what actually goes wrong in the real situation if that error is made.
• Explain how to increase power. The safest answer is to increase the sample size, since that raises power without increasing the Type I error rate.

Common Trap

Writing error definitions in the abstract instead of in context. A response that says "we rejected a true null" without tying it to the actual claim usually will not support a stronger score on a context question.

Worked Example

A researcher tests the claim that 85% of people are satisfied with their personal reading goals. The researcher suspects the true proportion is lower and that a new public library would help. The hypotheses are:

H0: p = 0.85

Ha: p < 0.85

a) Describe a Type II error in context and give a consequence.

A Type II error here means failing to reject H0 when it is actually false. The researcher would conclude there is not convincing evidence that the true proportion satisfied is less than 0.85, when in reality it is less than 0.85. A consequence is that people stay unhappy with their reading progress and the community misses out on a new library that could have helped them reach their goals.

b) How can the researcher increase the power of this test?

Increase the sample size. A larger sample lowers the standard error and makes it easier to detect that the true proportion is below 0.85, which raises power and lowers the probability of a Type II error.

Common Misconceptions

• Failing to reject the null does not prove the null is true. It only means there was not enough evidence for the alternative.
• A Type I error is not caused by a calculation mistake. It can happen even when every step is done correctly, just from a rare sample.
• Lowering alpha does not make a test better overall. It reduces Type I error risk but increases Type II error risk.
• Power and beta are not the same thing. Power is the chance of correctly rejecting a false null, and beta is the chance of missing it, so power = 1 minus beta.
• Increasing alpha does increase power, but it is not a free fix, because it also raises the chance of a Type I error.
• Which error is worse is not fixed. It depends on the real-world consequences of each mistake in the given situation.

Related AP Statistics Guides

• Unit 6 Overview: Inference for Categorical Data: Proportions
• 6.2 Constructing a Confidence Interval for a Population Proportion
• 6.1 Introducing Statistics: Why Be Normal?
• 6.4 Setting Up a Test for a Population Proportion
• 6.3 Justifying a Claim Based on a Confidence Interval for a Population Proportion
• 6.5 Interpreting p-Values