Evaluating Statistical Claims: Observational Studies and Experiments

The Digital SAT includes questions that describe a research study and ask you to evaluate what conclusions can be drawn from it. These questions test whether you understand the difference between observational studies and experiments, when results can be generalized to a larger population, and when a causal relationship can be claimed. You'll typically see 1-3 questions on this topic per test. No calculations are needed; instead, you need to read carefully and apply a few clear-cut rules about study design.

Observational Studies vs. Experiments

The single most important distinction on this topic is the difference between these two types of studies.

In observational studies, researchers simply observe and record data without intervening. They might survey people about their habits, track health outcomes over time, or compare groups that already differ in some way. For example, a researcher might compare test scores of students who chose to study with music versus students who chose to study in silence. The researcher didn't assign anyone to a group; students self-selected.

In experiments, researchers actively impose a treatment on participants. They divide subjects into groups and control what each group experiences. For example, a researcher might take 100 students and randomly assign 50 to study with music and 50 to study in silence, then compare their test scores.

This difference matters because of one critical rule:

• Observational studies can only establish correlation (an association between two variables).
• Experiments with random assignment can establish causation (one variable actually causes a change in another).

Why? In an observational study, there could be hidden differences between the groups that explain the results. Maybe students who choose to study with music are already more relaxed or more confident. You can't tell whether the music helped or whether those students would have done well anyway. In an experiment with random assignment, those hidden differences get distributed evenly across both groups, so any difference in outcomes is likely due to the treatment itself.

Random Assignment and Causal Claims

Random assignment means participants are placed into treatment and control groups by a random process, like flipping a coin or using a random number generator. This is what makes causation possible.

Here's why random assignment provides evidence for a causal relationship: when you randomly assign a large group of people to different conditions, the groups will be roughly similar in every way except the treatment. Age, motivation, background, health, personality traits, and every other variable you can think of (including ones you haven't thought of) will be approximately balanced across groups. So if one group ends up with significantly different outcomes, the most likely explanation is the treatment itself.

Example question: A researcher randomly assigned 300 employees at a company to either use standing desks or traditional desks for six months. At the end of the study, employees with standing desks reported less back pain on average. Which conclusion is best supported?

• (A) Standing desks cause less back pain among all office workers nationwide.
• (B) Standing desks cause less back pain among employees at this company.
• (C) There is an association between standing desks and less back pain among employees at this company.
• (D) Standing desks will reduce back pain for anyone who uses them.

The correct answer is (B). Because the study used random assignment, a causal claim is supported. But the participants were employees at one specific company, so the causal claim applies only to that population. Choices (A) and (D) go too far by extending the claim beyond the study's population. Choice (C) is too weak because it ignores the fact that random assignment was used, which supports causation, not just correlation.

Without random assignment, even if the study is well-designed in other ways, you're limited to claiming an association or correlation. If the question describes a study where participants chose their own group, or where the researcher simply observed existing differences, the answer should never use the word "cause."

Sampling Methods and Generalizability

Generalizability is about who the results apply to. The rule is straightforward: results can only be extended to the population from which the sample was actually drawn.

Random sampling means every member of a population had an equal (or known) chance of being selected for the study. When a study uses random sampling from a specific population, the results can be generalized to that entire population. When the sample is not random, the results can only describe the people who were actually studied.

Here's how sampling methods affect conclusions:

Example question: A researcher surveyed 150 randomly selected students at Riverside High School about their sleep habits and found that 60% get fewer than 7 hours of sleep per night. To which population can this result be generalized?

• (A) All high school students in the United States
• (B) All students at Riverside High School
• (C) All teenagers
• (D) Only the 150 students surveyed

The correct answer is (B). The sample was randomly selected from Riverside High School students, so the results extend to all students at that school. The sample wasn't drawn from all U.S. high school students or all teenagers, so (A) and (C) are too broad. And because the sample was random, (D) is too narrow.

Putting It All Together: The Two-Question Framework

Every statistical claims question on the SAT boils down to two separate questions:

1. Can we generalize? (Was the sample randomly selected from a larger population?)

• Yes → Results extend to that population.
• No → Results describe only the participants in the study.

2. Can we claim causation? (Was there random assignment to treatment groups?)

• Yes → Causation is supported.
• No → Only correlation or association is supported.

These are independent. A study can have one, both, or neither:

Example question: Researchers selected 400 people at random from all registered voters in a state. They then randomly assigned half to read a policy summary and half to read nothing. Afterward, all 400 were asked whether they supported the policy. Supporters were more common in the group that read the summary. Which conclusion is best supported?

• (A) Reading the summary is associated with increased support among registered voters in the state.
• (B) Reading the summary caused increased support among the 400 participants.
• (C) Reading the summary caused increased support among registered voters in the state.
• (D) Reading the summary caused increased support among all adults in the state.

The correct answer is (C). Random assignment was used, so causation is supported. Random sampling was used from registered voters in the state, so the result generalizes to all registered voters in the state. Choice (A) is too weak (it says "associated" when causation is justified). Choice (B) is too narrow (it limits the conclusion to the 400 participants when random sampling allows generalization). Choice (D) is too broad (the sample came from registered voters, not all adults).

What to Watch For on Test Day

1. Read the study description carefully for two specific details: Was the sample randomly selected from a population? Were participants randomly assigned to groups? These two facts determine everything.

2. Wrong answers almost always overstate or understate the conclusion. If there's no random assignment, eliminate any answer with "cause," "due to," or "result of." If the sample came from one school, eliminate any answer that says "all students nationwide."

3. Don't confuse random sampling with random assignment. Random sampling is about how participants were chosen from a population (affects generalizability). Random assignment is about how participants were placed into groups within the study (affects causation). They serve completely different purposes.

4. "Association" and "correlation" are safe; "cause" requires random assignment. When in doubt about whether a study is observational or experimental, look for language like "assigned," "placed into groups," or "given a treatment." If participants simply "were observed" or "reported their habits," it's observational.

5. Match the population exactly. If the study sampled from "customers at Store X," the results generalize to customers at Store X, not to all shoppers everywhere. The SAT is precise about this, and so should you be.