Experimental Design Principles

Why This Matters

Experimental design is the backbone of Unit 3 and shows up repeatedly throughout AP Statistics, from understanding how data should be collected to interpreting results in inference problems. When you see an FRQ asking whether a study can establish causation or merely association, you're being tested on these principles. The concepts here, randomization, control, blocking, and replication, aren't just vocabulary words. They're the tools that separate valid experiments from flawed ones.

Every design choice exists to solve a specific problem. Randomization eliminates confounding. Blocking reduces variability. Control groups provide a baseline. Blinding prevents bias. Don't just memorize these terms. Understand what problem each principle solves, because that's exactly what the exam will ask you to explain.

---

Establishing Causation: The Core Principles

The fundamental goal of an experiment is to establish a cause-and-effect relationship. Three principles work together to make that possible, and without all three, your experiment cannot support causal conclusions.

Randomization

Random assignment means each experimental unit has an equal chance of receiving any treatment, which creates roughly equivalent groups before the experiment even begins.

• Eliminates confounding variables by distributing both known and unknown lurking variables approximately evenly across treatment groups
• Enables causal inference, which is the key distinction between experiments and observational studies. Without random assignment, you can only claim association, never causation.

Replication

Having multiple experimental units per treatment lets you estimate the natural variability in responses and distinguish real effects from random noise.

• Increases statistical power by reducing the standard error of treatment effect estimates. Recall that $SE = \frac{s}{\sqrt{n}}$, so as $n$ grows, the standard error shrinks.
• Demonstrates that observed differences are consistent across many subjects, not driven by a few unusual cases

Control Groups

A control group provides a baseline for comparison. Without knowing what happens in the absence of treatment, you can't measure the treatment's effect.

• Can receive a placebo or a standard treatment, depending on the research question and ethical considerations
• Isolates the treatment effect by holding all other conditions constant between groups

---

Reducing Bias: Blinding and Placebo Controls

Even with perfect randomization, human psychology can introduce systematic errors. These techniques address bias that comes from participants' and researchers' expectations.

Blinding

• Single-blind design: participants don't know which treatment they receive, preventing their expectations from influencing their responses
• Double-blind design: neither participants nor researchers measuring outcomes know group assignments, eliminating bias from both sides
• Blinding is especially important for subjective outcomes like pain levels, mood, or perceived improvement, where expectations strongly influence what people report

Placebo Effect

The placebo effect is a psychological response to perceived treatment. Participants may improve simply because they believe they're receiving help, not because the treatment actually works.

• A placebo control group receives an inert treatment (like a sugar pill) that looks identical to the real treatment, so you can separate genuine effects from expectation-driven changes
• This is particularly important in medical and behavioral studies where outcomes depend partly on participant beliefs

---

Controlling Variability: Blocking Strategies

When experimental units differ in ways that affect the response, blocking groups similar units together before randomization. This reduces noise in your data. Think of it as sorting before shuffling.

Blocking

• Groups similar experimental units based on a characteristic expected to affect the response (age, gender, location, baseline ability)
• Randomization occurs within each block, so block differences can't confound treatment comparisons
• Reduces unexplained variability by accounting for known sources of variation, making treatment effects easier to detect

Randomized Block Design

Here's how to set up a randomized block design:

1. Identify a variable that you expect will affect the response (this becomes your blocking variable)
2. Divide experimental units into homogeneous blocks based on that variable
3. Randomly assign treatments within each block, so every treatment appears in every block

This design is more precise than a completely randomized design when the blocking variable is strongly related to the response, because it removes that source of variability from the comparison.

Matched-Pairs Design

Matched-pairs is a special case of blocking with only two treatments.

• Pairs are formed based on similar characteristics, then one member of each pair is randomly assigned to each treatment
• Alternatively, each subject can serve as their own control in a before/after design, which eliminates individual differences entirely
• Analyzed using paired differences ($\bar{x}_d$ and $s_d$), which typically have smaller variability than comparing two independent samples

---

Experimental Design Structures

Different research questions call for different experimental frameworks. The choice depends on how many factors you're studying, what resources you have, and the nature of your experimental units.

Completely Randomized Design (CRD)

The simplest structure: all units are assigned to treatments purely by chance, with no blocking or matching.

• Best for homogeneous populations where experimental units are similar enough that blocking wouldn't provide much benefit
• Straightforward to set up and analyze, but may miss real effects if there's substantial variability among units that blocking could have controlled

Factorial Design

A factorial design studies multiple factors simultaneously. For example, a $2 \times 2$ factorial examines two factors, each at two levels, creating four treatment combinations total.

• The big advantage: it can reveal interaction effects, meaning the effect of one factor depends on the level of another factor
• More efficient than running separate one-factor-at-a-time experiments, because every observation provides information about all factors at once

Crossover Design

In a crossover design, each participant receives all treatments in sequence, serving as their own control. This dramatically reduces between-subject variability.

• Requires washout periods between treatments so that the effects of one treatment don't carry over and influence the response to the next
• Works best for chronic conditions where treatment effects are temporary and reversible

---

Threats to Validity: Confounding and Bias

Understanding what can go wrong helps you design better experiments and critique flawed studies. A common FRQ task is identifying problems in a described study.

Confounding Variables

A confounding variable is associated with both the treatment and the response, making it impossible to isolate the treatment's true effect. It provides an alternative explanation for the results.

• Managed through randomization (distributes confounders evenly), blocking (controls known confounders), or holding conditions constant (same environment for all groups)
• Confounding is the core reason observational studies can't establish causation. Without random assignment, treatment groups may differ systematically in ways you can't account for.

Bias Reduction Techniques

• Randomization eliminates selection bias by preventing systematic differences between treatment groups at the start
• Blinding eliminates response bias and measurement bias by preventing expectations from influencing outcomes or data collection
• Standardized protocols reduce procedural bias by ensuring all subjects are treated identically except for the experimental treatment itself

Sample Size Determination

Larger samples increase power, which is the ability to detect a real treatment effect when one exists.

• The needed sample size depends on the expected effect size, the variability in the population, and the chosen significance level. Smaller expected effects or higher variability require larger samples.
• In practice, sample size always involves balancing precision against constraints like cost, time, and availability of experimental units.

---

Quick Reference Table

---

Self-Check Questions

1. A researcher wants to test whether a new fertilizer increases tomato yield. She has 30 plants of varying ages and sizes. Should she use a completely randomized design or a randomized block design? Explain what blocking variable she might use and why it would improve the experiment.

2. What do randomization and blinding have in common, and how do they differ? Which one allows an experiment to establish causation, and which one prevents psychological bias?

3. An experiment compares two pain medications by giving each participant both drugs (one per week) in random order. What design is this, and why is a washout period necessary between treatments?

4. A study finds that coffee drinkers have lower rates of heart disease. A journalist claims coffee prevents heart disease. Explain why this conclusion is flawed and identify at least one potential confounding variable.

5. Compare matched-pairs design and randomized block design. When would you choose matched-pairs over blocking with multiple treatments? How does the analysis of matched-pairs data differ from analyzing two independent samples?