3.6 Selecting an Experimental Design

Choosing an experimental design means matching the design to your question, your resources, and the kind of experimental units you have. The three you need to know are completely randomized design, randomized block design, and matched pairs design, and each one has trade-offs you can explain.

Why This Matters for the AP Statistics Exam

This topic builds the skill of explaining why a particular experimental design is appropriate for a given situation. On the exam, you may be asked to recognize a design from a description, compare designs, or justify a choice in a written response. The key reasoning is that every design has advantages and disadvantages depending on the question of interest, the resources available, and the nature of the experimental units.

A common point that trips students up is precise language. Use random assignment when you talk about experiments and treatments. Save random selection for talking about generalizing a sample to a population. When you justify that treatments caused a difference, point to random assignment, not other features of the experiment.

more resources to help you study

practice multiple choiceFRQ practice & scoringcheatsheetsscore calculatorkey terms

Key Takeaways

• A completely randomized design assigns treatments to all experimental units purely at random, which tends to balance out confounding variables.
• A randomized block design groups similar units into blocks first, then randomly assigns treatments within each block to reduce variability from a known variable.
• A matched pairs design is a special case of blocking that compares two treatments using paired units or by giving each unit both treatments.
• No design is automatically best. Justify your choice using the question, available resources, and the nature of the units.
• Use random assignment language for experiments and random selection language for generalizing to a population.
• When you claim treatments caused a difference, cite random assignment specifically.

The Three Designs You Need

Completely randomized designs are the simplest, but the simplest method is not always the most precise. After you select your experimental units, you choose how to assign treatments. Keep the language of experiments separate from the language of sample surveys.

Completely Randomized Design

Treatments are assigned to experimental units completely at random, so each unit has an equal chance of landing in any treatment group. Random assignment tends to balance the effects of confounding variables across groups, which lets you attribute differences in the response to the treatments. This design is straightforward to set up and works well when you do not need to control for a specific known variable.

Advantage: simple to plan and carry out. Disadvantage: it does not separate out variability from a known influencing variable, so results can be less precise.

Randomized Block Design

Experimental units are sorted into groups called blocks based on a variable known to influence the response. Units within a block are similar with respect to that blocking variable. Treatments are then randomly assigned within each block, and you use every block. Blocking separates natural variability from differences due to the blocking variable, which can make your comparison more precise.

Advantage: controls for a known variable and reduces unwanted variability. Disadvantage: takes more planning and is not always practical depending on resources and the number of units.

Matched Pairs Design

This is a special case of a blocking design. Units are arranged in pairs matched on relevant factors like age or ability level. Each pair gets both treatments, either by randomly assigning one treatment to each member of the pair, or by giving each subject both treatments. It is useful for comparing two treatments while controlling for known confounding variables.

Advantage: tight control over a specific variable with pairs. Disadvantage: can be time consuming and may not be feasible with limited resources or large groups.

How to Use This on the AP Statistics Exam

Free Response

When you write a plan for a design, be specific and complete. A clear completely randomized design plan usually includes these steps:

• Compile a list of all experimental units that will participate.
• Randomly assign each unit to a treatment group using a random number generator or another chance device.
• Carry out each treatment with its assigned group.
• Measure the response variable for every unit.
• Compare the groups (often by comparing means) to evaluate the treatments.

For a blocking design, add the blocking steps before assignment:

• Gather information on the blocking variable for each unit.
• Sort units into blocks so units within a block are similar on that variable.
• Within each block, randomly assign units to the treatment groups.
• Carry out treatments, measure the response, and compare groups.

Justifying a Design

When a question asks you to recommend a design, name the design and tie your reasoning to the situation. State the advantage that matters here and the disadvantage you are accepting. For example, if a variable like course load could affect grades, a blocking design lets you control for it, and it is often more practical than matched pairs for a large group.

Common Trap

When you explain why treatments caused a difference, cite the random assignment of treatments. Do not list other features of a well-designed experiment as your reason for causation, and do not slip into random selection language.

Practice Problems

(1) A researcher is interested in studying the effectiveness of a new teaching method for high school math students. The researcher plans to randomly assign 50 students to either the control group or the experimental group. The control group will receive the traditional teaching method, while the experimental group will receive the new teaching method.

At the end of the study, the researcher will administer a math achievement test to all of the students. The researcher will then compare the mean math achievement scores of the two groups to determine if the new teaching method is more effective than the traditional method.

The researcher wants to use a completely randomized design for this study.

Write a detailed plan for how the researcher should set up the completely randomized design, including how the students should be assigned to the control and experimental groups.

(2) A researcher is interested in studying the effectiveness of a new study technique on college students' grades. The researcher plans to recruit 100 students from a large university and randomly assign them to either the control group or the experimental group. The control group will receive the traditional study technique, while the experimental group will receive the new study technique.

However, the researcher is aware that students' grades can be affected by their major and the difficulty of their course load. To control for these factors, the researcher wants to use a blocking design in the study.

Write a detailed plan for how the researcher should set up the blocking design, including how the students should be assigned to the control and experimental groups and how the researcher should control for major and course load.

(3) A researcher is interested in studying the effectiveness of a new study technique on college students' grades. The researcher plans to recruit 100 students from a large university and randomly assign them to either the control group or the experimental group. The control group will receive the traditional study technique, while the experimental group will receive the new study technique.

The researcher is considering using a completely randomized design, a blocking design, or a matched pairs design for the study.

Write a detailed explanation of the advantages and disadvantages of each design and recommend which design the researcher should use, providing a rationale for your recommendation.

Answers

(1) To set up the completely randomized design, the researcher should first compile a list of all 50 students who will participate. The researcher should then randomly assign each student to either the control group or the experimental group using a random number generator or a computer program that generates random assignments.

Once the students are assigned, the researcher implements the two teaching methods: the traditional method for the control group and the new method for the experimental group.

After the study is complete, the researcher administers the math achievement test to all students and calculates the mean score for each group.

To determine if the new method is more effective, the researcher compares the mean scores. If the experimental group's mean is significantly higher than the control group's mean, that is evidence the new method is more effective.

In bullet form:

• Compile a list of all 50 students who will participate.
• Randomly assign each student to the control or experimental group using a random number generator.
• Implement the two teaching methods: traditional for the control group, new for the experimental group.
• Administer a math achievement test to all students.
• Calculate the mean math achievement scores for both groups.
• Compare the mean scores to evaluate whether the new method is more effective.

(2) To set up the blocking design, the researcher should first gather information on each student's major and course load, for example from transcripts or a survey.

Next, the researcher divides students into blocks based on major and course load, grouping students who are similar on these variables.

Within each block, the researcher randomly assigns students to the control or experimental group using a random number generator. This keeps the two groups balanced within each block in terms of major and course load.

Then the researcher implements the two study techniques: traditional for the control group, new for the experimental group. At the end, the researcher collects grade data and compares the mean grades of the two groups.

In bullet form:

• Gather information on each student's major and course load.
• Sort students into blocks based on major and course load.
• Within each block, randomly assign students to the control or experimental group using a random number generator.
• Implement the two study techniques: traditional for the control group, new for the experimental group.
• Collect grade data and compare the mean grades to evaluate the new technique.

(3)

• A completely randomized design randomly assigns participants to the two groups without sorting on other factors. Advantage: simple and easy to implement. Disadvantage: it does not control for a known variable that could affect the outcome.
• A randomized block design sorts participants into blocks based on a blocking variable, then randomly assigns participants within each block to the treatments. Advantage: it reduces the influence of a known variable on the outcome. Disadvantage: it takes more planning and may not always be feasible given resources and the number of participants.
• A matched pairs design pairs participants on a factor such as ability level, then randomly assigns treatments within each pair. Advantage: it controls tightly for a specific variable. Disadvantage: it can be time consuming and may not be feasible with a large number of participants.

In this case, a blocking design is a strong choice. It lets the researcher control for major and course load, both of which can affect grades, and it is more practical than matched pairs for a large group of participants. The researcher should still weigh the available resources and time before finalizing the design.

Common Misconceptions

• Mixing up random selection and random assignment. Random selection is about choosing who is in your sample so you can generalize to a population. Random assignment is about assigning treatments so you can claim cause.
• Thinking a completely randomized design is always worse than blocking. It is simpler and often perfectly appropriate. Blocking only helps when there is a known variable worth controlling for.
• Believing there is one correct design. The right choice depends on the question, the resources, and the nature of the units, and you have to justify it.
• Forgetting to use every block. In a blocked experiment, you assign treatments within each block and use all blocks, not just some.
• Citing the wrong reason for causation. When justifying that treatments caused a difference, point to random assignment, not just the presence of a control group or other design features.
• Treating matched pairs as separate from blocking. Matched pairs is a special case of a randomized block design, not a totally different idea.

Related AP Statistics Guides

• 3.1 Introducing Statistics: Do the Data We Collected Tell the Truth?
• 3.3 Random Sampling and Data Collection
• 3.5 Introduction to Experimental Design
• 3.2 Introduction to Planning a Study
• 3.4 Potential Problems with Sampling
• 3.7 Inference and Experiments