Applied Impact Evaluation Unit 3 ReviewExperimental Designs

Key Concepts and Terminology

• Experimental design involves planning and conducting experiments to test hypotheses and draw conclusions about cause-and-effect relationships
• Treatment group receives the intervention or independent variable being tested while the control group does not receive the treatment
• Randomization assigns participants to treatment and control groups by chance to minimize bias and ensure groups are comparable
• Dependent variable measures the outcome or effect of the treatment on participants
• Confounding variables are extraneous factors that can influence the dependent variable and must be controlled for in the experimental design
• Statistical significance indicates the likelihood that the observed results are due to chance rather than the treatment effect
  • Commonly accepted significance level is p < 0.05, meaning there is less than a 5% probability that the results occurred by chance
• External validity refers to the extent to which the results can be generalized to other populations and settings beyond the experiment

Types of Experimental Designs

• Completely randomized design randomly assigns participants to treatment and control groups without any blocking or matching
• Randomized block design groups participants into blocks based on a relevant characteristic (age, gender) and then randomly assigns treatments within each block
• Factorial design tests the effects of two or more independent variables simultaneously by combining different levels of each variable
• Crossover design exposes each participant to both the treatment and control conditions in a randomized order with a washout period in between
• Within-subjects design measures the same participants under all treatment conditions, reducing variability but potentially introducing order effects
• Between-subjects design assigns each participant to only one treatment condition, eliminating order effects but requiring larger sample sizes
• Sequential design adjusts the allocation of participants to treatment groups based on interim results to optimize the experiment

Randomization Techniques

• Simple random sampling assigns each participant an equal probability of being selected into the treatment or control group
• Stratified random sampling divides the population into strata based on relevant characteristics and then randomly samples from each stratum
• Cluster random sampling randomly selects clusters (schools, neighborhoods) and then includes all members of the selected clusters in the experiment
• Matched pairs design matches participants on key characteristics and then randomly assigns one member of each pair to the treatment group
• Adaptive randomization adjusts the probability of assignment to treatment groups based on the characteristics of previously enrolled participants
  • Minimization is a form of adaptive randomization that assigns participants to minimize imbalances in key characteristics between groups
• Permuted block randomization assigns participants to blocks of a fixed size and then randomly allocates treatments within each block
• Covariate-adaptive randomization uses participant characteristics to balance treatment assignments and minimize covariate imbalances

Data Collection Methods

• Surveys gather self-reported data from participants through questionnaires administered in person, by phone, or online
• Interviews involve in-depth questioning of participants by a trained interviewer to collect detailed qualitative data
• Observations involve researchers directly observing and recording participant behavior or outcomes
  • Structured observations use predetermined categories and coding schemes to systematically record data
  • Unstructured observations allow researchers to freely record any relevant behaviors or events
• Physiological measures collect biological data (heart rate, blood pressure) using specialized equipment
• Administrative data is collected routinely by organizations (schools, hospitals) and can be used for experimental analysis
• Timing of data collection can include pre-test measures before the intervention, post-test measures after, and follow-up measures to assess long-term effects
• Blinding involves concealing treatment assignment from participants, researchers, or both to minimize bias in data collection and analysis

Statistical Analysis for Experiments

• Hypothesis testing uses statistical methods to determine whether the observed results are likely due to chance or the treatment effect
  • Null hypothesis ($H_0$) states that there is no significant difference between the treatment and control groups
  • Alternative hypothesis ($H_a$) states that there is a significant difference between the groups
• t-tests compare the means of two groups to determine if they are significantly different
  • Independent samples t-test compares means between two separate groups
  • Paired samples t-test compares means within the same group at different time points
• ANOVA (analysis of variance) tests for differences between three or more group means
  • One-way ANOVA compares means across one independent variable
  • Two-way ANOVA examines the effects of two independent variables and their interaction
• Regression analysis models the relationship between the dependent variable and one or more independent variables
  • Linear regression assumes a linear relationship between variables
  • Logistic regression predicts binary outcomes (success/failure)
• Effect size measures the magnitude of the treatment effect, independent of sample size
  • Cohen's d compares the difference in means relative to the pooled standard deviation
  • Odds ratio compares the odds of an outcome between the treatment and control groups

Challenges and Limitations

• Hawthorne effect occurs when participants modify their behavior due to awareness of being observed, potentially confounding the results
• Experimenter bias can occur when researchers' expectations or preferences influence the study design, data collection, or analysis
• Attrition refers to participants dropping out of the study, which can introduce bias if attrition rates differ between treatment and control groups
• Generalizability is limited when the sample is not representative of the larger population or when the experimental setting differs from real-world conditions
• Ethical considerations may restrict the use of certain treatments or the assignment of participants to control groups
• Placebo effects can occur when participants' expectations of the treatment influence their outcomes, even if the treatment is inactive
• Spillover effects happen when the treatment affects individuals beyond the intended treatment group, contaminating the control group
• Cost and feasibility constraints may limit the scale, duration, or complexity of experiments

Real-World Applications

• Clinical trials test the safety and efficacy of new medical treatments (drugs, vaccines) before approval for widespread use
• Educational interventions evaluate the impact of teaching methods, curricula, or school policies on student outcomes
• Public policy experiments assess the effectiveness of government programs (welfare, job training) in achieving desired social outcomes
• Marketing experiments test the impact of advertising campaigns, pricing strategies, or product features on consumer behavior
• Agricultural experiments compare the yield, quality, or resistance of different crop varieties or farming practices
• Environmental studies evaluate the impact of conservation interventions (protected areas, pollution reduction) on ecological outcomes
• Behavioral economics experiments study how psychological factors influence economic decision-making (risk aversion, social preferences)

Ethical Considerations

• Informed consent requires that participants understand the purpose, procedures, risks, and benefits of the experiment before agreeing to participate
• Confidentiality protects participants' privacy by securely storing data and not disclosing individual identities in published results
• Minimizing harm ensures that the risks to participants are justified by the potential benefits and that appropriate safeguards are in place
• Equitable selection of participants prevents exploitation of vulnerable populations and ensures fair representation in the sample
• Deception may be necessary for some experiments but should be minimized and fully debriefed to participants afterwards
• Coercion or undue influence can occur when incentives or authority figures pressure individuals to participate against their will
• Independent review boards (IRBs) evaluate the ethical acceptability of proposed experiments and monitor ongoing studies for compliance
• Post-experiment follow-up may be necessary to address any adverse effects or to provide promised benefits to participants