📊Causal Inference Unit 3 – Randomized experiments

Key Concepts and Terminology

• Randomized experiments involve randomly assigning subjects to treatment and control groups to establish causal relationships
• Treatment group receives the intervention or independent variable being studied while the control group does not
• Confounding variables are extraneous factors that can influence the dependent variable and distort the causal relationship
  • Randomization helps to distribute confounding variables evenly across treatment and control groups
• Internal validity refers to the extent to which the observed effects can be attributed to the treatment rather than other factors
• External validity concerns the generalizability of the findings to other populations, settings, or contexts
• Placebo effect occurs when participants respond to an inactive treatment due to their belief in its effectiveness
• Hawthorne effect arises when subjects modify their behavior because they know they are being observed

Principles of Randomized Experiments

• Random assignment of subjects to treatment and control groups is essential for establishing causal relationships
• Blinding involves concealing the group assignment from participants, researchers, or both to minimize bias
  • Single-blind experiments hide the assignment from participants while double-blind experiments conceal it from both participants and researchers
• Replication involves conducting multiple trials or iterations of the experiment to ensure the reliability and consistency of results
• Blocking is a technique used to control for known sources of variation by grouping similar subjects together before randomization
• Stratification involves dividing the sample into subgroups based on relevant characteristics and then randomly assigning subjects within each stratum
• Balancing treatment and control groups in terms of sample size and key characteristics enhances the validity of the experiment
• Adherence to ethical guidelines, such as informed consent and minimizing harm to participants, is crucial in conducting randomized experiments

Experimental Design Strategies

• Between-subjects design assigns each participant to only one condition (treatment or control) and compares outcomes across groups
• Within-subjects design exposes each participant to all conditions in a randomized order and compares outcomes within individuals
• Factorial design manipulates two or more independent variables simultaneously to examine main effects and interactions
  • A 2x2 factorial design has two levels of each independent variable (high vs. low dosage, male vs. female)
• Crossover design involves randomly assigning participants to a sequence of treatments with a washout period in between to minimize carryover effects
• Matched-pairs design matches participants on key characteristics before randomly assigning one member of each pair to the treatment group
• Adaptive designs allow for modifications to the experiment based on interim results while maintaining the integrity of the randomization
• Sequential designs involve conducting the experiment in stages, with each stage informing the design of the next

Data Collection and Measurement

• Operational definitions specify how the variables of interest will be measured or quantified in the experiment
• Reliability refers to the consistency or stability of measurements across time, raters, or instruments
  • Test-retest reliability assesses the consistency of measurements taken at different times
  • Inter-rater reliability evaluates the agreement between multiple raters or observers
• Validity concerns the extent to which a measure accurately captures the construct it is intended to assess
  • Face validity is the degree to which a measure appears to be related to the construct based on subjective judgment
  • Construct validity examines the extent to which a measure correlates with other measures of the same construct
• Sampling techniques, such as random sampling or stratified sampling, are used to select participants from the target population
• Standardization of procedures ensures that all participants receive the same instructions, materials, and conditions to minimize extraneous variation
• Blinding of data collectors and analysts helps to prevent bias in the measurement and interpretation of results

Statistical Analysis Methods

• Hypothesis testing involves specifying null and alternative hypotheses and using statistical tests to determine the likelihood of observed results under the null hypothesis
• t-tests compare means between two groups (independent samples) or within the same group across two conditions (paired samples)
• Analysis of variance (ANOVA) tests for differences in means across multiple groups or conditions
  • One-way ANOVA examines the effect of one independent variable on the dependent variable
  • Factorial ANOVA assesses the main effects and interactions of multiple independent variables
• Regression analysis models the relationship between the independent and dependent variables and estimates the magnitude and direction of the effect
• Effect size measures, such as Cohen's d or eta-squared, quantify the strength of the relationship between variables
• Confidence intervals provide a range of plausible values for the population parameter based on the sample data
• Power analysis determines the sample size needed to detect a desired effect size with a specified level of significance and power

Interpreting Results and Drawing Conclusions

• Statistical significance indicates the likelihood that the observed results are due to chance rather than a true effect
  • A p-value less than the chosen significance level (e.g., 0.05) suggests rejecting the null hypothesis
• Practical significance considers the magnitude and real-world implications of the effect, beyond statistical significance
• Generalizability refers to the extent to which the findings can be applied to other populations, settings, or contexts
• Replication of results across multiple studies or samples strengthens the evidence for a causal relationship
• Consideration of alternative explanations helps to rule out other factors that could account for the observed effects
• Limitations of the study, such as sample size, selection bias, or measurement error, should be acknowledged and discussed
• Implications for theory, practice, and future research should be drawn based on the findings and their context

Limitations and Challenges

• Ethical considerations may limit the types of interventions or populations that can be studied in randomized experiments
• Hawthorne effect can occur when participants modify their behavior due to awareness of being observed, potentially confounding the results
• Demand characteristics arise when participants try to discern the purpose of the study and act in ways they believe are expected
• Attrition occurs when participants drop out of the study, which can introduce bias if the attrition is related to the treatment or outcome
• Generalizability may be limited if the sample is not representative of the target population or if the experimental conditions differ from real-world settings
• Cost and feasibility constraints can restrict the scope or duration of randomized experiments, particularly in large-scale or longitudinal studies
• Unintended consequences or side effects of the treatment may emerge, requiring careful monitoring and consideration in the analysis and interpretation of results

Real-World Applications and Case Studies

• Clinical trials in medical research use randomized experiments to test the safety and efficacy of new drugs, treatments, or interventions
  • The Women's Health Initiative studied the effects of hormone replacement therapy on postmenopausal women through a randomized controlled trial
• Educational interventions, such as teaching methods or curriculum changes, can be evaluated using randomized experiments in schools or classrooms
  • The Tennessee STAR experiment randomly assigned students to different class sizes to assess the impact on academic achievement
• Public policy interventions, such as welfare programs or crime prevention strategies, can be tested using randomized experiments to inform decision-making
  • The Moving to Opportunity experiment randomly assigned low-income families to receive housing vouchers to move to lower-poverty neighborhoods
• Advertising and marketing campaigns can use randomized experiments to test the effectiveness of different messages, channels, or targeting strategies
  • Online A/B testing randomly assigns website visitors to different versions of a page to compare conversion rates or engagement metrics
• Agricultural and environmental studies employ randomized experiments to evaluate the impact of different practices, technologies, or interventions on crop yields, soil health, or ecosystem services
  • The Soil Health Partnership conducts randomized trials on farms to assess the effects of cover crops, reduced tillage, and nutrient management on soil properties and crop performance