2.2 Counterfactuals and the potential outcomes framework

Counterfactuals are key to impact evaluation. They're hypothetical scenarios showing what would've happened without an intervention. By comparing these to actual outcomes, we can isolate the true effect of a program or policy.

The potential outcomes framework formalizes this approach. It assumes each unit has potential outcomes under all treatment conditions, helping us define and estimate causal effects in various settings. This framework guides experiment design and observational data analysis.

Counterfactuals in Impact Evaluation

Defining Counterfactuals

• Counterfactuals represent hypothetical scenarios of program participants' outcomes without the intervention
• Fundamental problem of causal inference arises from the impossibility of observing treated and untreated outcomes simultaneously for the same unit
• Essential for establishing causality in impact evaluation by providing a comparison basis against observed outcomes
• Credibility of impact evaluation depends on the quality of counterfactual construction or estimation
• Various methods approximate counterfactuals (randomized controlled trials, quasi-experimental designs)

Importance in Impact Evaluation

• Crucial for identifying and mitigating potential biases in impact evaluation studies
• Help isolate the causal effect of an intervention from other confounding factors
• Enable researchers to estimate the true impact of a program or policy
• Facilitate the comparison between what actually happened and what would have happened without the intervention
• Support policymakers in making informed decisions based on rigorous evidence

Challenges and Considerations

• Constructing reliable counterfactuals often requires careful study design and data collection
• Selection bias can threaten the validity of counterfactuals in non-randomized studies
• External validity concerns arise when generalizing results from a specific counterfactual scenario
• Temporal aspects must be considered, as counterfactuals may change over time
• Ethical considerations may limit the use of certain counterfactual designs (withholding treatment from control groups)

Potential Outcomes Framework

Core Concepts

• Formal approach to causal inference in impact evaluation (Rubin Causal Model)
• Defines causal effects as the difference between potential outcomes under treatment and control conditions for each unit
• Assumes the existence of potential outcomes for each unit under all possible treatment conditions
• Provides a clear conceptual basis for defining and estimating causal effects in various settings (experimental, observational)
• Facilitates the formulation of causal questions and the development of appropriate estimation strategies

Key Assumptions

• Stable Unit Treatment Value Assumption (SUTVA) states potential outcomes for any unit do not depend on other units' treatment status
• Conditional independence assumption (CIA) or unconfoundedness asserts treatment assignment is independent of potential outcomes given observed covariates
• Positivity assumption requires a non-zero probability of receiving each treatment level for all covariate values
• Consistency assumption ensures the observed outcome under a given treatment matches the potential outcome for that treatment
• These assumptions form the foundation for valid causal inference within the framework

Applications and Extensions

• Guides the design of randomized experiments and the analysis of observational data
• Supports the development of methods for handling complex treatment scenarios (multiple treatments, time-varying treatments)
• Facilitates the integration of machine learning techniques in causal inference
• Enables the exploration of heterogeneous treatment effects across subgroups
• Provides a framework for addressing issues such as non-compliance and missing data in causal studies

Average Treatment Effect (ATE)

Definition and Calculation

• Key causal parameter representing the expected difference in outcomes between treated and untreated states for the entire population
• Mathematically defined as $ATE = E[Y(1) - Y(0)]$, where Y(1) and Y(0) are potential outcomes under treatment and control conditions
• Estimation requires addressing the fundamental problem of causal inference using various statistical techniques and study designs
• Provides a measure of the overall impact of an intervention at the population level
• Interpretation depends on the nature of treatment and outcome variables, and the specific study context

Estimation Methods

• Randomized controlled trials (RCTs) offer unbiased estimation of ATE through simple comparison of treatment and control group means
• Propensity score methods balance observed covariates to estimate ATE in observational studies
• Instrumental variables approach can estimate ATE when valid instruments are available
• Difference-in-differences and regression discontinuity designs provide alternative strategies for ATE estimation in quasi-experimental settings
• Machine learning methods (causal forests, targeted maximum likelihood estimation) offer data-driven approaches to ATE estimation

Importance and Limitations

• Crucial for policy decisions and program evaluations, providing a comprehensive measure of intervention impact
• Allows for comparison of different interventions or policies on a common scale
• May mask heterogeneity in treatment effects across different subgroups or individuals
• Assumes SUTVA and unconfoundedness for consistent estimation, which may not always hold in practice
• Extrapolation of ATE to different populations or contexts requires careful consideration of external validity

ATE vs ATT

Conceptual Differences

• Average Treatment Effect on the Treated (ATT) focuses on the causal effect for those who received the treatment
• ATT defined as $ATT = E[Y(1) - Y(0) | T = 1]$, where T = 1 indicates treatment status
• ATE considers the entire population, while ATT is specific to the treated subgroup
• Distinction becomes important when treatment effects are heterogeneous across the population
• ATT often more relevant for evaluating voluntary programs or considering expansion of existing interventions

Estimation Considerations

• Selection bias can affect the relationship between ATE and ATT, especially in non-randomized studies
• Estimating ATT typically requires different assumptions and methods compared to ATE in observational studies
• Propensity score matching and weighting methods are commonly used to estimate ATT
• In randomized experiments, ATE and ATT are equivalent due to random assignment
• Instrumental variables can be used to estimate local average treatment effects (LATE), which may be closer to ATT than ATE

Choosing Between ATE and ATT

• Research question and policy implications guide the choice between focusing on ATE or ATT
• ATT may be more appropriate when interested in the effect on those who choose to participate in a program
• ATE is often preferred when considering universal implementation of a policy or intervention
• Feasibility of estimation given available data and study design influences the choice
• Reporting both ATE and ATT can provide a more comprehensive understanding of treatment effects