📈Applied Impact Evaluation Unit 11 – Advanced Topics in Impact Evaluation

Key Concepts and Definitions

• Impact evaluation assesses the changes that can be attributed to a particular intervention, program, or policy
• Counterfactual analysis compares what actually happened with what would have happened in the absence of the intervention
• Selection bias occurs when the reasons for participating in a program are correlated with outcomes
• Randomized controlled trials (RCTs) randomly assign individuals to treatment and control groups to estimate causal effects
  • RCTs are considered the gold standard for impact evaluation due to their ability to minimize selection bias
• Quasi-experimental designs estimate causal effects when random assignment is not possible or ethical
  • These designs include difference-in-differences, regression discontinuity, and propensity score matching
• External validity refers to the extent to which the results of an impact evaluation can be generalized to other contexts or populations
• Heterogeneous treatment effects occur when the impact of an intervention varies across different subgroups within the population

Theoretical Foundations

• The potential outcomes framework is a key theoretical foundation for impact evaluation
  • It defines causal effects as the difference between potential outcomes under treatment and control conditions
• The stable unit treatment value assumption (SUTVA) states that the potential outcomes for any individual are independent of the treatment status of other individuals
• The conditional independence assumption (CIA) holds that potential outcomes are independent of treatment assignment, conditional on observed covariates
• The overlap assumption requires that there is a positive probability of being in either the treatment or control group for all values of the covariates
• Violations of these assumptions can lead to biased estimates of causal effects
  • For example, spillover effects violate the SUTVA by allowing the treatment status of one individual to affect the outcomes of others
• The local average treatment effect (LATE) is the average effect of treatment on those individuals whose treatment status is affected by an instrumental variable
• Structural models incorporate economic theory to estimate the parameters governing individual behavior and welfare

Advanced Methodologies

• Difference-in-differences (DID) compares the changes in outcomes over time between a treatment and control group
  • DID requires the parallel trends assumption, which states that the treatment and control groups would have experienced the same trends in the absence of the intervention
• Regression discontinuity (RD) designs exploit a discontinuity in treatment assignment based on a continuous eligibility score
  • RD estimates the local average treatment effect at the threshold for treatment eligibility
• Propensity score matching (PSM) matches treated and control individuals based on their estimated probability of receiving treatment, conditional on observed covariates
• Synthetic control methods construct a weighted combination of control units that closely resembles the treatment unit prior to the intervention
• Instrumental variables (IV) estimate causal effects by using a variable that affects treatment assignment but not outcomes directly
  • Valid instruments must satisfy the relevance and exclusion restrictions
• Regression adjustment can improve the precision of treatment effect estimates by controlling for covariates that predict outcomes
• Machine learning techniques, such as lasso regression and random forests, can be used for variable selection and estimation in high-dimensional settings

Data Collection and Analysis Techniques

• Power calculations determine the sample size required to detect a desired treatment effect with a given level of statistical significance
• Stratified and clustered sampling can improve the precision of estimates and ensure representativeness across subgroups
• Survey design must minimize measurement error, nonresponse bias, and social desirability bias
  • Techniques such as double-blinding and the use of multiple measures can help address these issues
• Administrative data, such as government records or hospital databases, can provide a cost-effective source of information for impact evaluation
• Big data, including satellite imagery and social media data, can be used to measure outcomes and construct counterfactuals
• Data cleaning involves identifying and correcting errors, inconsistencies, and missing values in the dataset
• Exploratory data analysis (EDA) uses visualizations and summary statistics to identify patterns, outliers, and relationships in the data
• Sensitivity analysis assesses the robustness of the results to alternative specifications, assumptions, and estimation methods

Challenges and Limitations

• Spillover effects occur when the treatment status of one individual affects the outcomes of others, violating the SUTVA
  • Techniques such as randomized saturation designs can help estimate spillover effects
• Attrition bias arises when individuals drop out of the study or fail to respond to follow-up surveys
  • Inverse probability weighting and bounds analysis can be used to address attrition bias
• Hawthorne effects occur when individuals change their behavior in response to being observed or studied
• John Henry effects occur when individuals in the control group compensate for not receiving the treatment, biasing the estimated treatment effect
• General equilibrium effects capture the impact of an intervention on market prices and quantities, which can differ from the partial equilibrium effects estimated by most impact evaluations
• Publication bias occurs when studies with statistically significant results are more likely to be published than those with null results
  • Pre-registration of study protocols can help mitigate publication bias
• External validity may be limited if the study population or context differs from the target population or setting

Real-World Applications

• Impact evaluations have been used to assess the effectiveness of conditional cash transfer programs (Progresa in Mexico) in improving health and education outcomes
• The Graduation Approach, which provides a comprehensive package of support to ultra-poor households, has been evaluated using RCTs in multiple countries (Bangladesh, India, Ghana)
• The Jameel Poverty Action Lab (J-PAL) conducts impact evaluations of poverty alleviation programs in developing countries, using RCTs to estimate causal effects
• The World Bank's Development Impact Evaluation (DIME) initiative supports impact evaluations of development projects across sectors, including agriculture, education, and infrastructure
• Impact evaluations have been used to assess the effectiveness of community-driven development (CDD) programs in improving local governance and service delivery (Indonesia's PNPM program)
• The Abdul Latif Jameel Poverty Action Lab (J-PAL) has conducted impact evaluations of microcredit programs (Spandana in India) to assess their effects on household welfare and business outcomes
• The Millennium Villages Project, which provided an integrated package of interventions to rural communities in Africa, was evaluated using a difference-in-differences approach

Ethical Considerations

• Impact evaluations must adhere to ethical principles, including respect for persons, beneficence, and justice
  • This includes obtaining informed consent, minimizing risks to participants, and ensuring equitable selection of study populations
• Random assignment to treatment and control groups can raise ethical concerns, particularly when the intervention is believed to be beneficial
  • Waitlist designs and randomized phase-in designs can help address these concerns
• Impact evaluations should be designed to maximize the social value of the research while minimizing the risks and burdens to participants
• The use of administrative data and big data raises privacy concerns and requires appropriate safeguards and data protection measures
• Impact evaluations should be culturally sensitive and respect local norms and values
  • This may involve engaging with local communities and stakeholders throughout the evaluation process
• The results of impact evaluations should be disseminated to policymakers, practitioners, and the public in an accessible and timely manner
• Impact evaluations should aim to build local capacity and ownership, ensuring that the benefits of the research are sustained beyond the study period

Future Directions and Emerging Trends

• Machine learning techniques are increasingly being used for causal inference, including the estimation of heterogeneous treatment effects and the construction of synthetic control units
• Network analysis is being applied to study the role of social networks in the diffusion of interventions and the spillover effects of treatments
• Geospatial analysis, using satellite imagery and remote sensing data, is being used to measure outcomes and construct counterfactuals in hard-to-reach areas
• Adaptive trial designs, which allow for the modification of study parameters based on interim results, are being used to improve the efficiency and ethical acceptability of impact evaluations
• Long-term follow-up studies are being conducted to assess the persistence and fadeout of treatment effects over time
  • This includes the use of administrative data linkages to track outcomes beyond the study period
• Impact evaluations are being integrated with cost-effectiveness analysis to inform resource allocation decisions and prioritize interventions
• There is growing interest in using impact evaluations to study the effectiveness of complex, multi-component interventions, such as health system strengthening and governance reforms