impact evaluation for evidence-based policy making
DESCRIPTION
Impact Evaluation for Evidence-Based Policy Making. Arianna Legovini Lead Specialist Africa Impact Evaluation Initiative. How to turn this child…. …into this child. Why Evaluate?. Fiscal accountability Allocate limited budget to what works best Program effectiveness - PowerPoint PPT PresentationTRANSCRIPT
Impact Evaluation Impact Evaluation for Evidence-Based Policy Makingfor Evidence-Based Policy Making
Arianna LegoviniArianna LegoviniLead SpecialistLead Specialist
Africa Impact Evaluation InitiativeAfrica Impact Evaluation Initiative
2
How to turn this child…
3
…into this child
4
Why Evaluate?
• Fiscal accountability– Allocate limited budget to what works best
• Program effectiveness– Managing by results: do more of what works
• Political sustainability– Negotiate budget– Inform constituents
5
Traditional M&E and Impact Evaluation
• monitoring to track implementation efficiency (input-output)
INPUTS OUTCOMESOUTPUTS
MONITOR EFFICIENCY
EVALUATE EFFECTIVENESS
$$$
BEHAVIOR
impact evaluation to measure effectiveness (output-outcome)
6
Question types and methods
• M&E: monitoring & process evaluation
Descriptive Descriptive analysisanalysis
Causal Causal analysisanalysis
▫What was the effect of the program on outcomes?
▫How would outcomes change under alternative program designs?
▫Is the program cost-effective?
▫Is program being implemented efficiently?
▫Is program targeting the right population?
▫Are outcomes moving in the right direction?
• Impact Evaluation:
7
Answer with traditional M&E or IE?
• Are nets being delivered as planned?
• Do IPTs increase cognitive ability?
• What is the correlation between HIV treatment and prevalence?
• How does HIV testing affect prevention behavior?
M&E
IE
M&E
IE
8
Efficacy & Effectiveness
• Efficacy: – Proof of Concept– Pilot under ideal conditions
• Effectiveness:– At scale– Normal circumstances & capabilities– Lower or higher impact?– Higher or lower costs?
9
Use impact evaluation to….
• Test innovations• Scale up what works (e.g. de-worming)• Cut/change what does not (e.g. HIV
counseling)• Measure effectiveness of programs (e.g.
JTPA )• Find best tactics to change people’s
behavior (e.g. bring children to school)• Manage expectations
10
What makes a good impact evaluation?
11
Evaluation problem• Compare same individual with & without a program at
the same point in time
• BUT Never observe same individual with and without program at same point in time
• Formally the impact of the program is:
α = (Y | P=1) - (Y | P=0)
• Example
– How much does an anti-malaria program lower under-five mortality?
12
Solving the evaluation problem
• Counterfactual: what would have happened without the program
• Estimate counterfactual– i.e. find a control or comparison group
• Counterfactual Criteria– Treated & counterfactual groups have
identical initial average characteristics– Only reason for the difference in
outcomes is due to the intervention
13
“Counterfeit” Counterfactuals
• Before and after:– Same individual before the treatment
• Non-Participants:– Those who choose not to enroll in program, or– Those who were not offered the program
– Problem:
We can not determine why some are treated and some are not
14
Before and After Example
• Food Aid– Compare mortality before and after– Observe mortality increases– Did the program fail?– “Before” normal year, but “after” famine
yearCannot separate (identify) effect of food
aid from effect of drought
15
Before & After
• Compare Y before & after intervention
Before & after counterfactual = BEstimated impact = A-
B
• Control for time varying factors
True counterfactual = CTrue impact = A-C
A-B is under-estimatedTime
Y
AfterBefore
A
B
C
t-1 t
Treatment
B
16
Non-Participants….
• Compare non-participants to participants
• Counterfactual: non-participant outcomes
• Problem: why did they not participate?
• Estimated Impact
αi = (Yit | P=1) - (Ykt| P=0) ,
• Hypothesis :
(Ykt| P=0) = (Yit| P=0)
17
• Mothers who came to the health unit for ORT and mothers who did not?
• Communities that applied for funds for IRS and communities that did not?
• People who receive ART and people who do not?
Child had diarrhea
Access to clinic
Costal and mountain
Epidemic and non-epidemic
People with HIV
Access to clinic
Exercise: Why participants and non-participants might differ?
18
Health program example
• Treatment offered
• Who signs up?– Those who are sick– Areas with epidemics
• Have lower health status that those who do not sign up
• Healthy people/communities are a poor estimate of counterfactual
19
What's wrong?
• Selection bias: People choose to participate for specific reasons
• Many times reasons are directly related to the outcome of interest
• Cannot separately identify impact of the program from these other factors/reasons
20
Need to know…
• Why some get assigned to treatment and others to control group. If reasons correlated with outcome – cannot separately identify program impact
from– these other “selection” factors
• The process by which data is generated
21
Possible Solutions…
• Guarantee comparability of treatment and control groups
• ONLY remaining difference is intervention
• How?– Experimental design/randomization– Quasi-experiments
• Regression Discontinuity
• Double differences
– Instrumental Variables
22
These solutions all involve…
• EITHER Randomization– Give all equal chance of being in
control or treatment groups– Guarantees that all factors/characteristics will
be on average equal between groups– Only difference is the intervention
• OR Transparent & observable criteria for assignment into the program
23
Finding controls: opportunities
• Budget constraints: – Eligible who get it = potential treatments– Eligible who do not = potential controls
• Roll-out capacity:– Those who go first = potential treatments– Those who go later = potential controls
24
Finding controls: ethical considerations
• Do not delay benefits: Rollout based on budget/capacity constraints
• Equity: equally deserving populations deserve an equal chance of going first
• Transparent & accountable method
– Give everyone eligible an equal chance
– If rank based on criteria, then criteria should be measurable and public
25
Thank you