africa program for education impact evaluation dakar, senegal december 15-19, 2008 experimental...
TRANSCRIPT
![Page 1: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/1.jpg)
Africa Program for Education Impact EvaluationDakar, SenegalDecember 15-19, 2008
Experimental Methods
Muna MekyEconomist
Africa Impact Evaluation Initiative
![Page 2: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/2.jpg)
2
Motivation
• Objective in evaluation is to estimate the CAUSAL effect of intervention X on outcome Y– What is the effect of a housing upgrade on
household income?
• For causal inference, we need to understand exactly how benefits are distributed– Assigned / targeted– Take-up
![Page 3: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/3.jpg)
3
Causation versus Correlation
• Correlation is NOT causation– Necessary but not sufficient condition– Correlation: X and Y are related
• Change in X is related to a change in Y
• And….
• A change in Y is related to a change in X
– Example: age and income
– Causation – if we change X how much does Y change• A change in X is related to a change in Y
• Not necessarily the other way around
![Page 4: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/4.jpg)
4
Causation versus Correlation
Three criteria for causation:
– Independent variable precedes the dependent variable.
– Independent variable is related to the dependent variable.
– There are no third variables that could explain why the independent variable is related to the dependent variable.
![Page 5: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/5.jpg)
5
• Statistical analysis: Typically involves inferring the causal relationship between X and Y from observational data– Many challenges & complex statistics
– We never know if we’re measuring the true impact
• Impact Evaluation: – Retrospectively:
• same challenges as statistical analysis
– Prospectively:• we generate the data ourselves through the program’s design
evaluation design• makes things much easier!
Statistical Analysis & Impact Evaluation
![Page 6: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/6.jpg)
6
How to assess impact
• What is the effect of a housing upgrade on household income?
• Ideally, compare same individual with & without programs at same point in time
• What’s the problem?
• The need for a good counterfactual– What are the requirements?
![Page 7: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/7.jpg)
7
Case study: housing upgrade
• Informal settlement of 15,000 households
• Goal: upgrade housing of residents
• Evaluation question:
What is the impact of upgrading housing on household income? on employment?
• Counterfeit counterfactuals
![Page 8: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/8.jpg)
8
Gold standard:Experimental design
• Only method that ensures balance in unobserved (and observed) characteristics Only difference is treatment
• Equal chance of assignment into treatment and control for everyone
• With large sample, all characteristics average out
• Experimental design = Randomized evaluation
![Page 9: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/9.jpg)
9
“Random”
• What does the term “random” mean here?– Equal chance of participation for everyone
• How could one really randomize in the case of housing upgrading?
• Options– Lottery– Lottery among the qualified– Phase-in– Encouragement– Randomize across treatments
![Page 10: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/10.jpg)
10
Kinds of randomization
• Random selection: external validity– Ensure that the results in the sample represent the
results in the population – What does this program tell us that we can apply to
the whole country?
• Random assignment: internal validity– Ensure that the observed effect on the outcome is
due to the treatment rather than other factors – Does not inform scale-up without assumptions
• Example: Housing upgrade in Western Cape vs Sample from across country
![Page 11: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/11.jpg)
11
Randomization
Randomization
External Validity
(sample)
Internal Validity
(identification)
External vs Internal
![Page 12: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/12.jpg)
12
Example of Randomization
• What is the impact of providing free books to students on test scores?
• Randomly assign a group of school children to either:- Treatment Group – receives free books
- Control Group – does not receive free books
![Page 13: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/13.jpg)
13
Randomization
Random Assignment
![Page 14: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/14.jpg)
14
How Do You Randomize?
1) At what level? – Individual – Group
• School• Community • District
![Page 15: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/15.jpg)
15
When would you use randomization?
• Universe of eligible individuals typically larger than available resources at a single point in time– Fair and transparent way to assign benefits– Gives an equal chance to everyone in the sample
• Good times to randomize:– Pilot programs– Programs with budget/capacity constraints – Phase in programs
![Page 16: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/16.jpg)
16
Basic Setup of an Experimental Evaluation
Target Population
Potential Participants
Evaluation Sample
Random Assignment
TreatmentGroup
ControlGroup
Participants No-Shows Based on Orr (1999)
All informal settlement dwellers
Communities that might participate or a targeted sub-group
Select those you want to work with right now
![Page 17: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/17.jpg)
17
Examples…
![Page 18: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/18.jpg)
18
Beyond simple random assignment
• Assigning to multiple treatment groups– Treatment 1, Treatment 2, Control– Upgrade housing in situ, relocation to better housing,
control– What do we learn?
• Assigning to units other than individuals or households– Health Centers (bed net distribution)– Schools (teacher absenteeism project)– Local Governments (corruption project)– Villages (Community-driven development projects)
![Page 19: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/19.jpg)
19
Unit of randomization
• Individual or household randomization is lowest cost option
• Randomizing at higher levels requires much bigger samples: within-group correlation
• Political challenges to unequal treatment within a community– But look for creative solutions: e.g., uniforms in Kenya
• Some programs can only be implemented at a higher level – e.g., strengthening school committees
![Page 20: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/20.jpg)
20
Efficacy & Effectiveness
• Efficacy– Proof of Concept– Pilot under ideal conditions
• Effectiveness – At scale– Normal circumstances & capabilities– Lower or higher impact?– Higher or lower costs?
![Page 21: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/21.jpg)
21
Advantages of experiments
• Clear causal impact
• Relative to other studies– Much easier to analyze– Cheaper! (smaller sample sizes)– Easier to convey– More convincing to policymakers– Not methodologically controversial
![Page 22: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/22.jpg)
22
What if randomization isn’t possible?
It probably is…• Budget constraints: randomize among the
needy
• Roll-out capacity: randomize who receives first
• Randomly promote the program to some
![Page 23: Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bfee1a28abf838cb9c68/html5/thumbnails/23.jpg)
23
When is it really not possible?
• The treatment has already been assigned and announced
and no possibility for expansion of treatment
• The program is over (retrospective)
• Universal eligibility and universal access– Example: free education, exchange rate
regime