joost de laat strategic impact evaluation fund, world...
TRANSCRIPT
Joost de LaatStrategic Impact Evaluation Fund, World Bank
Why bother with impact evaluations?
Education example: Computers in school: good idea?
That small miracle can be replicated in every school, rich and poor, across America ... Every child in American deserves a
chance to participate in the information revolution.
President Clinton (1998)
One Laptop Per Child -Peru
It depends…some rigorous impact evaluations find no impact…
• Peru: “One laptop per child” had no sizeable impact on learning.
• Colombia: “Computers for Education” program: no impact on learning
• Israel: the “tomorrow-98”, placing 35,000 computers across elementary and middles schools across Israel had no measurable impact.
…Computer-based learning was not integrated into the curriculum.
• India: but integrated computer assisted learning in India raised student performance
We need more than programs
we need programs that work
857 estimates of program impact from 207 evaluations
We need more than programs
we need programs that work
We need more than programs
we need programs that work
Other conclusions…
• Human capital programs (e.g. training): Zero (or negative) effects in short-run, larger effects in long-run
• Public sector work programs: Negative to zero effects across all time horizons
• Females and long-term unemployed: on average, larger employment impacts (compared with males and short-term unemployed)
• Youth and elderly: on average, smaller impacts than middle-aged people
Strong growth in impact evaluations, also in low and middle-income countries
0
50
100
150
200
250
300
350
400
2006 2007 2008 2009 2010 2011 2012
110141
170
269298
364 366
All
All
Example: International Impact Evaluation Initiative (3ie) database
Using impact evaluations to make program decisions
Sustain and Expand Successful Programs
Example: Mexico PROGRESA
Using IE to Sustain and Expand Programs
Countries implementing CCT programs in
1997
Using IE to Sustain and Expand Programs
Countries implementing CCT programs in
2011
Or Cut Non-Performing Programs
• Federal US program created to prevent drug use, gang membership, and violent behavior
• Police officers visit 5th and 6th grade classrooms to educate students
Evaluations show no significant impact → funding reduced(GAO, 2003; Weiss, 2003; West and O’Neil, 2004)
D.A.R.E Program (USA)
How do impact evaluations compare to other evaluations?
Our Toolbox: Monitoring and Evaluation Methods
1Needs Assessment
2Process Evaluation
3Impact Evaluation
Analysis of Beneficiary Needs
Analysis of Program
Implementation
Measures How Much the Program
Impacts Beneficiaries
1. Services being delivered?
2. Clients satisfied?3. Services reaching
target population?
1. Youth skills improved?
2. Employment rates higher?
1. Why are we doing the project?2. Who is the target population?3. Why do we think this approach will work in this context?
How to Go from Program to Results?
Education PROGRAM• Inputs: people,
money, supplies
• Activities: Teacher training
• Output: 200 Teacher trained
INTER-MEDIATE OUTCOMES
• Teacher’ knowledge increased?
• Teachers change classroom behavior?
OUTCOMES
• Student learning improved?
• Graduation rates increased?
• Employment rates improved?
How to Go from Program to Results?
Employment
PROGRAM• Inputs: people,
money, supplies
• Activities: Job search assistance
• Output: 600 Unemployed counselled
INTER-MEDIATE OUTCOMES
• Better soft skills for searching?
• Increased search activity?
OUTCOMES
• Higher employment rate?
• Higher job satisfaction?
How to measure impact?
Children’s Learning Outcomes
Time
Intervention: extra tutoring
Impact?
Before-After comparison ok?
Before After
Treatment schools
Time
Intervention
Counterfactual
Actual Impact
=Treatment
-Control
Need a comparison group (“control”) to measure impact
Control needs to be a good proxy for the counterfactual
Before After
Treatment schools
Control schools
Children’s learning
outcomes
Quasi-experimental
methods (require more assumptions)
Randomized Assignment
Difference-in-Differences
Instrumental Variables
Regression Discontinuity Design
Matching
IE Methods
Toolbox
Randomized Assignment(or “randomized control trials; RCTs”)
Why randomize? Non-random program selection leads to
comparisons of apples and oranges
Often impossible to measure impact!
Randomization creates the right comparison group to measure impact
Good so-called
“Internal Validity”
Randomization creates the right comparison group to measure impact
“Internal Validity” still
good
But less “External Validity”
Ethics and randomization:2 scenarios
1: Lottery is fair when demand for program exceeds program resources
Target population (e.g. poor communities or schools, unemployed etc.)
Treatment
Control
2. Lottery is fair when not everyone can be served at the same time
TimeInterventionBefore
evaluation period
After
Keep in Mind
Randomized Assignment
In Randomized Assignment, large enough samples produce two statistically equivalent groups.
We have identified the perfect group match!
• Sometimes we want to compare two or more different kinds of treatments– E.g., compare cost-effectiveness of efdifferent program intensities
such as 10 versus 20 weeks of training, etc.
PS.: Multi-treatment experiments
r1/3 1
T1 C
2/3
T2
1
PopulationSample
Thank you!Who wants to learn more about impact
evaluations?