overview on impact evaluation - world bankpubdocs.worldbank.org/en/208761469740539905/1... · 1...
TRANSCRIPT
![Page 1: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/1.jpg)
Overview on Impact EvaluationPaul Gertler | CEGA, UC-Berkeley
2016 Impact evaluation workshop & EAST AFRICA Evidence Summit
![Page 2: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/2.jpg)
Strong growth in impact evaluations
0
50
100
150
200
250
300
350
400
2006 2007 2008 2009 2010 2011 2012
110141
170
269298
364 366
All
All
Example: International Impact Evaluation Initiative (3ie) database
![Page 3: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/3.jpg)
Impact Evaluations: Overview
• Why
• What
• How
![Page 4: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/4.jpg)
We need more than programs
we need programs that work
![Page 5: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/5.jpg)
Scale Promising Programs
Example: Mozambique Pre-School
![Page 6: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/6.jpg)
Or Cut Non-Performing Programs
• Federal US program created to prevent drug use, gang membership, and violent behavior
• Police officers visit 5th and 6th grade classrooms to educate students
Evaluations show no significant impact → funding reduced(GAO, 2003; Weiss, 2003; West and O’Neil, 2004)
D.A.R.E Program (USA)
![Page 7: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/7.jpg)
Sustain Successful Programs
Example: Mexico PROGRESA
![Page 8: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/8.jpg)
Conditional Cash Transfer (CCT) Programs
Countries implementing CCT programs in
1997
![Page 9: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/9.jpg)
Conditional Cash Transfer (CCT) Programs
Countries implementing CCT programs in
2011
![Page 10: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/10.jpg)
Improve Existing Programs
Example: Zambia medicine supplies to clinics
![Page 11: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/11.jpg)
Impact Evaluation
• Why
• What
• How
![Page 12: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/12.jpg)
Our Toolbox: Monitoring and Evaluation Methods
1Needs Assessment
2Process Evaluation
3Impact Evaluation
Analysis of Beneficiary Needs
Analysis of Program
Implementation
Measures How Much the Program
Impacts Beneficiaries
1. Services being delivered?
2. Clients satisfied?3. Services reaching
target population?
1. People washing their hands more?
2. Diarrhea gone down?
3. Health of mothers and children improved?
4. Less poverty?
1. Why are we doing the project?2. Who is the target population?3. Why do we think this approach will work in this context?
![Page 13: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/13.jpg)
How to Go from Program to Results?
WASH PROGRAM• Inputs: people,
money supplies
• Activities: Construction
• Output: HandwashingFacilities
INTER-MEDIATE OUTCOMES
• People wash hands
• Less germs on hands
OUTCOMES
• Lower diarrhea and infant mortality
• Improved child well-being
![Page 14: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/14.jpg)
Example: does promoting hand-washing work?
From Meena's Three Wishes, “Meena tackles the issues of hygiene and sanitation.” UNICEF
![Page 15: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/15.jpg)
Outcome: Children’s
health
Time
Intervention
Impact?
Impact of the program
Before After
Treatment villages
![Page 16: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/16.jpg)
What would have happened without the program?
![Page 17: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/17.jpg)
Time
Intervention
Counterfactual
Actual Impact
=Treatment
-Control
Impact of the programControl needs to be a good proxy for the counterfactual
Before After
Treatment villages
Control villages
Outcome: Children’s
health
![Page 18: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/18.jpg)
Non-random program selection leads to comparisons of apples and oranges
![Page 19: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/19.jpg)
Non-random program selection leads to comparisons of apples and oranges
![Page 20: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/20.jpg)
Randomization creates the right comparison group to measure impact
![Page 21: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/21.jpg)
Randomization creates the right comparison group to measure impact
Good so-called
“Internal Validity”
![Page 22: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/22.jpg)
Randomization creates the right comparison group to measure impact
“Internal Validity” still
good
But less “External Validity”
![Page 23: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/23.jpg)
Impact Evaluation
• Why
• What
• How
![Page 24: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/24.jpg)
Pre-School Program in Mozambique• 76 communities identified to receive the program • 30 randomly assigned to the treatment. • 46 acted as the control. • Government so impressed that it expanded it to 600
communities
Randomize when there are not enough resources to reach every village
![Page 25: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/25.jpg)
…or randomize who receives the program first
Sanitation program in Indonesia
• 80 villages received the program in phase one.
• Remaining 80 villages in phase two
![Page 26: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/26.jpg)
Quasi-experimental
methods (require more assumptions)
Randomized Assignment
Difference-in-Differences
Instrumental Variables
Regression Discontinuity Design
Matching
IE Methods
Toolbox
![Page 27: Overview on Impact Evaluation - World Bankpubdocs.worldbank.org/en/208761469740539905/1... · 1 Needs Assessment 2 Process Evaluation 3 Impact Evaluation Analysis of Beneficiary Needs](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6c4af411979175cc3a354d/html5/thumbnails/27.jpg)
Thank you!Who wants to learn more about impact
evaluations?