evaluating impact: from promise to evidence overview: why...
TRANSCRIPT
![Page 1: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/1.jpg)
Evaluating Impact: From Promise to Evidence
Overview: Why, What and How of Impact Evaluation
East Asia Regional Impact Evaluation Workshop Seoul, South Korea
Joost de Laat, World Bank
![Page 2: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/2.jpg)
Introduction to Impact Evaluation
• Why
• What
• How
![Page 3: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/3.jpg)
We need more than programs
we need programs that work
![Page 4: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/4.jpg)
Scale Promising Programs
With good evidence, we can spend money where it matters most
![Page 5: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/5.jpg)
Scale Promising Programs
![Page 6: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/6.jpg)
Sustain Successful Programs
![Page 7: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/7.jpg)
Improve Existing Programs
![Page 8: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/8.jpg)
Supply of Malaria Medicine
52%
78%
94%
CONTROL COMMODITY PLANNER
DIRECT PURCHASE
![Page 9: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/9.jpg)
Improve Existing Programs Video Link
![Page 10: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/10.jpg)
Impact Evaluation
• Why
• What
• How
![Page 11: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/11.jpg)
Complimentary to Other Evaluation Methods
Needs Assessment
Process Evaluation
Impact Evaluation
Analysis of Beneficiary Needs
Analysis of Program
Implementation
Measures How Much the Program
Impacts Beneficiaries
![Page 12: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/12.jpg)
• Insert Animation…
Using impact evaluations to truly determine the impact of the program
![Page 13: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/13.jpg)
Using impact evaluations to truly determine the impact of the program
![Page 14: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/14.jpg)
Does promoting hand-washing work?
From Meena's Three Wishes, “Meena tackles the issues of hygiene and sanitation.” UNICEF
![Page 15: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/15.jpg)
Outcome: Children’s
health
Time
Intervention
Impact?
Impact of the program
Before After
Treatment villages
![Page 16: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/16.jpg)
What would have happened without the program?
![Page 17: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/17.jpg)
Time
Intervention
Counterfactual
Actual Impact
= Treatment
- Control
Impact of the program Control needs to be a good counterfactual
Before After
Treatment villages
Control villages
Outcome: Children’s
health
![Page 18: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/18.jpg)
Non-random program selection leads to comparisons of apples and oranges
![Page 19: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/19.jpg)
Non-random program selection leads to comparisons of apples and oranges
![Page 20: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/20.jpg)
Randomization creates the right comparison group to measure impact
![Page 21: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/21.jpg)
Randomization creates the right comparison group to measure impact
![Page 22: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/22.jpg)
Impact Evaluations • Why? Provide sound evidence that can help us: sustain,
scale, and improve promising new programs. • What? Need a good comparison group to measure impact.
Randomization can achieve this.
• How…?
![Page 23: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/23.jpg)
Impact Evaluation
• Why
• What
• How
![Page 24: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/24.jpg)
Randomize when not enough resources to reach everyone
Northern Uganda business grant • 560 youth groups
applied. • 280 received
funding through a lottery. 280 comparison.
![Page 25: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/25.jpg)
Pre-School Program in Mozambique • 76 communities identified to receive the program • 30 randomly assigned to the treatment. • 46 acted as the control. • Government so impressed that it expanded it to 600
communities
Randomize when there are not enough resources to reach every village
![Page 26: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/26.jpg)
…or randomize who receives the program first
Sanitation program in Indonesia • 80 villages received the program
in phase one. • Remaining 80 villages in phase
two
![Page 27: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/27.jpg)
Learn more with multiple treatments
Scholarship program in Cambodia. • In 52 schools, well-performing pupils received a merit
scholarship • In 51 schools, poor pupils received a poverty-targeted
scholarship • 105 schools were control
![Page 28: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/28.jpg)
Learn more with multiple treatments
Teacher bonus program in India • 100 schools – individual bonus • 100 schools – group bonus • 100 schools – cash grant • 100 control
![Page 29: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/29.jpg)
Learn more with multiple treatments
Early Child Nutrition and Development program in Colombia. Villages randomized into 4 groups: • 24 - stimulation • 24 - micro-nutrients • 24 received both. • 24 in control
![Page 30: Evaluating Impact: From Promise to Evidence Overview: Why ...pubdocs.worldbank.org/.../Core-03-Overview-of-the...From Promise to Evidence . Overview: Why, What and How . of Impact](https://reader035.vdocuments.mx/reader035/viewer/2022070805/5f0397ef7e708231d409d1b5/html5/thumbnails/30.jpg)
Now is your chance to integrate an impact evaluation in your own program