overview: why, what and how of impact evaluation · 12/9/2014 · overview: why, what and how of...
TRANSCRIPT
![Page 1: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/1.jpg)
Overview: Why, What and How of Impact Evaluation
Impact Evaluation Workshop Debre Zeit, Ethiopia
Joost de Laat, World Bank
December 9 – 12, 2014
![Page 2: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/2.jpg)
Introduction to Impact Evaluation
• Why
• What
• How
![Page 3: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/3.jpg)
We need more than programs
we need programs that work
![Page 4: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/4.jpg)
Scale Promising Programs
![Page 5: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/5.jpg)
Sustain Successful Programs
![Page 6: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/6.jpg)
Improve Existing Programs
![Page 7: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/7.jpg)
Impact Evaluation
• Why
• What
• How
![Page 8: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/8.jpg)
Our Toolbox:
Monitoring and Evaluation Methods 1
Needs Assessment
2 Process Evaluation
3 Impact Evaluation
Analysis of Beneficiary Needs
Analysis of Program
Implementation
Measures How Much the Program
Impacts Beneficiaries
1. Services being delivered?
2. Clients satisfied? 3. Services reaching
target population?
1. People washing their hands more?
2. Diarrhea gone down?
3. Health of mothers and children improved?
4. Less poverty?
1. Why are we doing the project? 2. Who is the target population? 3. Why do we think this approach will work in this context?
![Page 9: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/9.jpg)
How to Go from Program to Impact?
WASH PROGRAM • Inputs: people,
money supplies
• Activities: Construction
• Output: Handwashing Facilities
INTER-MEDIATE OUTCOMES
• People wash hands
• Less germs on hands
OUTCOMES
• Lower diarrhea and infant mortality
• Improved child well-being
![Page 10: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/10.jpg)
Using impact evaluations to truly determine the impact of the program
![Page 11: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/11.jpg)
Does promoting hand-washing work?
From Meena's Three Wishes, “Meena tackles the issues of hygiene and sanitation.” UNICEF
![Page 12: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/12.jpg)
Outcome: Children’s
health
Time
Intervention
Impact?
Impact of the program
Before After
Treatment villages
![Page 13: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/13.jpg)
What would have happened without the program?
![Page 14: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/14.jpg)
Time
Intervention
Counterfactual
Actual Impact
= Treatment
- Control
Impact of the program Control needs to be a good counterfactual
Before After
Treatment villages
Control villages
Outcome: Children’s
health
![Page 15: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/15.jpg)
Non-random program selection leads to comparisons of apples and oranges
![Page 16: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/16.jpg)
Non-random program selection leads to comparisons of apples and oranges
![Page 17: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/17.jpg)
Randomization creates the right comparison group to measure impact
![Page 18: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/18.jpg)
Randomization creates the right comparison group to measure impact
![Page 19: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/19.jpg)
Impact Evaluation
• Why
• What
• How
![Page 20: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/20.jpg)
Pre-School Program in Mozambique • 76 communities identified to receive the program • 30 randomly assigned to the treatment. • 46 acted as the control. • Government so impressed that it expanded it to 600
communities
Randomize when there are not enough resources to reach every village
![Page 21: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/21.jpg)
…or randomize who receives the program first
Sanitation program in Indonesia
• 80 villages received the program in phase one.
• Remaining 80 villages in phase two
![Page 22: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/22.jpg)
Learn more with multiple treatments
Teacher bonus program in India • 100 schools – individual bonus • 100 schools – group bonus • 100 schools – cash grant • 100 control
![Page 23: Overview: Why, What and How of Impact Evaluation · 12/9/2014 · Overview: Why, What and How of Impact Evaluation Impact Evaluation Workshop Debre Zeit, Ethiopia Joost de Laat,](https://reader035.vdocuments.mx/reader035/viewer/2022071003/5fc03d4721a37567045aad5d/html5/thumbnails/23.jpg)
Now is your chance to integrate an impact evaluation in your own program