going beyond effectiveness: using impact evaluation …€¦ · 09/12/2014 · using impact...
TRANSCRIPT
![Page 1: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/1.jpg)
GOING BEYOND EFFECTIVENESS:
USING IMPACT EVALUATION TO
IMPROVE PROGRAM DESIGN
Impact Evaluation Workshop
Debre Zeit, Ethiopia
Reimar Macaranas
December 9 – 12, 2014
![Page 2: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/2.jpg)
NETWORKING ACTIVITY
Impact Evaluation Workshop
Debre Zeit, Ethiopia
Reimar Macaranas
December 9 – 12, 2014
![Page 3: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/3.jpg)
Getting to know your fellow participants…
1. Find someone you don’t know
(and yes World Bank staff count)
2. Learn that person’s name and how they got their name
3. Share where you work and a project or accomplishment
you are passionate about or proud of
4. Share a funny or interesting fact about yourself
![Page 4: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/4.jpg)
GOING BEYOND EFFECTIVENESS:
USING IMPACT EVALUATION TO
IMPROVE PROGRAM DESIGN
Impact Evaluation Workshop
Debre Zeit, Ethiopia
Reimar Macaranas
December 9 – 12, 2014
![Page 5: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/5.jpg)
Using Impact Evaluations to Improve Program Design
Objective:
Impact Evaluation can be used as a tool for informed and
evidence-base decision making.
Example: Designing Conditional Cash Transfer Programs
![Page 6: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/6.jpg)
Going Beyond Effectiveness
5
Did it work?
How did it work?
Why did it work?
Which design
feature worked
best?
![Page 7: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/7.jpg)
6
Public Works Program
Household Assets
Building Program
Ethiopia’s Productive Safety Net Program
![Page 8: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/8.jpg)
7
Public Works + Asset
Building Program
vs
Public Works Program
Question: What Worked Best?
![Page 9: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/9.jpg)
8
General Findings:
The Program Matters
Food Security
Livestock Assets
Asset Shedding
Biggest gains are found when combining the
programs
Ethiopia’s Productive Safety Net Program
![Page 10: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/10.jpg)
9
Over a 12 Month Period
Beneficiaries with Both Programs (compared to PW only):
1.53 months of additional food security
147 kg of increased output of grain
per household per year
Ethiopia’s Productive Safety Net Program
![Page 11: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/11.jpg)
Using Impact Evaluations to Improve Program Design
Objective:
Impact Evaluation can be used as a tool for informed and
evidence-base decision making.
We can learn how to design programs to make them more effective
![Page 12: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/12.jpg)
Conditional Cash Transfers
Source: World Bank, Fizbein & Schady 2009
![Page 13: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/13.jpg)
Conditional Cash Transfers
Source: World Bank, Fizbein & Schady 2009
![Page 14: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/14.jpg)
Tanzania Social Action Fund
![Page 15: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/15.jpg)
Tanzania Social Action Fund
March 2014: Doubled enrollment to 100,000 poor
households
2014-2016: Expected enrollment of 900,000 poor
households
5.5 million citizens
![Page 16: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/16.jpg)
Program Design Questions
What types of questions related to program design do we
need to be asking?
(This is the audience participation portion of the presentation)
![Page 17: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/17.jpg)
Program Design Questions
What types of questions related to program design do we
need to be asking?
When?
Who?
How much?
How often?
How to transfer?
![Page 18: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/18.jpg)
Learning About Program Design
Conditional Cash Transfers
What should the size of the transfer be?
Who should receive the transfer?
Whom to target?
How to target?
When to distribute the money?
Should it be conditional?
Conditional on what?
![Page 19: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/19.jpg)
What should the size of the transfer be?
The size required to improve attendance is small, possibly only
2-4% of income.
Increasing the size of the transfer does not increase the impact on
attendance
School Attendance - Percentage Point Increase
25 25
8 8
0
10
20
30
$45 $60 $5 $15
Cambodia Malawi
Source: Filmer and Schady (2009), Baird et al., (2010)
![Page 20: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/20.jpg)
When to distribute the money?
• Distribute portion of the money when parents have to
pay for enrollment
Makes it easier for parents to save.
• Distribute portion of money when students reach
university
Incentive to perform better in school.
19
Timing matters.
![Page 21: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/21.jpg)
Timing Matters
20
Bogota, Colombia - Conditional Subsidies for School
Attendance program
Treatment Groups
Standard CCT
Design
Savings Treatment
Tertiary Treatment
• 2/3 Monthly Payment
• Remaining funds
given at enrollment
• Same total cost
• 2/3 Monthly Payment
• Remaining funds
given at enrollment
• Large subsidy for
tertiary
![Page 22: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/22.jpg)
Timing Matters
Savings Treatment vs. Standard CCT Design
21
3.3 3.6
0
1
2
3
4
5
Secondary Tertiary
School Enrollment - Percentage Point Increase
The enrollment gain is larger if money is dispersed at the time where
parents have to pay for enrolment.
Source: Barrera-Osorio et al., (2008)
![Page 23: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/23.jpg)
Timing Matters
Tertiary Treatment vs. Standard CCT Design
22
School Enrollment - Percentage Point Increase
The enrollment gain is larger if money is dispersed at the time where
parents have to pay for enrolment.
3.3
46
0
10
20
30
40
50
Secondary Tertiary
Source: Barrera-Osorio et al., (2008)
![Page 24: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/24.jpg)
Should it be conditional?
Conditions good for increasing attendance and
enrollment.
However, unconditional transfers are better at reducing
drop-outs for teenagers.
23
Conditions should depend on the age and gender of
the pupil.
![Page 25: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/25.jpg)
Should it be conditional?
Conditional Cash Transfer vs Control Group
24
School Attendance - Percentage Point Increase
8 9.2
0
2
4
6
8
10
2009 2010
Conditions good for increasing attendance and enrollment
Source: Baird et al., (2010)
![Page 26: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/26.jpg)
Should it be conditional?
Unconditional Cash Transfer vs Control Group
25
Marriage & Pregnancy - Percentage Point Decrease
Conditions not as good for reducing main sources for female
drop-out rates
8.6 7.7
0
2
4
6
8
10
Marriage Rate Pregnancy Rate
Source: Baird et al., (2010)
![Page 27: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/27.jpg)
26
Using Impact Evaluation for Program Design
Solutions Within Reach
Explore – Ask the right research
questions
Test – Employ rigorous methods
Improve – Go beyond effectiveness
![Page 28: GOING BEYOND EFFECTIVENESS: USING IMPACT EVALUATION …€¦ · 09/12/2014 · USING IMPACT EVALUATION TO IMPROVE PROGRAM DESIGN Impact Evaluation Workshop Debre Zeit, Ethiopia Reimar](https://reader035.vdocuments.mx/reader035/viewer/2022070112/60520edeac581a6e182e6680/html5/thumbnails/28.jpg)
27
Using Impact Evaluation for Program Design
Solutions Within Reach
Thank You!