Download - MEASURING IMPACT
![Page 1: MEASURING IMPACT](https://reader036.vdocuments.mx/reader036/viewer/2022062411/56816726550346895ddbb9c8/html5/thumbnails/1.jpg)
www.worldbank.org/hdchiefeconomist
The World Bank
Human Development
Network
Spanish Impact
Evaluation Fund
![Page 2: MEASURING IMPACT](https://reader036.vdocuments.mx/reader036/viewer/2022062411/56816726550346895ddbb9c8/html5/thumbnails/2.jpg)
MEASURING IMPACTImpact Evaluation Methods
for Policy Makers
This material constitutes supporting material for the "Impact Evaluation in Practice" book. This additional material is made freely but please acknowledge its use as follows: Gertler, P. J.; Martinez, S., Premand, P., Rawlings, L. B. and Christel M. J. Vermeersch, 2010, Impact Evaluation in Practice: Ancillary Material, The World Bank, Washington DC (www.worldbank.org/ieinpractice). The content of this presentation reflects the views of the authors and not necessarily those of the World Bank.
![Page 3: MEASURING IMPACT](https://reader036.vdocuments.mx/reader036/viewer/2022062411/56816726550346895ddbb9c8/html5/thumbnails/3.jpg)
1Causal Inference
Counterfactuals
False CounterfactualsBefore & After (Pre &
Post)Enrolled & Not Enrolled (Apples &
Oranges)
![Page 4: MEASURING IMPACT](https://reader036.vdocuments.mx/reader036/viewer/2022062411/56816726550346895ddbb9c8/html5/thumbnails/4.jpg)
2IE Methods Toolbox
Randomized Assignment
Discontinuity Design
Diff-in-Diff
Randomized Offering/Promotion
Difference-in-Differences
P-Score matchingMatching
![Page 5: MEASURING IMPACT](https://reader036.vdocuments.mx/reader036/viewer/2022062411/56816726550346895ddbb9c8/html5/thumbnails/5.jpg)
Choosing your IE method(s)
Prospective/Retrospective Evaluation?
Eligibility rules and criteria?
Roll-out plan (pipeline)?
Is the number of eligible units larger than available resources at a given point
in time?
o Poverty targeting?o Geographic
targeting?
o Budget and capacity constraints?
o Excess demand for program?
o Etc.
Key information you will need for identifying the right method for your program:
![Page 6: MEASURING IMPACT](https://reader036.vdocuments.mx/reader036/viewer/2022062411/56816726550346895ddbb9c8/html5/thumbnails/6.jpg)
Choosing your IE method(s)
Best Design
Have we controlled for everything?
Is the result valid for everyone?
o Best comparison group you can find + least operational risk
o External validityo Local versus global
treatment effecto Evaluation results apply to
population we’re interested in
o Internal validityo Good comparison group
Choose the best possible design given the operational context:
![Page 7: MEASURING IMPACT](https://reader036.vdocuments.mx/reader036/viewer/2022062411/56816726550346895ddbb9c8/html5/thumbnails/7.jpg)
Progresa Policy Recommendation?
Note: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**). If significant at 10% level, we label impact with +
Impact of Progresa on Consumption (Y)
Case 1: Before & After 34.28**Case 2: Enrolled & Not Enrolled -4.15Case 3: Randomized Assignment 29.75**Case 4: Randomized Offering 30.4**Case 5: Discontinuity Design 30.58**Case 6: Difference-in-Differences 25.53**Case 7: Matching 7.06+
![Page 8: MEASURING IMPACT](https://reader036.vdocuments.mx/reader036/viewer/2022062411/56816726550346895ddbb9c8/html5/thumbnails/8.jpg)
2IE Methods Toolbox
Randomized Assignment
Discontinuity Design
Diff-in-Diff
Randomized Offering/Promotion
Difference-in-Differences
P-Score matchingMatching
Combinations of methods
![Page 9: MEASURING IMPACT](https://reader036.vdocuments.mx/reader036/viewer/2022062411/56816726550346895ddbb9c8/html5/thumbnails/9.jpg)
Choosing your methodTargeted
(Eligibility Cut-off)Universal
(No Eligibility Cut-off)Limited
Resources(Never Able to
Achieve Scale)
Fully Resourced
(Able to Achieve Scale)
Limited Resources
(Never Able to Achieve Scale)
Fully Resourced
(Able to Achieve Scale)
Phased Implementation Over Time
o Randomized Assignment
o Discontinuity Design
o Randomized Assignment (roll-out)
o Discontinuity Design
o Randomized Assignment
o Matching with Diff-in-Diff
o Randomized Assignment (roll-out)
o Matching with Diff-in-Diff
Immediate Implementati
on
o Randomized Assignment
o Discontinuity Design
o Randomized Promotion
o Discontinuity Design
o Randomized Assignment
o Matching with Diff-in-Diff
o Randomized Promotion
![Page 10: MEASURING IMPACT](https://reader036.vdocuments.mx/reader036/viewer/2022062411/56816726550346895ddbb9c8/html5/thumbnails/10.jpg)
“Remember
The objective of impact evaluation is to estimate the causal effect or impact of a program on outcomes of interest.
![Page 11: MEASURING IMPACT](https://reader036.vdocuments.mx/reader036/viewer/2022062411/56816726550346895ddbb9c8/html5/thumbnails/11.jpg)
“Remember
To estimate impact, we need to estimate the counterfactual. o what would have happened in the
absence of the program ando use comparison or control groups.
![Page 12: MEASURING IMPACT](https://reader036.vdocuments.mx/reader036/viewer/2022062411/56816726550346895ddbb9c8/html5/thumbnails/12.jpg)
“Remember
We have a toolbox with 5 methods to identify comparison groups.
![Page 13: MEASURING IMPACT](https://reader036.vdocuments.mx/reader036/viewer/2022062411/56816726550346895ddbb9c8/html5/thumbnails/13.jpg)
“Remember
Choose the best evaluation method that is feasible in the program’s operational context.