world bank group - designing a randomized control trial: how to randomize? · 2014. 12. 9. ·...
TRANSCRIPT
![Page 1: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/1.jpg)
Designing a Randomized Control Trial: How to Randomize?
• Impact Evaluation Workshop • Debre Zeit, Ethiopia
Donald Baum, World Bank
December 9 – 12, 2014
![Page 2: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/2.jpg)
What to Consider When Randomizing
1. Opportunities to Randomize
2. What program features to evaluate?
3. Level of randomization
4. Should you evaluate?
![Page 3: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/3.jpg)
Video: Opportunities to Randomize
![Page 4: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/4.jpg)
Basic setup of a randomized evaluation Randomization methods Individuals/Clusters Multiple/Cross-cutting treatments
Random
Assignment
Treatment
group
Control group
Balance Check
Endline Survey
Measure Impact
Internal Validity
Target
Population
Not in
evaluation
Evaluation
Sample
Random Sampling
External Validity
Baseline Survey
![Page 5: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/5.jpg)
Randomly sample from area of interest
Random sampling and random assignment
![Page 6: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/6.jpg)
Randomly sample from area of interest
Randomly assign to treatment and control
Random sampling and random assignment
Randomly sample from both treatment and control
![Page 7: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/7.jpg)
Random sampling and random assignment
2006
Income per person, per
month, shillings
1000
500
0 Treat Compare
1457 1442
![Page 8: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/8.jpg)
Opportunities to Randomize when Resources are Limited
![Page 9: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/9.jpg)
Lottery when Oversubscription
More are eligible than can be served by the program
![Page 10: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/10.jpg)
Lottery for individual-level oversubscription: Youth Support – Uganda
• 535 eligible young men applied
• Grants available for 265
![Page 11: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/11.jpg)
Targeted Lottery
Exclude some from the lottery
Include others
![Page 12: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/12.jpg)
Targeted Lottery: Uganda Orphans
• 535 eligible young men applied • 25 orphaned youth • Grants available for 265
• 25 orphaned youth
• 510 randomly assigned to remaining 240 slots
![Page 13: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/13.jpg)
Randomly vary when to receive program
Time During Before
evaluation period
After
![Page 14: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/14.jpg)
Randomly vary when to receive the program: Tanzania Social Action Fund
• 2012: 50,000 poor households
• 2014: 100,000 poor households
• 2016: 900,000 poor households
![Page 15: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/15.jpg)
Randomly vary program design
![Page 16: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/16.jpg)
Testing multiple program features Basic Setup Randomization methods Individuals/Clusters
Target
Population
Not in
evaluation
Evaluation
Sample
Total
Population
Random
Assignment
Treatment
Group 3
Control Group
Treatment
Group 2
Treatment
Group 1
![Page 17: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/17.jpg)
Summary: opportunities to randomize
How to determine method of randomization?
1. Resources: Sufficient program resources to reach all eligible beneficiaries?
1. Targeting rules: Who is eligible for the program? Is program participation restricted, widely open, some combination?
2. Timing: How are potential beneficiaries enrolled in the program – all at once or in phases over time?
![Page 18: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/18.jpg)
What to Consider When Randomizing
1. Opportunities to Randomize
2. What program features to evaluate?
3. Level of randomization
4. Should you evaluate?
![Page 19: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/19.jpg)
Which level of randomization?
INDIVIDUAL unit:
CLUSTER unit:
e.g. pupil, mother, father, household
e.g. class, health clinic, village
![Page 20: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/20.jpg)
Which level of randomization?
Individual or Cluster?
![Page 21: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/21.jpg)
Which level of randomization?
Individual
![Page 22: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/22.jpg)
Which level of randomization? Cluster (group)
![Page 23: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/23.jpg)
How to determine level of randomization? Factors that matter
1. Unit of Implementation
2. Spillovers
3. Sampling unit and statistical power
4. Ethics and Fairness
![Page 24: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/24.jpg)
Which level of randomization?
Pupil
School
Community
Class
Unit of randomization should be at least at the level of the unit of program intervention: Example: IF program targets classes -> Randomization may be at class, school, community level
![Page 25: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/25.jpg)
Which Level of Randomization?
A. Unit of Implementation
B. Spillovers: individual in the control also benefits from the treatment
C. Sampling and Cost
D. Ethics and Fairness
Individual tutoring
Spillover: program pupils talk with class mates
![Page 26: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/26.jpg)
With spillovers over/under-estimate impact
Control
True Impact Spillover
Measured Impact
Treatment
Impact
![Page 27: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/27.jpg)
Higher unit of randomization reduces risk of spillovers
Example: randomize at school level to reduce interaction between treatment and control pupils
School
Pupil
![Page 28: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/28.jpg)
Question: Is there risk of spillovers in the following programs?
• Example 1: Farmer training program (individual randomization)
• Neighboring farmers may also learn about the new methods.
• Measured program impact = farm output in treatment – farm output in comparison
• As a result, measured program impact is lower than the real program impact
![Page 29: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/29.jpg)
Question: Is there risk of spillovers in the following programs?
• Example 2: HIV-AIDS education program (individual randomization)
• People who attend education program will tell their friends
• Measured program impact = HIV-aids awareness in treatment – HIV-aids awareness in comparison
• As a result, measured impact is lower than the real impact
![Page 30: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/30.jpg)
Which Level of Randomization?
A. Unit of Implementation
B. Spillovers
C. Sampling unit and statistical power
D. Ethics and Fairness
![Page 31: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/31.jpg)
Which Level of Randomization?
A. Unit of Implementation
B. Spillovers
C. Sampling unit and statistical power
D. Ethics and Fairness
![Page 32: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/32.jpg)
Sometimes perception of unfairness among control if randomizing peers, community
members etc. at individual level
![Page 33: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/33.jpg)
Less so if all in same group or village are in the control
![Page 34: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/34.jpg)
Which Level of Randomization?
A. Unit of Implementation
B. Spillovers
C. Sampling unit and statistical power
D. Ethics and Fairness
Randomize at the smallest feasible level
![Page 35: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/35.jpg)
Some Examples
![Page 36: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/36.jpg)
At what level should we randomize for a school deworming program?
A. School
B. Teacher
C. Classroom
D. Clinic
E. Community
F. Pupil
![Page 37: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/37.jpg)
At what level should we randomize for a conditional cash transfer program?
A. Individual
B. Household
C. Village/Community
![Page 38: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/38.jpg)
At what level should we randomize for a teacher training program?
A. School
B. Teacher
C. Classroom
D. Pupil
![Page 39: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/39.jpg)
At what level should we randomize for a small-business start-up grant?
A. Individual
B. Household
C. Clinic
D. School
E. Village/Community
![Page 40: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/40.jpg)
What to Consider When Randomizing
1. Opportunities to Randomize
2. What program features to evaluate?
3. Level of randomization
4. Should you evaluate?
![Page 41: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/41.jpg)
Does this need an Impact Evaluation?
Maximize learning
Consider 1. Do we know the answer already? 2. Is it policy relevant?
a) Is this a program that government could scale up if it is successful?
b) Is it strategically relevant for reducing poverty? 3. Does it add to knowledge? 4. Is the program ready for an impact evaluation? 5. Are there program design options that needs to be
compared? 6. Is there an resource constraint that provides an
opportunity for randomization?
![Page 42: World Bank Group - Designing a Randomized Control Trial: How to Randomize? · 2014. 12. 9. · group Control group Balance Check Endline SurveySample Measure Impact Internal Validity](https://reader033.vdocuments.mx/reader033/viewer/2022051808/600a350cbb9e6a762736ab25/html5/thumbnails/42.jpg)
Now is your chance to integrate an impact evaluation in your own program