translating research into action what is randomized evaluation? why randomize? j-pal south asia,...
TRANSCRIPT
![Page 1: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/1.jpg)
TRANSLATING RESEARCH INTO ACTION
What is Randomized Evaluation?Why Randomize?
J-PAL South Asia, April 29, 2011
![Page 2: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/2.jpg)
Life Cycle of an Evaluation
< Needs assessment > <~~~~~~~~~~~ Process Evaluation ~~~~~~~~~~~~> <~~~~~~~~~ Impact Evaluation ~~~~~~~~>Other types of Evaluations: Review / Cost-benefit analysis / Cost-effectiveness Analysis
![Page 3: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/3.jpg)
Balsakhi Program
Case 2: Remedial Education in IndiaEvaluating the Balsakhi Program
Incorporating random assignment into the program
Case 2: Remedial Education in IndiaEvaluating the Balsakhi Program
Incorporating random assignment into the program
![Page 4: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/4.jpg)
• Class sizes are large• Many children in 3rd and 4th standard are not
even at the 1st standard level of competency• Social distance between teacher and some of
the students is large
What was the issue?
![Page 5: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/5.jpg)
• Hire local women (balsakhis) from the community
• Train them to teach remedial competencies– Basic literacy, numeracy
• Identify lowest performing 3rd and 4th standard students– Take these students out of class (2 hours/day)– Balsakhi teaches them basic competencies
Proposed solution
Case 2: Remedial Education in IndiaEvaluating the Balsakhi Program
Incorporating random assignment into the program
Case 2: Remedial Education in IndiaEvaluating the Balsakhi Program
Incorporating random assignment into the program
![Page 6: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/6.jpg)
Pros• Reduced class size• Teaching at appropriate
level• Reduced social distance
• Improved learning for lower-performing students
• Improved learning for higher-performers
Cons• Less qualified• Teacher resentment• Reduced interaction with
higher-performing peers
Possible outcomes
![Page 7: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/7.jpg)
• We want to look at impact• What do we do?
Study design
![Page 8: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/8.jpg)
What is the impact of the balsakhi?
What would have happened without the balsakhi program?
![Page 9: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/9.jpg)
• Look at average change in test scores over the school year for the balsakhi children
Pre-post (Before vs. After)
![Page 10: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/10.jpg)
What was the impact of the balsakhi program?
Before vs. AfterImpact = 26.42 points?
75
50
25
0
02002 2003
26.42 points?
![Page 11: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/11.jpg)
Impact: What is it?
Time
Prim
ary
Out
com
e
Impact
Counterfactual
Intervention
![Page 12: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/12.jpg)
Impact: What is it?
Time
Prim
ary
Out
com
e
ImpactCounterfactual
Intervention
![Page 13: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/13.jpg)
Impact: What is it?
Time
Prim
ary
Out
com
e
ImpactCounterfactual
Intervention
![Page 14: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/14.jpg)
Impact is defined as a comparison between:
1. the outcome some time after the program has been introduced
2. the outcome at that same point in time had the program not been introduced
the ”counterfactual”
14
How to measure impact?
![Page 15: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/15.jpg)
1. Randomized Experiments • Also known as:
– Random Assignment Studies – Randomized Field Trials – Social Experiments– Randomized Controlled Trials (RCTs)– Randomized Controlled Experiments
15
Impact evaluation methods
![Page 16: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/16.jpg)
2. Non- or Quasi-Experimental Methods a. Pre-Postb. Simple Differencec. Differences-in-Differencesd. Multivariate Regressione. Statistical Matchingf. Interrupted Time Seriesg. Instrumental Variablesh. Regression Discontinuity
16
Impact evaluation methods
![Page 17: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/17.jpg)
• There are more sophisticated non-experimental methods to estimate program impacts:– Multivariable Regression– Matching– Instrumental Variables– Regression Discontinuity
• These methods rely on being able to “mimic” the counterfactual under certain assumptions
• Problem: Assumptions are not testable
Other Methods
![Page 18: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/18.jpg)
• We constructed results from the different ways of estimating the impacts using the data from the schools that got a balsakhi
1. Pre – Post (Before vs. After)2. Simple difference3. Difference-in-difference4. Other non-experimental methods5. Randomized Experiment
Methods to estimate impacts
![Page 19: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/19.jpg)
TRANSLATING RESEARCH INTO ACTION
II – What is a randomized experiment?
The Balsakhi Example
povertyactionlab.org
![Page 20: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/20.jpg)
The basics
Start with simple case: • Take a sample of schools• Randomly assign them to either:
Treatment Group – is offered treatment Control Group - not allowed to receive treatment
(during the evaluation period)
20
![Page 21: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/21.jpg)
• Suppose we evaluated the balsakhi program using a randomized experiment
• QUESTION #1: What would this entail? How would we do it?
• QUESTION #2: What would be the advantage of using this method to evaluate the impact of the balsakhi program?
21
6 – Randomized Experiment
Source: www.theoryofchange.org
![Page 22: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/22.jpg)
How to Randomize, Part I - 22
Random assignment in Vadodara
2006
Baseline Test Scores
20
10
0Treat Compare
28 28
![Page 23: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/23.jpg)
Key advantage of experiments
Because members of the groups (treatment and control) do not differ systematically at the outset of the experiment,
any difference that subsequently arises between them can be attributed to the program rather than to other factors.
23
![Page 24: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/24.jpg)
• Pratham had received funding to introduce balsakhis in all schools
• Principals wanted balsakhis in their school• Teachers wanted balsakhis in their class• If denied a balsakhi, they may block data
collection efforts
Constraints of design
![Page 25: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/25.jpg)
Random assignment
![Page 26: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/26.jpg)
Round 1Red: Std. 3Blue: Std. 4
Round 1Red: Std. 3Blue: Std. 4
Rotation design
Round 2Std 3 from Round 1 Std 4 in Round 2——————————————————————————
Std 4 from Round 1 Std 3 in Round 2
![Page 27: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/27.jpg)
Round 1Red: Std. 3Blue: Std. 4
ComparisonRed: TreatmentBlue: ControlStd 3Std 4
Rotation design
![Page 28: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/28.jpg)
The counterfactual:
School Groups
Year 1 (2001-2002) Year 2 (2002-2003)
Grade 3 Grade 4 Grade 3 Grade 4
Group A Balsakhi No Balsakhi No Balsakhi Balsakhi
Group B No Balsakhi Balsakhi Balsakhi No Balsakhi
![Page 29: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/29.jpg)
Key steps in conducting an experiment
1. Design the study carefully
2. Collect baseline data
3. Randomly assign people to treatment or control
4. Verify that assignment looks random
5. Monitor process so that integrity of experiment is not compromised
29
![Page 30: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/30.jpg)
Key steps in conducting an experiment (cont.)
6. Collect follow-up data for both the treatment and control groups
7. Estimate program impacts by comparing mean outcomes of treatment group vs. mean outcomes of control group.
8. Assess whether program impacts are statistically significant and practically significant.
30
![Page 31: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/31.jpg)
Impact of Balsakhi - Summary
Method Impact Estimate
(1) Pre-post 26.42*
(2) Simple Difference -5.05*
(3) Difference-in-Difference 6.82*
(4) Regression 1.92
*: Statistically significant at the 5% level
![Page 32: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/32.jpg)
Impact of Balsakhi - Summary
Method Impact Estimate
(1) Pre-post 26.42*
(2) Simple Difference -5.05*
(3) Difference-in-Difference 6.82*
(4) Regression 1.92
(5)Randomized Experiment 5.87*
*: Statistically significant at the 5% level
![Page 33: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/33.jpg)
Bottom Line: Which method we use matters!
Impact of Balsakhi - Summary
Method Impact Estimate
(1) Pre-post 26.42*
(2) Simple Difference -5.05*
(3) Difference-in-Difference 6.82*
(4) Regression 1.92
(5)Randomized Experiment 5.87*
*: Statistically significant at the 5% level
![Page 34: TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia, April 29, 2011](https://reader030.vdocuments.mx/reader030/viewer/2022032707/56649e4b5503460f94b3f3d9/html5/thumbnails/34.jpg)
• This program improved average test score by nearly 10%
• The lowest-performing kids in balsakhi schools improved the most (relative to the control group)
• Highly cost effective: intervention cost less than Rs.100 per child and results far outweigh the cost
• Can be scalable, especially in light of RTE Act: remedial education
Results