sampling and sample size - mit opencourseware
TRANSCRIPT
![Page 1: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/1.jpg)
TRANSLATING RESEARCH INTO ACTION
Planning sample size forrandomized evaluations
Simone SchanerDartmouth College
povertyactionlab.org1
![Page 2: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/2.jpg)
• Why evaluate? What is evaluation?
• Outcomes, indicators and measuring impact
• Impact evaluation – why randomize
• How to randomize
• Sampling and sample size
• Implementing an evaluation
• Analysis and inference
Course Overview
2
![Page 3: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/3.jpg)
• How large does the sample need to be to“credibly” detect a given treatment effect?
• What does credibly mean?
• Randomization removes bias, but it does notremove noise
• But how large must “large” be?
3
Today’s Question
![Page 4: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/4.jpg)
• Estimation• Intro to the scientific method• Hypothesis testing• Statistical significance• Factors that influence power• Effect size• Sample size• Cluster randomized trials
Lecture Overview
4
![Page 5: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/5.jpg)
Estimation
The sample average is our estimate of the population average
PopulationWe wish to learn about this
SampleBut we only see this
5
![Page 6: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/6.jpg)
Accuracy: Estimate is Right OnAverage
Which sampling strategy will give us a more accurate estimate?6
![Page 7: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/7.jpg)
Precision: Estimate Has Low Variability
Which sampling strategy will give us a more precise estimate?7
![Page 8: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/8.jpg)
Precision: Estimate Has Low Variability
But what about a more accurate estimate?8
![Page 9: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/9.jpg)
• When we do estimation• Sample size allows us to say something aboutthe variability of our estimate
• But it doesn’t ensure that our estimate will beclose to the truth on average
RANDOMIZATION IS THE GOLD STANDARDBECAUSE IT ENSURES ACCURACY. We thencontrol precision with sample size.
Estimation
9
![Page 10: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/10.jpg)
Review: Random Sampling vs. RandomAssignment to Treatment
What happens if werandomly sample…
…But don’t randomlyassign treatment?
Will our estimate of the treatment effect be unbiased?10
![Page 11: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/11.jpg)
Accuracy versus Precision
truth
estimates
11
![Page 12: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/12.jpg)
Accuracy versus Precision
truth truth
truth truth
estimates
estimates
estimates
estimates
12
![Page 13: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/13.jpg)
• Does the scientific method apply to socialscience?
• The scientific method involves:– 1) proposing a hypothesis
– 2) designing experimental studies to test thehypothesis
• How do we test hypotheses?
Measuring Significance: ScientificMethod
13
![Page 14: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/14.jpg)
• We start with our hypothesis
• At the end of an experiment, we test ourhypothesis
• We compare the outcome of interest in thetreatment and the comparison groups.
14
Basic set up
![Page 15: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/15.jpg)
• In criminal law, most institutions follow therule: “innocent until proven guilty”
• The prosecutor wants to prove theirhypothesis that the accused person is guilty
• The burden is on the prosecutor to show guilt
• The jury or judge starts with the “nullhypothesis” that the accused person isinnocent
15
Hypothesis testing
![Page 16: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/16.jpg)
• In program evaluation, instead of“presumption of innocence,” the rule is:“presumption of insignificance”
• Policymaker’s hypothesis: the program improveslearning
• Evaluators approach experiments using thehypothesis:– “There is zero impact” of this program– Then we test this “Null Hypothesis” (H0)
• The burden of proof is on the program– Must show a statistically significant impact
16
Hypothesis testing
![Page 17: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/17.jpg)
• If our measurements show a difference betweenthe treatment and control group, our firstassumption is:– In truth, there is no impact (our H0 is still true)– There is some margin of error due to sampling– “This difference is solely the result of chance (randomsampling error)”
• We (still assuming H0 is true) then use statisticsto calculate how likely this difference is in factdue to random chance
Hypothesis testing
17
![Page 18: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/18.jpg)
Is this difference due to randomchance?
Control
Treatment
Probably… 18
![Page 19: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/19.jpg)
Is this difference due to randomchance?
Control
Treatment
Probably not…. 19
![Page 20: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/20.jpg)
• If it is very unlikely (less than a 5% probability)that the difference is solely due to chance:– We “reject our null hypothesis”
• We may now say:– “our program has a statistically significant impact”
Hypothesis testing: conclusions
20
![Page 21: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/21.jpg)
• Are we now 100 percent certain there is animpact?– No, we may be only 95% confident– And we accept that if we use that 5% threshold,this conclusion may be wrong 5% of the time
– That is the price we’re willing to pay since we cannever be 100% certain
– Because we can never see the counterfactual,We must use random sampling and randomassignment, and rely on statistical probabilities
Hypothesis testing: conclusions
21
![Page 22: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/22.jpg)
Example: Pratham Balsakhi(Vadodarda)
22
![Page 23: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/23.jpg)
Baseline test score data in Vadodara
•This was the distribution of test scores in the baseline. •The test was out of 100. •Some students did really well, most, not so well•Many actually scored zero
23
![Page 24: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/24.jpg)
Endline test scores
Was there an impact?
Now, look at the improvement. Very few scored zero, and many scored much closer to the 40-point range…
24
![Page 25: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/25.jpg)
Post‐test: control & treatment
Stop! That was the control group. The treatment group is green. 25
![Page 26: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/26.jpg)
Average difference: 6 points
This is the true difference between the 2 groups 26
![Page 27: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/27.jpg)
• Population: what we want to learn about• Sample: what we see
– How many children would we need to randomlysample to detect that the difference between thetwo groups is statistically significantly differentfrom zero?OR
– How many children would we need to randomlysample to approximate the true difference withsufficient precision?
Population versus Sample
27
![Page 28: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/28.jpg)
Testing statistical significance
What’s the probability that the 6 point difference is due to chance?What’s the probability that the 6 point difference is due to chance?28
![Page 29: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/29.jpg)
That probability depends on samplesize (here: N=2)
0 6Treatment Mean – Control Mean
Difference under null
Observed difference
N=2
29
![Page 30: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/30.jpg)
“Significance level” (5%)
Difference under null
Observed difference
N=2
0 6Treatment Mean – Control Mean
Critical region30
![Page 31: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/31.jpg)
“Significance level” (5%)
0 6Treatment Mean – Control Mean equals 5% of
this total area
Difference under null
Observed difference
N=2
31
![Page 32: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/32.jpg)
Significance: Sample size = 8
0 6Treatment Mean – Control Mean
Difference under null
Observed difference
N=8
32
![Page 33: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/33.jpg)
Significance: Sample size = 18
0 6Treatment Mean – Control Mean
Difference under null
Observed difference
N=18
33
![Page 34: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/34.jpg)
Significance: Sample size = 100
0 6Treatment Mean – Control Mean
Difference under null
Observed difference
34
![Page 35: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/35.jpg)
Significance: Sample size = 6,000
0 6Treatment Mean – Control Mean
Difference under null
Observed difference
N=6,000
35
![Page 36: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/36.jpg)
• What if the probability is greater than 5%?– We can’t reject our null hypothesis– Are we 100 percent certain there is no impact?
• No, it just didn’t meet the statistical threshold toconclude otherwise
– Perhaps there is indeed no impact– Or perhaps there is impact,
• But not enough sample to detect it most of the time• Or we got a very unlucky sample this time• How do we reduce this error?
Hypothesis testing: conclusions
POWER! 36
![Page 37: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/37.jpg)
• When we use a “95% confidence interval”
• How frequently will we “detect” effectiveprograms?
• That is Statistical Power
Hypothesis testing: conclusions
37
![Page 38: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/38.jpg)
YOU CONCLUDEEffective No Effect
Effective ☺Type II Error (low power)
THE TRUTH
No Effect
Type I Error(5% of the time) ☺
38
Hypothesis testing: 95% confidence
![Page 39: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/39.jpg)
• How frequently will we “detect” effectiveprograms?
Power:
39
![Page 40: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/40.jpg)
1. Variance– The more “noisy” it is to start with, the harder it is tomeasure effects
2. Effect Size to be detected– The more fine (or more precise) the effect size wewant to detect, the larger sample we need
– Smallest effect size with practical / policysignificance?
3. Sample Size– The more children we sample, the more likely we areto obtain the true difference
40
Power: main ingredients
![Page 41: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/41.jpg)
Variance
41
![Page 42: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/42.jpg)
• There is very little we can do to reduce thenoise
• The underlying variance is what it is
• We can try to “absorb” variance:– using a baseline
– controlling for other variables
Variance
42
![Page 43: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/43.jpg)
• To calculate statistical significance we startwith the “null hypothesis”:
• To think about statistical power, we need topropose a secondary hypothesis
Effect Size
43
![Page 44: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/44.jpg)
• The following is an example…
2 Hypotheses & “significance level”
44
![Page 45: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/45.jpg)
Null Hypothesis: assume zero impact
“Impact = 0” There’s a sampling distribution around that. 45
![Page 46: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/46.jpg)
Effect Size: 1 “standard deviation”
We hypothesize another possible “true effect size”
1 StandardDeviation
46
![Page 47: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/47.jpg)
Effect Size: 1 “standard deviation”
And there’s a new sampling distribution around that
1 StandardDeviation
47
![Page 48: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/48.jpg)
Effect Size: 3 standard deviations
The less overlap the better…
3 StandardDeviations
48
![Page 49: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/49.jpg)
Significance level:reject H0 in critical region
49
![Page 50: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/50.jpg)
True effect is 1 SD
50
![Page 51: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/51.jpg)
Power: when is H0 rejected?
51
![Page 52: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/52.jpg)
Power: 26%If the true impact was 1SD…
The Null Hypothesis would be rejected only 26% of the time 52
![Page 53: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/53.jpg)
Power: if we change the effect size?
53
![Page 54: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/54.jpg)
Power: assume effect size = 3 SDs
54
![Page 55: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/55.jpg)
Power: 91%
The Null Hypothesis would be rejected 91% of the time 55
![Page 56: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/56.jpg)
• What is the smallest effect that should justifythe program being adopted?
• If the effect is smaller than that, it might aswell be zero: we are not interested in provingthat a very small effect is different from zero
• In contrast, if any effect larger than that wouldjustify adopting this program: we want to beable to distinguish it from zero
Picking an effect size
DO NOT USE: “Expected” effect size 56
![Page 57: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/57.jpg)
• How large an effect you can detect with agiven sample depends on how variable theoutcome is.
• The Standardized effect size is the effect sizedivided by the standard deviation of theoutcome
• Common effect sizes
57
Standardized effect sizes
![Page 58: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/58.jpg)
An effect size of…
Is considered… …and it means that…
0.2 Modest The average member of the treatment group had a better outcome than the 58th
percentile of the control group
0.5 Large The average member of the treatment group had a better outcome than the 69th
percentile of the control group
0.8 VERY Large The average member of the treatment group had a better outcome than the 79th
percentile of the control group
Standardized effect size
58
![Page 59: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/59.jpg)
• You should not alter the effect size to achievepower
• The effect size is more of a policy question• One variable that can affect effect size is take‐up!
– If your job training program increases income by 20%– But only ½ of the people in your treatment groupparticipate
– You need to adjust your impact estimate accordingly• From 20% to 10%
• So how do you increase power?
Effect Size: Bottom Line
Try: Increasing the sample size 59
![Page 60: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/60.jpg)
• Increasing sample size reduces the “spread” ofour bell curve
• The more observations we randomly pull, themore likely we get the “true average”
Sample size
60
![Page 61: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/61.jpg)
Power: Effect size = 1SD,Sample size = 1
61
![Page 62: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/62.jpg)
Power: Sample size = 4
62
![Page 63: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/63.jpg)
Power: 64%
63
![Page 64: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/64.jpg)
Power: Sample size = 9
64
![Page 65: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/65.jpg)
Power: 91%
65
![Page 66: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/66.jpg)
• In this example:– a sample size of 9 gave us good power
– But the effect size we used was very large (1 SD)
Sample size
66
![Page 67: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/67.jpg)
• When planning an evaluation, with some preliminaryresearch we can calculate the minimum sample weneed to get to.
• A power of 80% tells us that, in 80% of theexperiments of this sample size conducted in thispopulation, if Ho is in fact false (e.g. the treatmenteffect is not zero), we will be able to reject it.
• The larger the sample, the larger the power.
• Common Power used: 80%, 90%
67
Calculating power
![Page 68: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/68.jpg)
• You want to know how close the upcomingnational elections will be
• Method 1: Randomly select 50 people fromentire Indian population
• Method 2: Randomly select 5 families, and askten members of each family their opinion
68
Clustered design: intuition
![Page 69: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/69.jpg)
• If the response is correlated within a group,you learn less information from measuringmultiple people in the group
• It is more informative to measure unrelatedpeople
• Measuring similar people yields lessinformation
69
Clustered design: intuition
![Page 70: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/70.jpg)
• Cluster randomized trials are experiments inwhich social units or clusters rather thanindividuals are randomly allocated tointervention groups
• The unit of randomization (e.g. the school) isbroader than the unit of analysis (e.g.students)
• That is: randomize at the school level, but usechild‐level tests as our unit of analysis
70
Clustered design
![Page 71: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/71.jpg)
• The outcomes for all the individuals within aunit may be correlated
• We call ρ (rho) the correlation between theunits within the same cluster
71
Consequences of clustering
![Page 72: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/72.jpg)
Values of r (rho)
• Like percentages, ρmust be between 0 and 1
• When working with clustered designs, a lower ρ is more desirable
• It is sometimes low, 0, .05, .08, but can be high:0.62
72
Madagascar Math + Language 0.5
Busia, Kenya Math + Language 0.22
Udaipur, India Math + Language 0.23
Mumbai, India Math + Language 0.29
Vadodara, India Math + Language 0.28
Busia, Kenya Math 0.62
![Page 73: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/73.jpg)
Some examples of sample size
Study # ofinterventions(+ Control)
Total Number ofClusters
Total Sample Size
Women’s Empowerment 2 Rajasthan: 100West Bengal: 161
1996 respondents2813 respondents
Pratham Read India 4 280 villages 17,500 children
Pratham Balsakhi 2 Mumbai: 77 schoolsVadodara: 122schools
10,300 children12,300 children
Kenya Extra TeacherProgram
8 210 schools 10,000 children
Deworming 3 75 schools 30,000 children
Bednets 5 20 health centers 545 women
73
![Page 74: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/74.jpg)
74
Implications for design and analysis
• Analysis: The standard errors will need to beadjusted to take into account the fact that theobservations within a cluster are correlated.
• Adjustment factor (design effect) for given totalsample size, clusters of size m, intra‐clustercorrelation of r, the size of smallest effect we candetect increases by compared to anon‐clustered design
• Design: We need to take clustering into accountwhen planning sample size
![Page 75: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/75.jpg)
• If experimental design is clustered, we nowneed to consider ρ when choosing a samplesize (as well as the other effects)
• It is extremely important to randomize anadequate number of groups
• Often the number of individuals within groupsmatter less than the total number of groups
75
Implications
![Page 76: Sampling and sample size - MIT OpenCourseWare](https://reader030.vdocuments.mx/reader030/viewer/2022020913/6203496e24f6b61e9c663e2a/html5/thumbnails/76.jpg)
MIT OpenCourseWare http://ocw.mit.edu
Resource: Abdul Latif Jameel Poverty Action Lab Executive Training: Evaluating Social Programs Dr. Rachel Glennerster, Prof. Abhijit Banerjee, Prof. Esther Duflo
The following may not correspond to a particular course on MIT OpenCourseWare, but has been provided by the author as an individual learning resource.
For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.