sharon wolf nyu abu dhabi additional insights summer training institute june 15, 2015 1
TRANSCRIPT
Individual-level Moderation in Cluster-Randomized Control Trials
Sharon WolfNYU Abu Dhabi Additional Insights Summer Training InstituteJune 15, 2015 1
Outline
Conceptual overview
Analytic considerations Power/Minimum detectable differential
effects Cross-level interactions in MLM Centering variables
Recommendations and tips2
Conceptual Overview
When is the story in the subgroups?
3
Research Questions that Motivate Subgroup Analyses
• Guide questions about how to target resources most efficiently:• How widespread are the effects of an
intervention? • Is the intervention effective for a specific
subgroup?• Is the intervention effective for any subgroup?
• Exploratory* versus confirmatory subgroup findings
4
Individual-level Moderation versus Cluster-level Moderation
Two examples from welfare reform in the United States and the different policy implications.
Michalopoulos & Schwartz (2000) assessed two types of subgroups:
1. A range of person-level subgroups (e.g., education level, prior employment experience, and risk of depression).
2. The nature of the program and program office practices.
5
Characteristics that Define Individual-level Subgroups
• Characteristics believed to be related to the need for a particular intervention or the likelihood of benefiting from it.• Demographic characteristics – e.g., gender,
age, education level• Risk factors - past smoking, drug abuse,
severity of disease, poverty status• Combinations of characteristics – e.g.,
gender and age; cumulative levels of risk/risk index
6
Subgroups that are Exogenous versus Endogenous to the Intervention
Exogenous to the intervention: not affected by the intervention or correlated with its receipt (all pre-random assignment characteristics).
Endogenous to the intervention: affected by the intervention or correlated with its receipt (e.g., dosage of the intervention). Valid causal inferences much more difficult. Gambia: higher “dosage” (i.e., higher attendance)
more learning? Increased attendance could bring less advantaged
students into the intervention group, biasing the average treatment effect (ATE) downward.
7
Exploratory versus Confirmatory Subgroup Analyses
Exploratory subgroup analyses Provide a basis for hypothesis-generation Essential step in the scientific method Should be considered suggestive
Confirmatory subgroup analyses Appropriate basis for testing hypotheses Provide strong evidence if findings are: (a)
consistent with existing findings, (b) large enough magnitude to be meaningful, (c) robust.
Bloom & Michalopoulos, 20108
Contextual considerations
Internal contextual considerations Features of findings internal to a given
study E.g., pattern across all outcomes for a
particular subgroup in a study
External contextual considerations Features of findings external to a given
study E.g., consistency with prior study
findings
9
Subgroup analysis assess two questions
1. What is the impact of the program for each subgroup?
2. What are the relative impacts of the program across subgroups?
10
Power Calculations for Subgroup Analyses
Minimum detectable differential effects
11
Some considerations for power calculations of subgroups1. Did the program work for a particular subgroup ?
Assess impacts separately for this subgroup Assess power to detect impacts for this
subgroup
2. Were the effects different for particular subgroups?
Assess impacts using a cross-level interaction Assess power to detect a cross-level
interaction12
#1: Did the program work for a particular subgroup?
Minimum Detectable Effect Size (MDES): the smallest true effect, in standard deviations of the outcome, that is detectable for a given level of power and statistical significance.
Accepted parameters:Power: 80% Statistical significance level: 0.05
13
Establishing common notation
ρ = intraclass correlationδ = MDESλ = non-centrality parameter J = number of clustersn = number of units per cluster
14
Power in a 2-level CRT (J. Spybrook)
Main effect:
Main effect with covariate:
Cluster level Moderator:
Individual level Moderator:
Jn //14
2
JnRWSWS //1116 2
|
2
|
JnR XX //1116 2
|
2
|
JnRWW //114 2
|
2
|
Considerations for power analysis
The number of clusters (highest level units) is more important than the size of the cluster (lower level units) in reducing the MDES.
A higher intra-cluster correlation (ICC) increases the MDES (i.e., if τ00 is relatively large).
The proportion of variance in the outcome you can predict with L1 and L2 variables (i.e., R|X
2 and R|W2) reduces
the MDES.
16
Subgroup specific analysis
Maintains a significant portion of power because the number of clusters (or L2 units) remains the same.
The only statistical difference between the subexperiment and the full experiment is the number of L1 units per cluster.
17
Example using OD software
18
#2: Did the program work differently for different subgroups?
Minimum Detectable Effect Size Differences (MDESD): the smallest true effect of the difference in program impacts for two subgroups, in standard deviations of the outcome, that is detectable for a given level of power and statistical significance.
Accepted parameters: Power: 80% Statistical significance level: 0.05 19
Power in a 2-level CRT (J. Spybrook)
Main effect:
Main effect with covariate:
Cluster level Moderator:
Individual level Moderator:
Jn //14
2
JnRWSWS //1116 2
|
2
|
JnR XX //1116 2
|
2
|
JnRWW //114 2
|
2
|
Considerations for power analysis
Within-level variance becomes increasingly important. Implications include: The number of cases per cluster (lower
level units) become more important for increasing power.
The intra-cluster correlation (ICC) becomes less significant in affecting power (though still important).
The proportion of variance in the outcome you can predict with L1 variables (not L2; i.e., R|X
2) increases power.
21
Cross-level interactions
Assessing individual-level moderation in cluster-randomized trials using multi-level models
22
Testing relationships using multi-level modeling
1. Lower level direct effects. Does a L1 predictor X (e.g., student gender) have a relationship with the L1 outcome variable Y (e.g., student reading)?
2. Cross-level direct effects. Does a L2 predictor (e.g., school treatment status) have a relationship with an L1 outcome variable Y (e.g., student reading)?
3. Cross-level interaction effects. Does the nature or strength of the relationship between a L1 variable (e.g., gender) and the outcome (e.g., reading) change as a function of a higher-level variable (e.g., school treatment status)?
23
Model of Program Impacts
Level 1Level 2
Yij = outcome for individual i in cluster jTj = 1 for program-group members, 0 for control-groupγ 00 = mean outcome for the control groupγ 01 = true program impactrij = error component for individual i from cluster ju0j = error component for cluster j
24
Combined equation
25
Adding in a cross-level interaction
Level 2 (e.g., school treatment status) and Level 1 (e.g., student gender) variables interacting to produce an effect on the outcome (e.g., student reading scores).
In terms of your impact estimation equation: (a) Add Level 1 predictor (moderator). (b) Expand Level 2 model to include a fixed
slope (1). (c) Add a level 2 predictor (treatment status) to
the slope. 26
Model of Moderated Program Impacts
Level 1 Level 2
Added L1 predictor (moderator)
Expand L2 slope
Add L2 predictor to the slope
27
Model of Moderated Program Impacts
Level 1 Level 2
γ00 = mean outcome for the control groupγ01 = estimated program impact for Mij=0γ10 = main effect for the moderating variable, Tj=0γ11 = moderated effect (i.e., interaction)
Coefficient for cross-level interaction
28
Combined equation
29
How to graph a cross-level interaction effect
“Simple Regression Equation”: Calculate the expected values of Yij under different conditions of Tj and Mij
For continuous moderators, plot at values of one standard deviation below the mean, the mean, and one standard deviation above the mean for M.
It may also be useful to choose additional values that may be informative in specific contexts.
30
Simple Regression Equation
E(Yij | Mij ,Tj) = γ00 + γ01(Tj )+ γ10(Mij )+ γ11(Mij)(Tj)
Under control conditions:E(Yij | Mij ,Tj = 0) = γ00 + γ10(Mij)
Under treatment conditions:E(Yij | Mij ,Tj = 1) = γ00 + γ01 + γ10(Mij) + γ11 (Mij)
31
BREAKIf we need it
32
Centering Variables in Multi-level Models
Implications for interpreting effect estimates and detecting impact variation
33
Decisions about centering depend on your data and your research question How do you want to interpret the
intercept in your model? The coefficients? Example: School diversity/cultural awareness
programH1: Improved sense of belonging for
minority students (L1 moderator).H2: Improved sense of belonging for
minority students in less diverse schools (L1 & L2 moderators).
The distribution of the moderator variable across clusters needs to be considered.
34
Two options to center in multi-level models
CGM = Centering at the grand mean Deviations calculated from the sample
mean for all individuals CGM L1 with all individuals; L2 with all
clusters
CWC = Centering within clusters aka, group-mean centering Deviations calculated around the mean
of the cluster j to which case i belongs35
Y(outcome)
X(predictor)
The distribution of M is highly variable across clusters.
36
Cluster 1Cluster 2Cluster 3
Y(outcome)
CGM
M
37
X(predictor)
Y(outcome)
X(predictor)
CGM
M
38
Centering at the Grand Mean (CGM)
Does not affect the rank order of scores on the variable. The complex, multilevel association between the L1 and L2 variables is unaffected.
Yields scores that are correlated with variables at both levels of the hierarchy. (This is a critical differences with CWC.)
Produces an interaction coefficient (γ11) that is a weighted combination of the within- and between-cluster regression coefficients. 39
Y(outcome)
X(predictor)
CWC
M1
M2
M3
40
The distribution of M is highly concentrated within clusters.
Y(outcome)
X(predictor)
CWC
M1
M2
M3
41
Centering within Cluster (CWC) Affects the rank order of scores of
variables within the sample. Produces scores that are uncorrelated
with Level 2 variables (because the mean for all L2 variables is zero).
Produces an interaction coefficient (γ11) that is an unbiased estimate of the Level 1 association γ11 is a pure estimate of the cross-level
interaction, no longer confounded with the Level 2 interaction. 42
Y(outcome)
X(predictor)
43
The distribution of M is even across clusters.
Y(outcome)
X(predictor)
CGM
M
44
Y(outcome)
X(predictor)
CWC
M1M2M3
45
Main Takeaways
Centering will affect estimates more if the predictor variable is not evenly distributed across clusters.
Cross-level interaction term using CGM will provide a coefficient estimate that is a mix of the L1 and L2 effects.
Cross-level interaction term using CWC will provide a pure estimate of the L1 relationship.
Decisions on how to center depend on your data and your research question (!!).
46
Example
Predictor: Treatment status (L2)
Individual level moderator: Student age (L1) (continuous)
Outcome: Reading score
Some options on how to center the data and what it means for interpreting your moderated effect…
47
Option 1: No centering
Level 1 Level 2
γ00 is the average school mean reading score for the control group when age=0
γ 10 is the composite of the relationship of within school age-reading scores and between-school age reading scores
γ 11 is the composite of the interaction between treatment and within school age-reading scores and treatment and between-school age reading scores.
48
Option 1: CGM age
Level 1 Level 2
γ00 is the average school mean reading score for schools for the control group.
γ 10 is the composite of the relationship of within school age-reading scores and between-school age reading scores.
γ 11 is still the composite of the interaction between treatment and within school age-reading scores and treatment and between-school age reading scores.
49
Option 3: CWC age at L1, CGM age at L2
50
Level 1Level 2
γ00 is the average school mean reading score across the schools for the control group.
γ10 is the average change in school mean reading score for a 1 unit increase in school mean age across schools (between school age relationship)
γ 11 is the composite of the interaction between treatment and within school age-reading scores and treatment and between-school age reading scores.
Option 4: CWC age at L1, CGM age at L2 interacted with treatment status
51
Level 1Level 2
γ00 is the average school mean reading fluency across the schools for the control group.
γ10 is the average change in school mean reading fluency for a 1 unit increase in school mean age across schools (between school age relationship).
γ03 is the moderated relationship between treatment and between school age reading scores.
γ11 is the moderated relationship between treatment and within school age-reading scores.
General Recommendations and Tips
52
Multiple Hypothesis Testing• Distortions to statistical inferences can occur when
multiple related hypothesis tests are conducted.
• Suggested approaches:1. Explicitly distinguish between exploratory and
confirmatory findings2. Minimize the number of confirmatory hypothesis tests
conducted by a given study.3. Create an omnibus hypothesis test about the
intervention’s effects that considers all outcome measures and subgroups together. (e.g., composite measure of individual outcomes).
4. Consider family-wise error correction (reduces statistical power considerably).
53
Practical tips and recommendations
1. Calculate ρ for all levels.
2. Determine your research question and relevant approach to assessing subgroup affects.
3. Calculate the power needed to detect a subgroup effect (either for a particular subgroup, or for a cross-level interaction, depending on your research question).
4. Rescale (i.e., center) predictor variables as needed.
5. Assess the practical significance of your findings (i.e., calculate effect sizes).
6. Report results regarding each step of the model building process including all coefficients, standard errors and variance components.
54
References
Aguinis, H., Gottfredson, R. K., & Culpepper, S. A. (2013). Best-practice recommendations for estimating cross-level interaction effects using multilevel modeling. Journal of Management, 0149206313478188.
Bloom, H. S. (Ed.). (2005). Learning more from social experiments: Evolving analytic approaches. Russell Sage Foundation.
Bloom, H. & Michalopoulos, M. (2013). When Is the Story in the Subgroups? MDRC Working Paper.
Enders, C. K., & Tofighi, D. (2007). Centering predictor variables in cross-sectional multilevel models: a new look at an old issue. Psychological methods, 12(2), 121.
Mathieu, J. E., Aguinis, H., Culpepper, S. A., & Chen, G. (2012). Understanding and estimating the power to detect cross-level interaction effects in multilevel modeling. Journal of Applied Psychology, 97(5), 951. 55