hierarchical linear modeling: understanding applications ...€¦ · 16/05/2016 · introduce...
TRANSCRIPT
![Page 1: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/1.jpg)
Hierarchical Linear Modeling:
Understanding Applications in the MSP
Projects
NSF # DRL1238120
![Page 2: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/2.jpg)
The work of TEAMS is supported with funding provided by
the National Science Foundation, Award Number DRL
1238120. Any opinions, suggestions, and conclusions or
recommendations expressed in this presentation are those
of the presenter and do not necessarily reflect the views of
the National Science Foundation; NSF has not approved or
endorsed its content.
2
![Page 3: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/3.jpg)
Strengthening the quality of the MSP project evaluation
and building the capacity of the evaluators by
strengthening their skills related to evaluation design,
methodology, analysis, and reporting.
3
![Page 4: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/4.jpg)
Website at http://teams.mspnet.org
Online Help-Desk for submitting requests
Assistance with instruments
Consultation and targeted TA
Webinar series on specific evaluation topics
White papers/focused topic papers
4
![Page 5: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/5.jpg)
Hierarchical Linear Modeling: Understanding
Applications in the MSP Projects
Presenters:
Karen Drill, RMC Research Corporation
Emma Espel, RMC Research Corporation
Moderator:
John Sutton, RMC Research Corporation,
TEAMS Project PI
5
![Page 6: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/6.jpg)
6
Introduce Hierarchical Linear Modeling (HLM) principles and techniques
Discuss appropriate use of HLM within MSP projects
Provide concrete examples of the use of HLM within MSP projects
Goals:
NSF # DRL1238120
![Page 7: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/7.jpg)
7
What is HLM?
When to use HLM
Example HLM Use
Pro Tips: What (not) to do
Webinar Sections
Tools & Resources
![Page 8: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/8.jpg)
8
What is HLM?
When to use HLM
Example HLM Use
Pro Tips: What (not) to do
Webinar Sections
Tools & Resources
![Page 9: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/9.jpg)
9
What is HLM?
How familiar are you with HLM?
![Page 10: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/10.jpg)
A complex form of
ordinary least squares
regression
Can be used to analyze
variance in outcome
variables when predictor
variables are at different
hierarchical levels
10
What is HLM?
![Page 11: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/11.jpg)
Linear regression attempts
to model the relationship
between two variables by
fitting a linear equation to
observed data.
11
Review of Linear Regression
Math interest
Ma
th a
ch
ieve
me
nt
![Page 12: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/12.jpg)
Y’= 𝐵0+ 𝐵𝑌𝑋𝑋 + 𝜀
Y’ = The predicted value
𝐵0 = Y-intercept—the value of Y’ when X = 0
𝐵𝑌𝑋 = Slope—the regression coefficient for
predicting Y
X = Independent variable or predictor
𝜀 = Error
12
Review of Linear Regression
0%
50%
100%
0 1 2 3 4 5
Y’= 0.002 + 0.180 𝑋 + .210
Student interest in math
Pe
rce
nt
co
rre
ct
on
ma
th c
on
ten
t e
xam
![Page 13: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/13.jpg)
Y’= 𝐵0+ 𝐵𝑌𝑋𝑋 + 𝜀
Y’ = The predicted value
𝐵0 = Y-intercept—the value of Y’ when X = 0
𝐵𝑌𝑋 = Slope—the regression coefficient for
predicting Y
X = Independent variable or predictor
𝜀 = Error
13
Review of Linear Regression
0%
50%
100%
0 1 2 3 4 5
Y’= 0.002 + 0.180 𝑋 + .210
Student interest in math
Pe
rce
nt
co
rre
ct
on
ma
th c
on
ten
t e
xam
![Page 14: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/14.jpg)
Y’= 𝐵0+ 𝐵𝑌𝑋𝑋 + 𝜀
Y’ = The predicted value
𝐵0 = Y-intercept—the value of Y’ when X = 0
𝐵𝑌𝑋 = Slope—the regression coefficient for
predicting Y
X = Independent variable or predictor
𝜀 = Error
14
Review of Linear Regression
0%
50%
100%
0 1 2 3 4 5
Y’= 0.002 + 0.180 𝑋 + .210
Student interest in math
Pe
rce
nt
co
rre
ct
on
ma
th c
on
ten
t e
xam
![Page 15: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/15.jpg)
Y’= 𝐵0+ 𝐵𝑌𝑋𝑋 + 𝜀
Y’ = The predicted value
𝐵0 = Y-intercept—the value of Y’ when X = 0
𝐵𝑌𝑋 = Slope—the regression coefficient for
predicting Y
X = Independent variable or predictor
𝜀 = Error
15
Review of Linear Regression
0%
50%
100%
0 1 2 3 4 5
Y’= 0.002 + 0.180 𝑋 + .210
Student interest in math
Pe
rce
nt
co
rre
ct
on
ma
th c
on
ten
t e
xam
![Page 16: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/16.jpg)
Y’= 𝐵0+ 𝐵𝑌𝑋𝑋 + 𝜀
Y’ = The predicted value
𝐵0 = Y-intercept—the value of Y’ when X = 0
𝐵𝑌𝑋 = Slope—the regression coefficient for
predicting Y
X = Independent variable or predictor
𝜀 = Error
16
Review of Linear Regression
0%
50%
100%
0 1 2 3 4 5
Y’= 0.002 + 0.180 𝑋 + .210
Student interest in math
Pe
rce
nt
co
rre
ct
on
ma
th c
on
ten
t e
xam
![Page 17: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/17.jpg)
Based on linear regression
17
HLM Similarities to Linear Regression
![Page 18: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/18.jpg)
Models the relationship between the
observed to the expected
18
HLM Similarities to Linear Regression
![Page 19: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/19.jpg)
Can be cross-sectional or longitudinal
19
HLM Similarities to Linear Regression
![Page 20: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/20.jpg)
20
Differences from Linear Regression
Level-3 (school)
Level-2 (teacher)
Level-1 (students)
Green = Level 1
Orange = Level 2
![Page 21: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/21.jpg)
21
Differences from Linear Regression
Level-3 (school)
Level-2 (teacher)
Level-1 (students)
Intraclass Correlation (ICC)
Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5
![Page 22: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/22.jpg)
22
Differences from Linear Regression
HLM: Multiple Levels Ecological Fallacy: One Level
Green = Level 1
Orange = Level 2
![Page 23: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/23.jpg)
23
Questions?
![Page 24: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/24.jpg)
24
MSP Scenario and HLM Equations
![Page 25: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/25.jpg)
25
MSP Scenario and HLM Equations
You are the evaluator of an MSP that is implementing an innovative
math curriculum for 6th graders.
You are interested in whether implementing this curriculum influences
students’ math achievement scores.
Your sample also includes a matched comparison group of teachers
not implementing the curriculum.
To what extent does teacher implementation of the math curriculum
influence students’ math achievement scores?
![Page 26: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/26.jpg)
Variables
Y = Students’ achievement scores (level-1 outcome)
X = Female (level-1 predictor)
W = Treatment (the math curriculum) (level-2 predictor)
26
To what extent does teacher implementation of the math
curriculum influence students’ math achievement scores?
![Page 27: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/27.jpg)
(level-1)𝑌𝑖𝑗= 𝛽0𝑗 + 𝛽1𝑗 (𝐹𝑒𝑚𝑎𝑙𝑒𝑖𝑗)* + 𝑟𝑖𝑗
𝑌𝑖𝑗= dependent variable measured for 𝑖th level-1 (student) unit
nested within the 𝑗th level-2 (teacher) unit
𝑌𝑖𝑗 = students’ math achievement score
27
To what extent does teacher implementation of the math
curriculum influence students’ math achievement scores?
*dummy coded
![Page 28: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/28.jpg)
(level-1)𝑌𝑖𝑗= 𝛽0𝑗 + 𝛽1𝑗 (𝐹𝑒𝑚𝑎𝑙𝑒𝑖𝑗) + 𝑟𝑖𝑗
𝛽0𝑗 = intercept for the 𝑗th level-2 (teacher) unit
𝛽0𝑗 = best estimate for predicting
math achievement for males
28
To what extent does teacher implementation of the math
curriculum influence students’ math achievement scores?
![Page 29: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/29.jpg)
(level-1)𝑌𝑖𝑗= 𝛽0𝑗 + 𝛽1𝑗 (𝐹𝑒𝑚𝑎𝑙𝑒𝑖𝑗) + 𝑟𝑖𝑗
𝛽1𝑗 = regression coefficient associated
with 𝑋𝑖𝑗 for the 𝑗th level-2 (teacher) unit
𝛽1𝑗 = level-1 slope
𝛽1𝑗 = the effect of being female on math achievement
29
To what extent does teacher implementation of the math
curriculum influence students’ math achievement scores?
![Page 30: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/30.jpg)
(level-1)𝑌𝑖𝑗= 𝛽0𝑗 + 𝛽1𝑗 (𝐹𝑒𝑚𝑎𝑙𝑒𝑖𝑗) + 𝑟𝑖𝑗
(𝐹𝑒𝑚𝑎𝑙𝑒)𝑖𝑗 = value on the level-1 (student) predictor
(𝐹𝑒𝑚𝑎𝑙𝑒𝑖𝑗) = value for female (0 = not female, 1 = female)*
*dummy coded
30
To what extent does teacher implementation of the math
curriculum influence students’ math content achievement scores?
![Page 31: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/31.jpg)
(level-1)𝑌𝑖𝑗= 𝛽0𝑗 + 𝛽1𝑗 (𝐹𝑒𝑚𝑎𝑙𝑒𝑖𝑗) + 𝑟𝑖𝑗
𝑟𝑖𝑗 = random error associated with the 𝑖th level-1 unit (student)
nested within the 𝑗th level-2 (teacher) unit
𝑟𝑖𝑗 = deviation for each student from the fitted model
31
To what extent does teacher implementation of the math
curriculum influence students’ math content knowledge?
![Page 32: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/32.jpg)
(level-1) 𝑌𝑖𝑗= 𝛽0𝑗 + 𝛽1𝑗 (𝐹𝑒𝑚𝑎𝑙𝑒𝑖𝑗) + 𝑟𝑖𝑗
(level -2) 𝛽0𝑗= 𝛾00 + 𝛾01 (𝑇𝑥)1𝑗 + 𝑢0𝑗
𝛽0𝑗 = intercept for the 𝑗th level-2 unit
32
To what extent does teacher implementation of the math
curriculum influence students’ math content knowledge?
![Page 33: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/33.jpg)
(level-1) 𝑌𝑖𝑗= 𝛽0𝑗 + 𝛽1𝑗 (𝐹𝑒𝑚𝑎𝑙𝑒𝑖𝑗) + 𝑟𝑖𝑗
(level -2) 𝛽0𝑗= 𝛾00 + 𝛾01 (𝑇𝑥)1𝑗 + 𝑢0𝑗
Υ00 = level-2 intercept
Υ00 = mean math achievement
for comparison schools
33
To what extent does teacher implementation of the math
curriculum influence students’ math achievement scores?
![Page 34: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/34.jpg)
(level-1) 𝑌𝑖𝑗= 𝛽0𝑗 + 𝛽1𝑗 (𝐹𝑒𝑚𝑎𝑙𝑒𝑖𝑗) + 𝑟𝑖𝑗
(level -2) 𝛽0𝑗= 𝛾00 + 𝛾01 (𝑇𝑥)1𝑗 + 𝑢0𝑗
Υ01 = level-2 slope for treatment
34
To what extent does teacher implementation of the math
curriculum influence students’ math content knowledge?
![Page 35: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/35.jpg)
(level-1) 𝑌𝑖𝑗= 𝛽0𝑗 + 𝛽1𝑗 (𝐹𝑒𝑚𝑎𝑙𝑒𝑖𝑗) + 𝑟𝑖𝑗
(level -2) 𝛽0𝑗= 𝛾00 + 𝛾01 (𝑇𝑥)1𝑗 + 𝑢0𝑗
(𝑇𝑥)1𝑗 = value on the level-2 predictor
(𝑇𝑥)1𝑗= value for treatment (0 = no treatment, 1 = treatment)*
*dummy coded
35
To what extent does teacher implementation of the math
curriculum influence students’ math content knowledge?
![Page 36: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/36.jpg)
(level-1) 𝑌𝑖𝑗= 𝛽0𝑗 + 𝛽1𝑗 (𝐹𝑒𝑚𝑎𝑙𝑒𝑖𝑗) + 𝑟𝑖𝑗
(level -2) 𝛽0𝑗= 𝛾00 + 𝛾01 (𝑇𝑥)1𝑗 + 𝑢0𝑗
𝑢0𝑗 = random effects of the 𝑗th level-2 unit
adjusted for treatment on the intercept
𝑢0𝑗 = unique effect for each school on mean math achievement
36
To what extent does teacher implementation of the math
curriculum influence students’ math content knowledge?
![Page 37: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/37.jpg)
37
HLM Challenges
x
![Page 38: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/38.jpg)
Insufficient power at level -1 or level-2
38
HLM Challenges: Power
![Page 39: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/39.jpg)
Measures need strong psychometric properties
39
HLM challenges: Meeting model assumptions
![Page 40: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/40.jpg)
Level-1 residuals need to be independent and
normally distributed
40
HLM challenges: Meeting model assumptions
![Page 41: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/41.jpg)
41
Questions?
![Page 42: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/42.jpg)
42
What is HLM?
When to use HLM
Example HLM Use
Pro Tips: What (not) to do
Webinar Sections
Tools & Resources
![Page 43: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/43.jpg)
Does the data have a nested
structure?
Is there sufficient
power at the highest level?
Does your ICC reach an
acceptable level?
Are assumptions
met?
Is there sufficient
power at the lowest level?
HLM is likely a good choice
Consider HLM with
reservations
Consider regression or another more appropriate
design.
NO
NO
YES
YES
YES
NO
When to use HLM
YES
NO
![Page 44: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/44.jpg)
Does the data have a nested
structure?
Is there sufficient
power at the highest level?
Does your ICC reach an
acceptable level?
Are assumptions
met?
Is there sufficient
power at the lowest level?
HLM is likely a good choice
Consider HLM with
reservations
Consider regression or another more appropriate
design.
NO
NO
NO
YES
YES
YES
NO
When to use HLM
YES
NO
![Page 45: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/45.jpg)
Does the data have a nested
structure?
Is there sufficient
power at the highest level?
Does your ICC reach an
acceptable level?
Are assumptions
met?
Is there sufficient
power at the lowest level?
HLM is likely a good choice
Consider HLM with
reservations
Consider regression or another more appropriate
design.
YES NO
NO
NO
YES
YES
YES
NO
When to use HLM
YES
NO
![Page 46: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/46.jpg)
Does the data have a nested
structure?
Is there sufficient
power at the highest level?
Is there an adequate ICC to warrant multi-
level modelling?
Are assumptions met?
Is there sufficient power at the lowest level?
HLM is likely a good choice
Consider HLM with reservations
Consider regression or another more appropriate
design.
YES NO
NO
NO
NO
YES
YES
YES
YES
When to use HLM
Are you familiar with Power Analysis? Are you familiar with Optimal Design?
![Page 47: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/47.jpg)
Does the data have a nested
structure?
Is there sufficient
power at the highest level?
Does your ICC reach an
acceptable level?
Are assumptions
met?
Is there sufficient
power at the lowest level?
HLM is likely a good choice
Consider HLM with
reservations
Consider regression or another more appropriate
design.
YES NO
NO
NO
YES
YES
YES
NO
When to use HLM
YES
NO
![Page 48: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/48.jpg)
Does the data have a nested
structure?
Is there sufficient
power at the highest level?
Is there an adequate ICC to warrant multi-
level modelling?
Are assumptions met?
Is there sufficient power at the lowest level?
HLM is likely a good choice
Consider HLM with reservations
Consider regression or another more appropriate
design.
YES NO
NO
NO
YES
YES
NO
Bonus! WWC Recommends clustering adjustment for single-level analyses with multiple levels for significant findings.
1. Compute test statistic for effect size 2. Adjust test statistic and degrees of freedom for effect size 3. Identify significance value
Handy Resource: http://www.air.org/resource/wwc-phase-i-computation-tools-4-15-10
When to use HLM
![Page 49: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/49.jpg)
Does the data have a nested
structure?
Is there sufficient
power at the highest level?
Does your ICC reach an
acceptable level?
Are assumptions
met?
Is there sufficient
power at the lowest level?
HLM is likely a good choice
Consider HLM with
reservations
Consider regression or another more appropriate
design.
YES NO
NO
NO
YES
YES
YES
NO
When to use HLM
NO
![Page 50: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/50.jpg)
Does the data have a nested
structure?
Is there sufficient
power at the highest level?
Does your ICC reach an
acceptable level?
Are assumptions
met?
Is there sufficient
power at the lowest level?
HLM is likely a good choice
Consider HLM with
reservations
Consider regression or another more appropriate
design.
YES NO
NO
NO
YES
YES
YES
NO
When to use HLM
NO
![Page 51: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/51.jpg)
Does the data have a nested
structure?
Is there sufficient
power at the highest level?
Does your ICC reach an
acceptable level?
Are assumptions
met?
Is there sufficient
power at the lowest level?
HLM is likely a good choice
Consider HLM with
reservations
Consider regression or another more appropriate
design.
YES NO
NO
NO
YES
YES
YES
NO
When to use HLM
NO
YES
![Page 52: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/52.jpg)
Does the data have a nested
structure?
Is there sufficient
power at the highest level?
Does your ICC reach an
acceptable level?
Are assumptions
met?
Is there sufficient
power at the lowest level?
HLM is likely a good choice
Consider HLM with
reservations
Consider regression or another more appropriate
design.
YES NO
NO
NO
YES
YES
YES
NO
When to use HLM
NO
YES
![Page 53: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/53.jpg)
Does the data have a nested
structure?
Is there sufficient
power at the highest level?
Does your ICC reach an
acceptable level?
Are assumptions
met?
Is there sufficient
power at the lowest level?
HLM is likely a good choice
Consider HLM with
reservations
Consider regression or another more appropriate
design.
YES NO
NO
NO
YES
YES
YES
NO
When to use HLM
NO
YES
![Page 54: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/54.jpg)
Does the data have a nested
structure?
Is there sufficient
power at the highest level?
Does your ICC reach an
acceptable level?
Are assumptions
met?
Is there sufficient
power at the lowest level?
HLM is likely a good choice
Consider HLM with
reservations
Consider regression or another more appropriate
design.
YES NO
NO
NO
YES
YES
NO
When to use HLM
NO
YES
![Page 55: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/55.jpg)
Does the data have a nested
structure?
Is there sufficient
power at the highest level?
Does your ICC reach an
acceptable level?
Are assumptions
met?
Is there sufficient
power at the lowest level?
HLM is likely a good choice
Consider HLM with
reservations
Consider regression or another more appropriate
design.
YES NO
NO
NO
YES
YES
NO
When to use HLM
NO
YES
YES
![Page 56: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/56.jpg)
To what extent does teacher participation in the MSP contribute to student
science content knowledge?
Is HLM appropriate?
57
When to use HLM
Scenario
You are the evaluator of an MSP designed to train teams of teachers in
science content knowledge for 8th graders.
Why or Why Not?
Major activities include an intensive summer institute, learning teams of
involved teachers, teacher leaders, and research activities.
Teachers randomly assigned to training or not (cluster randomized trial)
N teachers = 17 Tx, 42 Control
N students = 2,025
![Page 57: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/57.jpg)
Does the data have a nested
structure?
Is there sufficient
power at the highest level?
Does your ICC reach an
acceptable level?
Are assumptions
met?
Is there sufficient
power at the lowest level?
HLM is likely a good choice
Consider HLM with
reservations
Consider regression or another more appropriate
design.
YES NO
NO
NO
YES
YES
YES
NO
When to use HLM
YES
NO
![Page 58: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/58.jpg)
Does the data have a nested
structure?
Is there sufficient
power at the highest level?
Does your ICC reach an
acceptable level?
Are assumptions
met?
Is there sufficient
power at the lowest level?
HLM is likely a good choice
Consider HLM with
reservations
Consider regression or another more appropriate
design.
YES NO
NO
NO
YES
YES
YES
NO
When to use HLM
YES
NO
![Page 59: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/59.jpg)
60
When to use HLM: Power
Raudenbush, S. W., et al. (2011). Optimal Design Software for Multi-level and
Longitudinal Research (Version 3.01) [Software]. Available from
www.wtgrantfoundation.org.
http://sitemaker.umich.edu/group-based/optimal_design_software
![Page 60: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/60.jpg)
Does the data have a nested structure?
Is there sufficient power at the highest level?
What is the ICC?
Are assumptions met?
Is there sufficient power at the lowest
level?
HLM is likely a good choice
Consider HLM with reservations
Consider regression or another more
appropriate design.
YES NO
NO
NO
YES
YES
YES
NO
When to use HLM
Keep in mind as you move forward with analysis planning.
![Page 61: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/61.jpg)
Decision
OLS Regression was used to analyze the data
due to insufficient power to detect an effect of
the program.
63
When to use HLM (or not)
Do you agree? Why or Why not?
![Page 62: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/62.jpg)
64
Questions?
![Page 63: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/63.jpg)
65
What is HLM?
When to use HLM
Example HLM Use
Pro Tips: What (not) to do
Webinar Sections
Tools & Resources
![Page 64: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/64.jpg)
66
When to use HLM
Scenario: Reading HLM Reports
You are the evaluator of an MSP designed to train teams of teachers in
science content knowledge for 8th graders.
Major activities include an intensive summer institute, learning teams of
involved teachers, teacher leaders, and research activities.
N teachers = 148 Tx, 150 Control*
Teachers randomly assigned to training or not (cluster randomized trial)
N students = 2,358**
To what extent does teacher participation in the MSP contribute to student
science content knowledge (assuming all students have scores for the
standardized state science test)?
You are developing the analysis plan for this project.
* Teacher level variables: MSP teacher, MSP Leader
**Student level covariates: Gender, Title I status, Individualized Education Plan (IEP), Hispanic, English
Language Learner, prior Normal Curve Equivalent score (NCE)
![Page 65: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/65.jpg)
67
Example HLM Use
Model 1 Model 2 Model 3 Model 4
Est. SE Est. SE Est. SE Est. SE
Intercept 52.004 1.660 52.025 1.664 51.200 0.483 52.214 0.560
Gender 1.323 0.731 1.323 0.731 1.283 0.731
Title I -0.956 1.972 -0.956 1.973 -0.904 2.010
IEP -17.525*** 1.250 -17.524*** 1.251 -17.505*** 1.250
Hispanic -10.348*** 1.250 -10.348*** 1.250 -10.287*** 1.250
ELL -6.659*** 1.236 -6.659*** 1.236 -6.686*** 1.235
Normal Curve Equivalent
(NCE) pretest
1.244 0.124 1.168*** 0.112
MSP Teacher 2.522* 0.897
MSP Leader 1.045 1.349
HLM Deviance 15,939.2 15,362.9 15,341.6 15,330.3
Intraclass Correlation (ICC) .077
Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)
*p < .05, **p < .01, ***p < .001
The first model is always a null
model.
![Page 66: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/66.jpg)
68
Example HLM Use
Model 1 Model 2 Model 3 Model 4
Est. SE Est. SE Est. SE Est. SE
Intercept 52.004 1.660 52.025 1.664 51.200 0.483 52.214 0.560
Gender 1.323 0.731 1.323 0.731 1.283 0.731
Title I -0.956 1.972 -0.956 1.973 -0.904 2.010
IEP -17.525*** 1.250 -17.524*** 1.251 -17.505*** 1.250
Hispanic -10.348*** 1.250 -10.348*** 1.250 -10.287*** 1.250
ELL -6.659*** 1.236 -6.659*** 1.236 -6.686*** 1.235
Normal Curve Equivalent
(NCE) pretest
1.244 0.124 1.168*** 0.112
MSP Teacher 2.522* 0.897
MSP Leader 1.045 1.349
HLM Deviance 15,939.2 15,362.9 15,341.6 15,330.3
Intraclass Correlation (ICC) .077
Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)
*p < .05, **p < .01, ***p < .001
On average, participants had a
NCE score of 52.004, with a
standard error of 1.660.
![Page 67: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/67.jpg)
69
Example HLM Use
Model 1 Model 2 Model 3 Model 4
Est. SE Est. SE Est. SE Est. SE
Intercept 52.004 1.660 52.025 1.664 51.200 0.483 52.214 0.560
Gender 1.323 0.731 1.323 0.731 1.283 0.731
Title I -0.956 1.972 -0.956 1.973 -0.904 2.010
IEP -17.525*** 1.250 -17.524*** 1.251 -17.505*** 1.250
Hispanic -10.348*** 1.250 -10.348*** 1.250 -10.287*** 1.250
ELL -6.659*** 1.236 -6.659*** 1.236 -6.686*** 1.235
Normal Curve Equivalent
(NCE) pretest
1.244 0.124 1.168*** 0.112
MSP Teacher 2.522* 0.897
MSP Leader 1.045 1.349
HLM Deviance 15,939.2 15,362.9 15,341.6 15,330.3
Intraclass Correlation (ICC) .077
Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)
*p < .05, **p < .01, ***p < .001
Deviance indicates model fit,
and lower deviance indicates
better fit.
![Page 68: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/68.jpg)
70
Example HLM Use
Model 1 Model 2 Model 3 Model 4
Est. SE Est. SE Est. SE Est. SE
Intercept 52.004 1.660 52.025 1.664 51.200 0.483 52.214 0.560
Gender 1.323 0.731 1.323 0.731 1.283 0.731
Title I -0.956 1.972 -0.956 1.973 -0.904 2.010
IEP -17.525*** 1.250 -17.524*** 1.251 -17.505*** 1.250
Hispanic -10.348*** 1.250 -10.348*** 1.250 -10.287*** 1.250
ELL -6.659*** 1.236 -6.659*** 1.236 -6.686*** 1.235
Normal Curve Equivalent
(NCE) pretest
1.244 0.124 1.168*** 0.112
MSP Teacher 2.522* 0.897
MSP Leader 1.045 1.349
HLM Deviance 15,939.2 15,362.9 15,341.6 15,330.3
Intraclass Correlation (ICC) .077
Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)
*p < .05, **p < .01, ***p < .001
7.7% of the variance in science
achievement was due to variation
between teachers.
![Page 69: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/69.jpg)
71
Example HLM Use
Model 1 Model 2 Model 3 Model 4
Est. SE Est. SE Est. SE Est. SE
Intercept 52.004 1.660 52.025 1.664 51.200 0.483 52.214 0.560
Gender 1.323 0.731 1.323 0.731 1.283 0.731
Title I -0.956 1.972 -0.956 1.973 -0.904 2.010
IEP -17.525*** 1.250 -17.524*** 1.251 -17.505*** 1.250
Hispanic -10.348*** 1.250 -10.348*** 1.250 -10.287*** 1.250
ELL -6.659*** 1.236 -6.659*** 1.236 -6.686*** 1.235
Normal Curve Equivalent
(NCE) pretest
1.244 0.124 1.168*** 0.112
MSP Teacher 2.522* 0.897
MSP Leader 1.045 1.349
HLM Deviance 15,939.2 15,362.9 15,341.6 15,330.3
Intraclass Correlation (ICC) .077
*p < .05, **p < .01, ***p < .001
Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358) Model 2 typically adds
Level 1 predictors
![Page 70: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/70.jpg)
72
Example HLM Use
Model 1 Model 2 Model 3 Model 4
Est. SE Est. SE Est. SE Est. SE
Intercept 52.004 1.660 52.025 1.664 51.200 0.483 52.214 0.560
Gender 1.323 0.731 1.323 0.731 1.283 0.731
Title I -0.956 1.972 -0.956 1.973 -0.904 2.010
IEP -17.525*** 1.250 -17.524*** 1.251 -17.505*** 1.250
Hispanic -10.348*** 1.250 -10.348*** 1.250 -10.287*** 1.250
ELL -6.659*** 1.236 -6.659*** 1.236 -6.686*** 1.235
Normal Curve Equivalent
(NCE) pretest 1.244 0.124 1.168*** 0.112
MSP Teacher 2.522* 0.897
MSP Leader 1.045 1.349
HLM Deviance 15,939.2 15,362.9 15,341.6 15,330.3
Intraclass Correlation (ICC) .077
*p < .05, **p < .01, ***p < .001
Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)
Model 3
typically
adds
predictors of
interest
![Page 71: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/71.jpg)
73
Example HLM Use
Model 1 Model 2 Model 3 Model 4
Est. SE Est. SE Est. SE Est. SE
Intercept 52.004 1.660 52.025 1.664 51.200 0.483 52.214 0.560
Gender 1.323 0.731 1.323 0.731 1.283 0.731
Title I -0.956 1.972 -0.956 1.973 -0.904 2.010
IEP -17.525*** 1.250 -17.524*** 1.251 -17.505*** 1.250
Hispanic -10.348*** 1.250 -10.348*** 1.250 -10.287*** 1.250
ELL -6.659*** 1.236 -6.659*** 1.236 -6.686*** 1.235
Normal Curve Equivalent
(NCE) pretest 1.244 0.124 1.168*** 0.112
MSP Teacher 2.522* 0.897
MSP Leader 1.045 1.349
HLM Deviance 15,939.2 15,362.9 15,341.6 15,330.3
Intraclass Correlation (ICC) .077
*p < .05, **p < .01, ***p < .001
Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)
Deviance decreased from Model 1 to Model 4.
![Page 72: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/72.jpg)
74
Example HLM Use
Model 4
Est. SE
Intercept 52.214 0.560
Gender 1.283 0.731
Title I -0.904 2.010
IEP -17.505*** 1.250
Hispanic -10.287*** 1.250
ELL -6.686*** 1.235
Normal Curve Equivalent
(NCE) pretest 1.168*** 0.112
MSP Teacher 2.522* 0.897
MSP Leader 1.045 1.349
HLM Deviance 15,330.3
Intraclass Correlation (ICC)
*p < .05, **p < .01, ***p < .001
Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)
Model 4 is the final model.
![Page 73: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/73.jpg)
75
Example HLM Use
Model 4
Est. SE
Intercept 52.214 0.560
Gender 1.283 0.731
Title I -0.904 2.010
IEP -17.505*** 1.250
Hispanic -10.287*** 1.250
ELL -6.686*** 1.235
Normal Curve Equivalent
(NCE) pretest 1.168*** 0.112
MSP Teacher 2.522* 0.897
MSP Leader 1.045 1.349
HLM Deviance 15,330.3
Intraclass Correlation (ICC)
*p < .05, **p < .01, ***p < .001
Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)
On average, students scored 52.214 NCE
units.
![Page 74: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/74.jpg)
78
Example HLM Use
Model 4
Est. SE
Intercept 52.214 0.560
Gender 1.283 0.731
Title I -0.904 2.010
IEP -17.505*** 1.250
Hispanic -10.287*** 1.250
ELL -6.686*** 1.235
Normal Curve Equivalent
(NCE) pretest 1.168*** 0.112
MSP Teacher 2.522* 0.897
MSP Leader 1.045 1.349
HLM Deviance 15,330.3
Intraclass Correlation (ICC)
*p < .05, **p < .01, ***p < .001
Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)
On average, students with an IEP scored
17.505 points lower than those without.
![Page 75: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/75.jpg)
79
Example HLM Use
Model 4
Est. SE
Intercept 52.214 0.560
Gender 1.283 0.731
Title I -0.904 2.010
IEP -17.505*** 1.250
Hispanic -10.287*** 1.250
ELL -6.686*** 1.235
Normal Curve Equivalent
(NCE) pretest 1.168*** 0.112
MSP Teacher 2.522* 0.897
MSP Leader 1.045 1.349
HLM Deviance 15,330.3
Intraclass Correlation (ICC)
*p < .05, **p < .01, ***p < .001
Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)
On average, Hispanic students scored
10.287 points lower than non-Hispanic
students.
![Page 76: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/76.jpg)
80
Example HLM Use
Model 4
Est. SE
Intercept 52.214 0.560
Gender 1.283 0.731
Title I -0.904 2.010
IEP -17.505*** 1.250
Hispanic -10.287*** 1.250
ELL -6.686*** 1.235
Normal Curve Equivalent
(NCE) pretest 1.168*** 0.112
MSP Teacher 2.522* 0.897
MSP Leader 1.045 1.349
HLM Deviance 15,330.3
Intraclass Correlation (ICC)
*p < .05, **p < .01, ***p < .001
Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)
On average, ELL students scored 6.686
points lower than non-ELL students.
![Page 77: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/77.jpg)
81
Example HLM Use
Model 4
Est. SE
Intercept 52.214 0.560
Gender 1.283 0.731
Title I -0.904 2.010
IEP -17.505*** 1.250
Hispanic -10.287*** 1.250
ELL -6.686*** 1.235
Normal Curve Equivalent
(NCE) pretest 1.168*** 0.112
MSP Teacher 2.522* 0.897
MSP Leader 1.045 1.349
HLM Deviance 15,330.3
Intraclass Correlation (ICC)
*p < .05, **p < .01, ***p < .001
Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)
For every NCE unit score on the pre-test,
students gained 1.168 NCE units on the
post-test, on average.
![Page 78: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/78.jpg)
82
Example HLM Use
Model 4
Est. SE
Intercept 52.214 0.560
Gender 1.283 0.731
Title I -0.904 2.010
IEP -17.505*** 1.250
Hispanic -10.287*** 1.250
ELL -6.686*** 1.235
Normal Curve Equivalent
(NCE) pretest 1.168*** 0.112
MSP Teacher 2.522* 0.897
MSP Leader 1.045 1.349
HLM Deviance 15,330.3
Intraclass Correlation (ICC)
*p < .05, **p < .01, ***p < .001
Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)
Students who had an MSP Teacher scored
2.522 points higher than those who did
not.
![Page 79: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/79.jpg)
83
Example HLM Use
Model 4
Est. SE
Intercept 52.214 0.560
Gender 1.283 0.731
Title I -0.904 2.010
IEP -17.505*** 1.250
Hispanic -10.287*** 1.250
ELL -6.686*** 1.235
Normal Curve Equivalent
(NCE) pretest 1.168*** 0.112
MSP Teacher 2.522* 0.897
MSP Leader 1.045 1.349
HLM Deviance 15,330.3
Intraclass Correlation (ICC)
*p < .05, **p < .01, ***p < .001
Exhibit X. Summary of Regression Analyses of the Effects of MSP Teacher Participation on Student Science Achievement in Grades 4-8 (N = 2,358)
There was no difference in scores for
students in classes that were taught by
MSP Leaders compared to those who were
not.
![Page 80: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/80.jpg)
84
Questions?
![Page 81: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/81.jpg)
85
What is HLM?
When to use HLM
Example HLM Use
Pro Tips: What (not) to do
Webinar Sections
Tools & Resources
![Page 82: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/82.jpg)
86
Pro Tips: What (not) to do
Make sure all variables are dummy
coded appropriately, with 1/0 for
each category and a reference
group.
Race X1 X2 X3 X4
White 1 0 0 0
Black 0 1 0 0
Hispanic 0 0 1 0
Other 0 0 0 0
![Page 83: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/83.jpg)
87
Think strategically about centering your
variables.
Uncentered: Xij
Group-mean centered: Xij − 𝑿 j
Grand-mean centered: Xij − 𝑿 ··
Pro Tips: What (not) to do
![Page 84: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/84.jpg)
88
Strategically build your model.
Null Model
Covariates
Level 1 Predictors of Interest
Level 2 Predictors of Interest
Pro Tips: What (not) to do
![Page 85: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/85.jpg)
89
Make sure to report relevant statistics. According to Abt Associates’
Guide for rigorous MSP evaluations and reporting:
Pro Tips: What (not) to do
![Page 86: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/86.jpg)
90
What is a challenge you have faced when running HLM or
considering whether or not to use HLM?
Pro Tips: What (not) to do
![Page 87: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/87.jpg)
91
What is HLM?
When to use HLM
Example HLM Use
Pro Tips: What (not) to do
Webinar Sections
Tools & Resources
![Page 88: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/88.jpg)
Software Options for HLM Analysis
92
SPSS
Tools and Resources
![Page 89: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/89.jpg)
93
Tools and Resources
Resources
SSI Website to download HLM and find resources:
http://www.ssicentral.com/hlm/resources.html
Schochet, P. Z., Puma, M., & Deke, J. (2014). Understanding variation in treatment effects in
education impact evaluations: An overview of quantitative methods (NCEE 2014–4017).
Washington, DC: U.S. Department of Education, Institute of Education Sciences, National
Center for Education Evaluation and Regional Assistance, Analytic Technical Assistance and
Development. Retrieved from http://ies.ed.gov/ncee/edlabs.
Raudenbush, S. W., et al. (2011). Optimal Design Software for Multi-level and Longitudinal
Research (Version 3.01) [Software]. Available from www.wtgrantfoundation.org or
//sitemaker.umich.edu/group-based/optimal_design_software.
Variance Almanac of Academic Achievement : https://arc.uchicago.edu/reese/variance-
almanac-academic-achievement
![Page 90: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/90.jpg)
94
Questions?
![Page 91: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/91.jpg)
95
TEAMS Resources & Tools
TEAMS MSP Project Document
Self-Appraisal
Purpose of the Evaluation
Evaluation Design &
Measurement
Analysis
Generalizability,
Representativeness, Utility
http://teams.mspnet.org/
index.cfm/27152
![Page 92: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/92.jpg)
Contact Us!
Karen Drill
Emma Espel
RMC Research Corporation
111 SW Columbia St., Suite 1030
Portland, OR 98201-5883
RMC Research Corporation
633 17th St., Suite 2100
Denver, CO 80202-1620
96
John T. Sutton, PI
Dave Weaver, Co-PI
RMC Research Corporation
633 17th St., Suite 2100
Denver, CO 80202-1620
Phone: 303-825-3636
Toll Free: 800-922-3636
Fax: 303-825-1626
RMC Research Corporation
111 SW Columbia St., Suite 1030
Portland, OR 97201-5883
Phone: 503-223-8248
Toll Free: 800-788-1887
Fax: 503-223-8399
![Page 93: Hierarchical Linear Modeling: Understanding Applications ...€¦ · 16/05/2016 · Introduce Hierarchical Linear Modeling (HLM) principles and techniques Discuss appropriate use](https://reader035.vdocuments.mx/reader035/viewer/2022063012/5fc9f6a4aacf7e0f4a19c4bc/html5/thumbnails/93.jpg)
97
Follow Up Survey
Please Complete the Feedback
Survey
URL: http://www.rmccorp.com/TEAMS/
Username: 23569
Password: TEAMS
Thank You!
NSF # DRL 1238120