example of models for the study of change
DESCRIPTION
Example of Models for the Study of Change. David A. Kenny. Example Data. An Honors Thesis done by Allison Gillum of Skidmore College supervised by John Berman on the effects of an semester-long class on the Environment on Environmental Responsible Behaviors. Pretest-Posttest Design - PowerPoint PPT PresentationTRANSCRIPT
Example of Models for the Study of Change
David A. Kenny
December 15, 2013
2
Example DataAn Honors Thesis done by Allison Gillum of
Skidmore College supervised by John Berman on the effects of an semester-long class on the Environment on Environmental Responsible Behaviors.
Pretest-Posttest Design
41 Treated and 199 Controls
2 Treated classes and 8 Control classes. No clustering effect due to class.
Outcome: Environmentally Responsible Behaviors (ERB), a 12 item scale ranging for 1 to 7. For latent variable analyses, 3 parcels of 4 items were created.
3
Models• Models
– Controlling for Baseline• Simple• Allowing for Unreliability at Time 1
– Change Score Analysis• Raykov• LCS• Kenny-Judd
– Standardized Change Analysis• Types
– Univariate (average of 12 items)– Latent Variable (3 parcels of 4 items)
4
Latent Variable Measurement Models
• Unconstrained– 2(9) = 13.22, p = .153– RMSEA = 0.044; TLI = .992
• Equal Loadings– 2(11) = 18.63, p = .068– RMSEA = 0.054; TLI = .988• The equal loading model has
reasonable fit.
5
Pretest Difference• Mean for Controls: 4.79• Mean for Treateds: 5.21• A mean difference of 0.42• t(238) = 3.191, p = .002• d = 0.64, a moderate effect size• There is a difference at the pretest!• The mean difference on latent variable at time 1 is 0.45.
6
More on the Pretest Difference
• Likely more environmentally conscious students more likely to take an environmental course.
• Would you expect the difference to persist (CSA) or narrow (CfB)?
7
Controlling for Baseline: Univariate
•A beneficial effect of the course on the outcome: 0.3049.• Z = 3.992, p < .001• = 0.776 (expect a narrowing of the gap)•We shall see that this is the largest estimate
of the treatment effect.
8
9
CfB: Measurement Error in the Pretest
Coefficient alpha of .872 for pretestLord-Porter Correction
Convert (.872 - .2112)/( 1 - .2112) = .866
Adjusted pretest score (MT is the mean for the Treated and MC is the mean for the controls):
(X1 – MT) + MT
(X1 – MC) + MC
b = 0.2569, Z = 3.3308, p < .001
10
11
Williams & Hazer Method
Set X1 = X1T + E1
Fix the variance of E1 to (1 - )sY12
or (1 - .872)(0.614) = 0.079. b = 0.2543, Z = 3.233, p = .001
(with = .897)Same estimates of b and as Lord-
Porter (standard error a bit different)!
12
13
Controlling for Baseline: Latent Variables
–b = 0.3256, Z = 4.124, p < .001– = 0.816 (surprisingly relatively
low)–Cannot directly compare estimates
to the univariate analysis.
14
15
Change Score Analysis: Univariate–All 3 methods (see next 3 slides) show a
beneficial effect:• 0.2105, Z = 2.619, p = .009
–Smallest effect of any analysis,–Note that the Treateds improve (0.0874),
and the Controls decline (-0.1231).
16
Raykov
17
LCS
18
Kenny-Judd
19
Change Score Analysis: Latent Variables
–All 3 methods show a beneficial effect 0.2435, Z = 2.964, p < .003
–Again, you cannot directly compare the univariate and latent variable results.
–Smallest effect of any latent variable analysis.
20Raykov
21LCS
22
KJ
23
Standardized Change Score Analysis: Univariate Analysis
Residual variance decreases slightly over time (but not significantly, p = .31)
•Time 1: 0.59•Time 2: 0.52
Effect: .3078, Z = 2.775 , p = .006Recalibrated to units of Time 2: 0.2266 , Z = 2.826 , p = .005.
24SCSA
25SCSA-Y2
26
Standardized Change Score Analysis: Latent Variables
Residual variance decreases slightly over time (but not significantly, p = .30)
•Time 1: 0.56•Time 2: 0.52
b = 0.3653, Z = 3.116 , p = .002Units of Time 2: b = 0.2627, Z = 3.194 , p = .001
27
SCSA
28SCSA-Y2
29
Summary of Univariate Effects
CfB: 0.3049CfB with Reliability Correction:
0.2543CSA: 0.2105SCSA: 0.2266
30
Summary of Latent Variable Effects
CfB: 0.3256CSA: 0.2435SCSA: 0.2627
31
What Estimate Would I Report?
CSA Latent Variable: 0.2435(Z = 2.964, p < .003) No reason to think that the factors that created the Time 1 difference to change. Note too the variance does not change.Others might respectfully disagree.