overlooking stimulus variance
DESCRIPTION
Overlooking Stimulus Variance. Jake Westfall University of Colorado Boulder Charles M. Judd David A. Kenny University of Colorado BoulderUniversity of Connecticut. Cornfield & Tukey (1956): “The two spans of the bridge of inference”. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/1.jpg)
Overlooking Stimulus Variance
Jake WestfallUniversity of Colorado Boulder
Charles M. Judd David A. KennyUniversity of Colorado Boulder University of Connecticut
![Page 2: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/2.jpg)
Cornfield & Tukey (1956):“The two spans of the bridge of inference”
![Page 3: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/3.jpg)
My actual samples
50 University of Colorado undergraduates;40 positively/negatively valenced English adjectives
![Page 4: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/4.jpg)
Ultimate targets of generalization
My actual samples
All healthy, Western adults; All non-neutral visual stimuli
50 University of Colorado undergraduates;40 positively/negatively valenced English adjectives
![Page 5: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/5.jpg)
Ultimate targets of generalization
My actual samples
All healthy, Western adults; All non-neutral visual stimuli
All CU undergraduates takingPsych 101 in Spring 2014;All short, common, stronglyvalenced English adjectives
50 University of Colorado undergraduates;40 positively/negatively valenced English adjectives
All potentially sampled participants/stimuli
![Page 6: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/6.jpg)
Ultimate targets of generalization
My actual samples
All healthy, Western adults; All non-neutral visual stimuli
“Subject-matter span”
“Statistical span”
50 University of Colorado undergraduates;40 positively/negatively valenced English adjectives
All potentially sampled participants/stimuli
![Page 7: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/7.jpg)
Difficulties crossing the statistical span• Failure to account for uncertainty associated with
stimulus sampling (i.e., treating stimuli as fixed rather than random) leads to biased, overconfident estimates of effects
• The pervasive failure to model stimulus as a random factor is probably responsible for many failures to replicate when future studies use different stimulus samples
![Page 8: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/8.jpg)
Doing the correct analysis is easy!
• Modern statistical procedures solve the statistical problem of stimulus sampling
• These linear mixed models with crossed random effects are easy to apply and are already widely available in major statistical packages– R, SAS, SPSS, Stata, etc.
![Page 9: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/9.jpg)
Illustrative Design• Participants crossed with Stimuli
– Each Participant responds to each Stimulus • Stimuli nested under Condition
– Each Stimulus always in either Condition A or Condition B• Participants crossed with Condition
– Participants make responses under both Conditions
Sample of hypothetical dataset:
5 4 6 7 3 8 8 7 9 5 6 5
4 4 7 8 4 6 9 6 7 4 5 6
5 3 6 7 4 5 7 5 8 3 4 5
![Page 10: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/10.jpg)
Typical repeated measures analyses (RM-ANOVA)
MBlack MWhite Difference
5.5 6.67 1.17
5.5 6.17 0.67
5.0 5.33 0.33
5 4 6 7 3 8 8 7 9 5 6 5
4 4 7 8 4 6 9 6 7 4 5 6
5 3 6 7 4 5 7 5 8 3 4 5
How variable are the stimulus ratings around each of the participant means? The variance is lost due to the aggregation
“By-participant analysis”
![Page 11: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/11.jpg)
Typical repeated measures analyses (RM-ANOVA)
5 4 6 7 3 8 8 7 9 5 6 5
4 4 7 8 4 6 9 6 7 4 5 6
5 3 6 7 4 5 7 5 8 3 4 5
4.00 3.67 6.33 7.33 3.67 6.33 8.00 6.00 8.00 4.00 5.00 5.33
Sample 1 v.s. Sample 2
“By-stimulus analysis”
![Page 12: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/12.jpg)
Simulation of type 1 error rates for typical RM-ANOVA analyses
• Design is the same as previously discussed• Draw random samples of participants and stimuli– Variance components = 4, Error variance = 16
• Number of participants = 10, 30, 50, 70, 90• Number of stimuli = 10, 30, 50, 70, 90• Conducted both by-participant and by-stimulus
analysis on each simulated dataset• True Condition effect = 0
![Page 13: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/13.jpg)
![Page 14: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/14.jpg)
Type 1 error rate simulation results• The exact simulated error rates depend on the
variance components, which although realistic, were ultimately arbitrary
• The main points to take away here are:1. The standard analyses will virtually always show
some degree of positive bias2. In some (entirely realistic) cases, this bias can be
extreme3. The degree of bias depends in a predictable way on
the design of the experiment (e.g., the sample sizes)
![Page 15: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/15.jpg)
The old solution: Quasi-F statistics• Although quasi-Fs successfully address the
statistical problem, they suffer from a variety of limitations– Require complete orthogonal design (balanced factors)– No missing data– No continuous covariates– A different quasi-F must be derived (often laboriously)
for each new experimental design – Not widely implemented in major statistical packages
![Page 16: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/16.jpg)
The new solution: Mixed models• Known variously as:– Mixed-effects models, multilevel models, random
effects models, hierarchical linear models, etc.• Most psychologists familiar with mixed models
for hierarchical random factors– E.g., students nested in classrooms
• Less well known is that mixed models can also easily accommodate designs with crossed random factors– E.g., participants crossed with stimuli
![Page 17: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/17.jpg)
![Page 18: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/18.jpg)
Grand mean = 100
![Page 19: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/19.jpg)
![Page 20: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/20.jpg)
MeanA = -5 MeanB = 5
![Page 21: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/21.jpg)
![Page 22: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/22.jpg)
ParticipantMeans5.86
7.09
-1.09
-4.53
![Page 23: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/23.jpg)
![Page 24: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/24.jpg)
Stimulus Means: -2.84 -9.19 -1.16 18.17
![Page 25: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/25.jpg)
![Page 26: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/26.jpg)
ParticipantSlopes3.02
-9.09
3.15
-1.38
![Page 27: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/27.jpg)
![Page 28: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/28.jpg)
Everything else = residual error
![Page 29: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/29.jpg)
![Page 30: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/30.jpg)
The linear mixed-effects modelwith crossed random effects
Fixed effects Random effects
![Page 31: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/31.jpg)
Fitting mixed models is easy: Sample syntaxlibrary(lme4)model <- lmer(y ~ c + (1 | j) + (c | i))
proc mixed covtest;class i j;model y=c/solution;random intercept c/sub=i type=un;random intercept/sub=j;run;
MIXED y WITH c /FIXED=c /PRINT=SOLUTION TESTCOV /RANDOM=INTERCEPT c | SUBJECT(i) COVTYPE(UN) /RANDOM=INTERCEPT | SUBJECT(j).
R
SAS
SPSS
![Page 32: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/32.jpg)
Mixed models successfully maintain the nominal type 1 error rate (α = .05)
![Page 33: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/33.jpg)
Conclusion• Stimulus variation is a generalizability issue• The conclusions we draw in the Discussion sections
of our papers ought to be in line with the assumptions of the statistical methods we use
• Mixed models with crossed random effects allow us to generalize across both participants and stimuli
![Page 34: Overlooking Stimulus Variance](https://reader036.vdocuments.mx/reader036/viewer/2022081503/56816140550346895dd0add7/html5/thumbnails/34.jpg)
The end
Further reading:Judd, C. M., Westfall, J., & Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely
ignored problem. Journal of personality and social psychology, 103(1), 54-69.