psy 1950 outliers, missing data, and transformations september 22, 2008
DESCRIPTION
PSY 1950 Outliers, Missing Data, and Transformations September 22, 2008. On Suspecting Fishiness Looking for outliers, gaps, and dips e.g., tests of clairvoyance When gaps or dips are hypothesized e.g., is dyslexia a distinct entity Cliffs - PowerPoint PPT PresentationTRANSCRIPT
PSY 1950Outliers, Missing Data, and
TransformationsSeptember 22, 2008
On Suspecting Fishiness• Looking for outliers, gaps, and dips
– e.g., tests of clairvoyance• When gaps or dips are hypothesized
– e.g., is dyslexia a distinct entity• Cliffs
– e.g., differences between rating of ingroup and outgroup
• Peaks– e.g., the blackout and baby boom
• The occurrence of impossible scores
Visualize your data!• “make friends with your data”
– Rosenthal• “don’t becomes lovers with your
data”– Me
• Statistics condense data• View raw data graphically
– Frequency distribution graphs– Scatter plots
Outliers• Extreme scores• Come from samples other than
those of interest• Can lead to Type I and II
errors
Outlier Detection• Graph
– Box plots– Scatter plots
• Numerical criterion– Extremity (central tendency +/- spread)
• Outside fences– lower: Q1 - 3(Q3 - Q1)– upper: Q3 + 3(Q3 - Q1)
• z-score
– Probability (Extremity + # measurements)• Chauvenat’s/Peirce’s criterion, Grubb’s test
– Absolute cutoff
Outlier Analysis• Determine nature of impact
– Quantitative• Changes numbers, not inferences
– Qualitative• Changes numbers and inferences
• Consider source of outlier– Quantitative
• Same underlying mechanism/sample
– Qualitative• Different underlying mechanisms/samples
– e.g., digit span = 107, simple RT = 1200 ms
Outlier Coping• Options
– Retain– Remove– Reduce
• Windsorize• Normalizing transformation
• Considerations– Impact/Source– Convention– Believability
• Justification• Replication
QuickTime™ and a decompressor
are needed to see this picture.
Transformations• Linear “rescaling”
– unit conversion•e.g., # items correct, # items wrong •e.g., standardization
• Curvilinear “reexpression”– variable conversion
•e.g., time (sec/trial) to speed (trials/sec)
•e.g., normalization
Standardization• Why standardize data?
– Intra-distribution statistics• You got 8 questions wrong on one exam• You were one standard deviation below
the mean
– Inter-distribution statistics• You got 8 questions wrong on the
midterm and 5 questions wrong on the final
• Aggregation: Overall, you were one standard deviation below the mean
• Comparison: You did better on the midterm than the final
z-score• # standard deviations
above/below the mean
raw-score z-score IQ-scale20 -2.48 75.225 -1.01 89.925 -1.01 89.926 -0.71 92.927 -0.42 95.827 -0.42 95.827 -0.42 95.828 -0.12 98.828 -0.12 98.829 0.17 101.730 0.47 104.731 0.76 107.631 0.76 107.631 0.76 107.632 1.06 110.633 1.35 113.533 1.35 113.5
M 28.41 0.00 100.0SD 3.39 1.00 10.0
Test Performance
Normal Distributions• “…normality is a myth; there
never was, and never will be, a normal distribution.”– Geary (1947)
• “Experimentalists think that it is a mathematical theorem while the mathematicians believe it to be an experimental fact.”– Lippman (1917)
Normalization• Why normalize DV?
– Meet statistical assumption of normality in situations when it matters• Small n• Unequal n• One-sample t and z tests
– Increase power• Why NOT normalize DV?
– Interpretability– Affects measurement scale
Tests of Normality• Frequency distribution• Skew/kurtosis statistics• Kolmorogov-Smirnov test • Probability plots (e.g., P-P
plot)QuickTime™ and a
TIFF (Uncompressed) decompressorare needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
Types of Curvilinear Transformations
Does normalization help?• Games & Lucas (1966): Normalizing
transformations hurt– Reduce interpretability, power
• Levin & Dunlap (1982): Transformations help– Increase power
• Games (1983): It Depends, Levin and Dunlap are stupid
• Levine & Dunlap (1984): It depends, Games is stupid
• Games (1984): This debate is stupid
Does non-normality hurt?
Normalize If and Only If• It matters
– In theory: Got robust?– In practice: Got change?
• Must assume normality (i.e., no non-parametric test available)
Missing Data
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Why are they missing?– MCAR
• Variable’s missingness unrelated to both its value and other variables’ values
• e.g., equipment malfunction• No bias
– MAR• Variable’s missingness unrelated to its value after
controlling for its relation to other variables• e.g., depression and income• Bias
– MNAR• Variable’s missingness related to its value after
controlling for its relation to other variables• e.g., income reporting• Bias
Diagnosing Missing Data• How much?• How concentrated?• How essential?• MCAR, MAR, MNAR?• How influential?
Dealing with Missing Data– Treat missing data as data– Note bias
• “lower income individuals are underrepresented”
– Delete variables– Delete cases
• Listwise• Casewise
– Estimation• Prior knowledge• Mean substitution• Regression substitution• Expectation-maximization (EM)• Hot decking• Multiple imputation (MI)
Missing Data: Conclusions• Avoid missing data!• If rare (<5%), MCAR,
nonessential, concentrated, or impotent, delete appropriately
• If frequent, patterned, essential, diffuse, influential, use MI
• If MNAR, treat missingness as DV
• Question: What’s the best method for identifying and removing RT outliers?
• Alternatives– RT cutoff (5 values)– z-score cutoff (1, 1.5)– Transformation (log, inverse)– Trimming– Medians– Windsorizing (2 SD)
Method• Conduct series of simulations
– DV: power (# sig simulations/1000)• 2 x 2 ANOVA
– One main effect (20, 30, 40 ms)• 7 observations/condition
– 10% outlier probability– Outliers 0-2000 ms
• 32 participants• Between-participants variability
SpreadDrift
ex-Gaussian distribution
Inferences• Absolute cutoffs resulted in greatest
power• Best cutoff values depended on type
of effect– Shift: 10-15% cutoff– Spread: 5% cutoff
• Inverse transformation good, too• With high between-participant
variability, SD cutoff becomes effective
Recommendations• Try range of cutoffs to
examine robustness • Replicate with inverse
transformation (or SD cutoff)• Replicate novel, unexpected,
or important effects• Choose method before
analyzing data