Misuses of Misuses of Quantitative Quantitative

Research Research

V. Darleen OpferV. Darleen Opfer

You can prove anything with You can prove anything with statisticsstatistics

VersusVersus

There are three kinds of lies: There are three kinds of lies: lies, damned lies, and lies, damned lies, and

statisticsstatistics

Competing Notions of the Use of Competing Notions of the Use of Quantitative DataQuantitative Data

Statistical PitfallsStatistical Pitfalls

Sources of biasSources of bias Errors in methodErrors in method Problems with interpretationProblems with interpretation

Sources of biasSources of bias

Representative samplingRepresentative sampling Statistical assumptionsStatistical assumptions

Gender & Achievement on Gender & Achievement on GCSEsGCSEs

GCSEGrades

1997 2000 Difference in Percentage

Points between

years (not between genders)

Percent Change

Boys Girls Boys Girls

Boys Girls Boys Girls

A*-C 39 49 43 53 4 4 10 8

Girls performance has increased more Girls performance has increased more rapidly than that of boys.rapidly than that of boys. Speed, 1998 Speed, 1998

Female students have improved their Female students have improved their performance markedly, whereas male performance markedly, whereas male students have not shown a similar students have not shown a similar improvement.improvement. Arnot, 1996 Arnot, 1996

There are increasing gender differences There are increasing gender differences in performance.in performance. Stobart, 1992 Stobart, 1992

Errors in MethodErrors in Method

Multiple comparisonsMultiple comparisons Measurement errorMeasurement error

Problems with interpretationProblems with interpretation

Confusion over significanceConfusion over significance Precision and accuracyPrecision and accuracy CausalityCausality Unequal comparisonsUnequal comparisons

Definitions of School EffectsDefinitions of School Effects School effects as the absolute effects of schooling, using School effects as the absolute effects of schooling, using

naturally occurring “control” groups of students who naturally occurring “control” groups of students who receive no schoolingreceive no schooling

School effects as the unadjusted average achievement of School effects as the unadjusted average achievement of all students in a schoolall students in a school

School effects as the impact of schooling on the average School effects as the impact of schooling on the average achievement of all students in a school, adjusted for family achievement of all students in a school, adjusted for family background and/or prior achievementbackground and/or prior achievement

School effects as measuring the extent of ‘between schools’ School effects as measuring the extent of ‘between schools’ variation in the total variation of their students’ individual variation in the total variation of their students’ individual test scorestest scores

School effects as measuring the unique effect of each School effects as measuring the unique effect of each school on their students’ outcomes school on their students’ outcomes

School effects as measuring the impact of schools on School effects as measuring the impact of schools on student performance over timestudent performance over time

What is Good Statistical Practice?What is Good Statistical Practice? Be sure your sample is representative of the Be sure your sample is representative of the

population in which you’re interestedpopulation in which you’re interested Be sure you understand the assumptions of your Be sure you understand the assumptions of your

statistical procedures and be sure they are statistical procedures and be sure they are satisfiedsatisfied

Be sure the sample size is not misleading you Be sure the sample size is not misleading you about the amount of significanceabout the amount of significance

Be sure to use the best measurement/collection Be sure to use the best measurement/collection toolstools

Be aware of impact of multiple comparisons and Be aware of impact of multiple comparisons and the lack of an a priori analysis planthe lack of an a priori analysis plan

Be clear about what you’re trying to discoverBe clear about what you’re trying to discover Use numerical notation in a rational wayUse numerical notation in a rational way Be sure you understand the conditions for causal Be sure you understand the conditions for causal

inferenceinference Be explicit about the measures you are comparingBe explicit about the measures you are comparing