Measurement in Psychology:Validity

Lawrence R. Gordon

Psychology Research Methods I

You have a reliable measure…

It gives you a consistent measure of something every time that you measure it

SO WHAT???Is that all a measure needs to do?

Validity

NOTE: a scale CAN be reliable but not valid; it CANNOT be unreliable scale, but be valid!

In short, a measure has to measure what it claims to measure (or what you claim it measures)

How can you show others that you are measuring what you say you are measuring?

Example

SAT’s and College Potential

How can we show that SAT’s are actually measuring college potential?

Types of ValidityContent/Face validity

does it look like it is measuring what it should?• Examples (Polygraph measures lying through GSR,

EEG, heart rate? Reaction time measured with a stopwatch?)

Does the content make sense in measuring the construct?

Content/Sampling Validity Where did the content come from?

• Examples (Why do we need to know insane vocabulary words and geometry on the SAT’s? Who wrote those questions?)

Types of ValidityCriterion validity

validity related to an outcome concurrent validity (Does it relate to other

variables the way the theory says it should?)• Example (Does the SAT relate to Achievement

Tests? IQ?) predictive validity (Does it predict something

related to the construct?)• Examples (Does the SAT predict first-year college

GPA? Does the SAT predict GRE scores? Does it “postdict” HS class rank?)

Types of Validity

Discriminant validity Does the measure fail to relate to measures of

other constructs?• Examples (Does the SAT relate to measures of SES

or self-esteem?)

Types of Validity

Construct validity Does the measure really measure the construct

it says it does? Is the construct (and its operational definition)

appropriate for the research? Examples (Does the SAT measure college

potential? Is this an appropriate measure for college potential?)

Old-fashioned RacismOld-fashioned racism is the overt, blatant

expression of racist attitudesIt’s easy to measure - ask questions like

“are you racist?” “do you like African-Americans?” “would you want African-Americans to attend

the same schools as your children?”

scale has what type of validity?

Old-fashioned Racism (cont.)Problems with “old-fashioned racism”,

didn’t relate to subsequent voting behavior (what type of validity lacking?)

people were not reporting that they were racist anymore

• 80% of White Americans were reporting that they were not racist at all (Dovidio & Gaertner, 1991)

Where do we go from here?

Validation of a MeasureSubtle measure of racismUse of political attitudes rather than direct

inquiry (not quite the same thing!)Modern Racism Scale (McConahay,

Hardee, & Batts, 1981) How much do you agree that:

• “over the past few years, Blacks have gotten more economically than they deserve”?

• “discrimination against Blacks is no longer a problem in the United States”?

Validation of a Measure (cont.)

MRS needed to show that it was reliable and valid

How could it show that it was reliable?How could it show that it was valid?

Modern Racism Scale (0-28)

20.017.515.012.510.07.55.02.50.0

Modern Racism Scale, F 2001

Fre

qu

en

cy80

60

40

20

0

Std. Dev = 3.85

Mean = 3.5

N = 190.00

Modern Racism Scale (0-28)

20.017.515.012.510.07.55.02.50.0

Modern Racism Scale, Fall 2002

Fre

qu

en

cy70

60

50

40

30

20

10

0

Std. Dev = 4.09

Mean = 4.1

N = 183.00

Modern Racism Scale

INTERNAL CONSISTENCY RELIABILITY (Homogeneity)

RELIABILITY ANALYSIS - SCALE (ALPHA)N of Cases = 183.0 Reliability Coefficients N of Items = 7Alpha = .7834

SPLIT-HALF RELIABILITY (Consistency)

RELIABILITY ANALYSIS - SCALE (SPLIT)N of Cases = 183.0Reliability Coefficients N of Items = 7Correlation between forms = .5993Unequal-length Spearman-Brown = .7525 4 Items in part 1 / 3 Items in part 2

Validation of a Measure (cont.)

Validity of MRS Face validity?

• Survey says...

Not Racism Related

5%

Missing24%

Racism-related71%

MRS FACE VALIDITY

Validation of a Measure (cont.)

Validity of MRS Face validity?

• Survey says... Criterion validity?

• Concurrent validity?

• Predictive validity?– McConahay (1983) - tell me about it...

Validation of a Measure (cont.)

Validity of MRS Face validity?

• Survey says... Criterion validity?

• Concurrent validity?

• Predictive validity?– McConahay (1983) - tell me about it...

Discriminant validity?

DISCRIMINANT VALIDITY OF THE MRS

Social Desirability (Marlowe-Crown)

56545250484644424038

Mo

de

rn R

aci

sm S

cale

20

10

0

Validation of a Measure (cont.)

Validity of MRS Construct validity?

Since its creation, the MRS has been probably the most widely used measure of racism in psychology

What do you think about its continued validity?

McConahay -- Major Result

-0.6-0.4

-0.2

0

0.2

0.4

0.6

0.8

r

White Black

Correlations Relating MRS to Willingness to Hire by Race of candidate and Decision Context

NegativePositive

Construct Validity -- Your Data and Civil Unions

How does the Attitudes to Civil Union scale do?

No reliability info, but similar scales work well Face validity? Concurrent validity? Correlation with MRS? Discriminant validity? Correlation with Soc Desirability CONSTRUCT

• which sex more “tolerant”?• which political party?• which type of hometown?• Related to weight? Height? GPA?