reliability & validity the bada & bing of your tailored survey design

Reliability & Validity

the

Bada

&

Bing

of YOUR tailored survey design

• This presentation has been influenced

not at all

a little bit

CONSIDERABLY

by the work & wisdom of Dan Koretz.

Thanks!

Pre-Presentation credit

Core concept of validity• You wish to measure a construct, but can never know true score for sure(e.g. 6th grade math proficiency, self esteem)

• You must draw an inference about the construct based on a sample or indicator of behavior--something you can actually “touch”

(e.g., 6th grade math test, self esteem survey)

• VALIDITY describes how well performance on your indicator justifies your inference about the construct

Error: Validity’s arch nemesis

• Sampling error: occurs from sampling units of observation (i.e, populations of humans)

• Measurement error (M.E.): occurs across instances of measurement from

each unit (i.e. individual humans)

More on measurement error

• Outcomei = True score + M.E. • Measurement error can be… systematic (does not “wash out” across repeated measurements) or random (does “wash out” across repeated measurements)

Systematic error

• Chris repeatedly takes a self esteem survey written in Latin

Outcome repeatedly affected by factor not relevant to construct being measured

“Construct irrelevant variance”

**Note: the only Latin that Chris speaks involves inesway**

Systematic error cont’d

• “Construct Under-representation”

If construct is poorly represented, repeated measurements will not converge on true score with regard to entire construct

e.g. global self esteem survey asks only about Chris’ confidence on golf course

Random error

• Chris repeatedly takes self esteem survey

Sometimes mood =

Sometimes mood =

• Over time, outcomes will converge on Chris’ true score

Some details• Validity is an attribute of your inference, not

the instrument itself, which may be more valid for some inferences and/or populations than others (e.g. self-esteem survey in Latin).

• Validity is not an all or nothing phenomenon, but a matter of degree, and we must piece together evidence suggesting how valid our inferences may or may not be.

Types of validity/validity evidencenote: other terms exist, but this will be the

focus of S-015

Content Validity Convergent Evidence Discriminant Evidence

Construct Validity

Types of validity/validity evidence

Construct Validity:

• How well does performance on our instrument justify inferences about the construct?

• “Validity”

• What we’re ultimately shooting for

Types of validity/validity evidence

Content based evidence(a/k/a content validation study): •Compare your instrument to your very

thoroughly defined construct… • Does the instrument adequately

represent the construct? • Harder than it seems (constructs can be messy)

Types of validity/validity evidence

Convergent-discriminant evidence: •Measures of similar constructs should

converge. •Measures of less similar constructs

should diverge.(e.g. Two math tests should correlate

more strongly than a math and reading test)

Multitrait-Multimethod Matrix(MTMM)

• A fun way to display convergent-discriminant validity (or not)

Pass out hand outs: Now

But alas, complications abound…

• What constitutes “similar”

• What constitutes “less similar”

• What constitutes “convergence”

• What constitutes “divergence”

????

Plausible toy correlations

Math 1

Math 1 1.00

Math 2 .82

Read 1 .74

Read 2 .70

Closing thoughts on validity

• We must piece together evidence that is often murky and incomplete to reach judgment.

• An instrument that is fairly valid for one use, inference, or population may not be valid for others.

Oh, BTW…

• The more reliable your instrument, the better your chance of drawing fairly valid inferences.

(Old Faithful)

Core concept of reliability

• Reliability is consistency of results across repeated measurements

(e.g. assuming no interventions or natural attitudinal shifts in between, a subject taking a highly reliable survey would perform quite similarly each time s/he took it.)

Some details

• Reliability is also a matter of degree, often expressed as a coefficient ranging from 0 - 1.

• A test or survey may be more reliable for some populations than others

(e.g. surveys tend to be more reliable among older/more educated populations.)

POP QUIZ

• True or false…

1.) An instrument that allows us to draw a

reasonably valid inference must

be reasonably reliable

2.) A reasonably reliable instrument must allow

us to draw a reasonably valid inference

POP QUIZ cont’d

Regarding R & V, how might one describe…

A few (of many) ways to assess reliability

• Assess internal consistency

Assuming a survey taps one and only one construct, the results from the first half should correlate highly with results from the second half; the odd items should correlate highly with the evens, etc. (split-half correlations)

Coefficient Alpha

• a/k/a Cronbach’s alpha

The average of all possible split-half correlations in a given sample

generally preferred to single split-half correlation

A few (of many) ways to assess reliability

• Test-retest

Assuming no interventions or natural shifts in attitude, a reasonably reliable survey will yield similar results from the same person across repeated administrations

WARNING

• A test or survey with a high reliability coefficient does not guarantee that your results will be highly reliable.

(e.g. Differences in administrative conditions can effect the consistency of your results across repeated administrations.

Questions???

References, etc

Linn, R.L. & Gronlund, N. E. (2000). Measurement and Assessment in Teaching, 8th Ed. New Jersey: Prentice-Hall, Inc.

See diagram displayed on page 75 of the reference textbook.

A very cool link that covers a TON of stuff on all forms of social research…

http://www.socialresearchmethods.net/kb/index.php