discussion overview: measurement i) reliability of measures i) reliability of measures ii) construct...

Discussion Overview: Discussion Overview: MeasurementMeasurement

I) Reliability of MeasuresI) Reliability of Measures

II) Construct ValidityII) Construct Validity

III) Measurement scalesIII) Measurement scales

I) Reliability of MeasuresI) Reliability of Measures ReliabilityReliability

– The consistency or stability of a measure The consistency or stability of a measure Assessing a restaurant’s foodAssessing a restaurant’s food

Three important variablesThree important variables– How many testers? (Observers)How many testers? (Observers)

Interrater reliabilityInterrater reliability– How many different entrees? (Observations)How many different entrees? (Observations)

Internal consistencyInternal consistency– How many times? (Occasions)How many times? (Occasions)

Test-retestTest-retest

Interrater ReliabilityInterrater Reliability

The degree to which The degree to which independent raters independent raters agree on an observationagree on an observation

Have two (or more) Have two (or more) judges rate the same judges rate the same peoplepeople

Trained and Trained and independent raters, independent raters, using a coding schemeusing a coding scheme

Observer 1 Observer 2

Complain about injection

-2 3

First negative comment

0 1

Second negative comment

-2 2

Rip up questionnaire -2 3


Observer 1 Observer 2

Complain about injection

22 22

First negative comment

00 00

Second negative comment

-2-2 -2-2

Rip up questionnaire 22 33


Internal ConsistencyInternal Consistency

Internal consistencyInternal consistency – the degree to – the degree to which all specific items of a measure which all specific items of a measure behave the same waybehave the same way

Measure the same people with Measure the same people with multiple itemsmultiple items– Different questions in a surveyDifferent questions in a survey– Different behaviors in observationDifferent behaviors in observation

ExtraversionExtraversion

1 2 3 4 5

Not at all

true

Very true

1.I am outgoing. ____

2.I am friendly. ____

3.I am talkative. ____

4.I am gregarious.____

Internal consistencyInternal consistency

Split-half reliabilitySplit-half reliability – correlation of – correlation of scores on one half of the test with scores on one half of the test with scores on the other halfscores on the other half

Cronbach’s alphaCronbach’s alpha – the average of all – the average of all possible correlations between itemspossible correlations between items

‘‘One of these things just One of these things just doesn’t belong’doesn’t belong’

One of these things is not like the others, One of these things is not like the others, One of these things just doesn't belongOne of these things just doesn't belong

Student 1 Student 2 Student 3

Ques 1 Ques 1 (Chpt 12)(Chpt 12)

1010 22 99


99 33 88


22 66 11


1010 22 99

Test-Retest ReliabilityTest-Retest Reliability The degree to which a measure correlates The degree to which a measure correlates

positively with itself over timepositively with itself over time– Consistency of the measure over timeConsistency of the measure over time

Measure the same people at two (or more) Measure the same people at two (or more) points in timepoints in time

Desirable for stable traits, but not for transient Desirable for stable traits, but not for transient statesstates

The “More is Better Rule”The “More is Better Rule”

Reliability is likely to increase as we Reliability is likely to increase as we increase the number of…increase the number of…– Observers (or raters)Observers (or raters)– Observations (or items)Observations (or items)– OccasionsOccasions

Measurement error will average outMeasurement error will average out

II) Construct Validity II) Construct Validity

How well an How well an operational definition operational definition represents the represents the construct of interestconstruct of interest

The degree to which The degree to which the construct can be the construct can be inferred from the inferred from the operational definition operational definition of that constructof that construct

Indicators of Construct Indicators of Construct ValidityValidity

Face validityFace validity Criterion validityCriterion validity

– Predictive validityPredictive validity– Concurrent validityConcurrent validity– Convergent validityConvergent validity– Discriminant validityDiscriminant validity

Face ValidityFace Validity

Face validityFace validity – Does the measure – Does the measure appear to measure the construct of appear to measure the construct of interest?interest?– Does the measure “on the face of it” Does the measure “on the face of it”

look like what it’s supposed to look like what it’s supposed to measure?measure?

Not necessary or sufficient for a Not necessary or sufficient for a good measuregood measure

Predictive ValidityPredictive Validity

Predictive validityPredictive validity – Is the measure – Is the measure associated with variables it should associated with variables it should theoretically predict?theoretically predict?

LSAT – Law school performanceLSAT – Law school performance Self-esteem – DepressionSelf-esteem – Depression Shyness – Social anxiety Shyness – Social anxiety

Concurrent ValidityConcurrent Validity

Concurrent validityConcurrent validity – Does the – Does the measure differ between groups it measure differ between groups it ought to differ between?ought to differ between?– Also called “known groups validity”Also called “known groups validity”

E.g., clinically depressed versus non-E.g., clinically depressed versus non-depressed groupsdepressed groups

Convergent ValidityConvergent Validity

Convergent validityConvergent validity – Is the – Is the measure associated with other measure associated with other established measures of the same established measures of the same construct?construct?

Self-report - ObservationsSelf-report - Observations Physiological measure - Self-reportPhysiological measure - Self-report Self-report 1 – Self-report 2Self-report 1 – Self-report 2

Discriminant ValidityDiscriminant Validity

Discriminant validityDiscriminant validity – Is the – Is the measure NOT associated with measure NOT associated with measures of other constructs?measures of other constructs?

Self-esteem scores not associated Self-esteem scores not associated with locus of control scoreswith locus of control scores

Problem solving knowledge not Problem solving knowledge not associated with factual knowledgeassociated with factual knowledge

Measurement Reliability & Measurement Reliability & ValidityValidity

ReliabilityReliability: Is the measure consistent?: Is the measure consistent? ValidityValidity: Does the measure : Does the measure

adequately reflect the construct of adequately reflect the construct of interest?interest?

Reliable and Valid Reliable, not Valid Not Reliable, not Valid

Relationship between Relationship between Reliability and ValidityReliability and Validity

Can be reliable but not validCan be reliable but not valid To be valid it must be reliable

– But reliability is not sole condition for validity

Both reliability and validity are necessary for accurate measurement in a research study.

Measurement Scales Measurement Scales

Nominal scalesNominal scales Ordinal scalesOrdinal scales Interval scalesInterval scales Ratio scalesRatio scales

Nominal ScalesNominal Scales AKA Categorical scalesAKA Categorical scales No numerical/quantitative properties. No numerical/quantitative properties.

Categories or group simply differ from Categories or group simply differ from one anotherone another

Examples:Examples:– Men or womenMen or women– Right or left handedRight or left handed– Catholic, Protestant, Jewish, Hindu, Catholic, Protestant, Jewish, Hindu,

Buddhist…Buddhist…– Numbers on basketball jerseysNumbers on basketball jerseys– Zip codesZip codes

Ordinal ScalesOrdinal Scales

Allow us to rank order the levels of Allow us to rank order the levels of the variables being studiedthe variables being studied

ExamplesExamples– Social classSocial class

lower class, working class, middle class, lower class, working class, middle class, and upper classand upper class

– College football standingsCollege football standings– Letterman’s Top TenLetterman’s Top Ten

Top Ten Questions to ask Yourself Top Ten Questions to ask Yourself Before Eating Spinach?Before Eating Spinach?

10.10. Was my spinach properly sprayed with Lysol? Was my spinach properly sprayed with Lysol? 9.9. Isn't it still safer than eating a New York City Isn't it still safer than eating a New York City

hot dog?hot dog? 8.8. So all those years my mom made me eat So all those years my mom made me eat

spinach, she was trying to kill me?spinach, she was trying to kill me? 7.7. Is this the right side dish for my Mad Cow Is this the right side dish for my Mad Cow

burger?burger? 6.6. Are my papers in order? Are my papers in order? 5.5. If I get sick, will my wife TiVo Ventriloquist If I get sick, will my wife TiVo Ventriloquist

Week on the Late Show?Week on the Late Show? 4.4. Should I also avoid kale? Should I also avoid kale? 3.3. If I'm going to eat something deadly, shouldn't If I'm going to eat something deadly, shouldn't

it be delicious Pop-Tarts?it be delicious Pop-Tarts? 2.2. What would Popeye do? What would Popeye do? 1.1. Do I really want my obituary to read: "Man Dies Do I really want my obituary to read: "Man Dies

A La Florentine?"A La Florentine?"

Interval ScalesInterval Scales

The difference between the numbers The difference between the numbers on the scale is meaningfulon the scale is meaningful

Scores separated by equal intervalsScores separated by equal intervals ExamplesExamples

– Temperature (Fahrenheit or Celsius)Temperature (Fahrenheit or Celsius)– Scores on personality measureScores on personality measure

Ratio ScalesRatio Scales

Scores separated by Scores separated by equal intervals and equal intervals and there is an absolute there is an absolute zerozero

ExamplesExamples– LengthLength– Weight Weight – TimeTime– Number of responsesNumber of responses

LevelLevel

QualitativeInfo

Has inherent order

‘more to less’

EqualIntervals

Has zero point

Nominal XX

Ordinal XX XX

Interval XX XX XX

Ratio XX XX XX XX

Scales of MeasurementScales of Measurement

Concept Check Concept Check

Which scale of measurement best Which scale of measurement best describes the following:describes the following:– Telephone numbersTelephone numbers– Distances from Budapest to cities in the USDistances from Budapest to cities in the US– Scores on an extraversion personality Scores on an extraversion personality

assessmentassessment– Ranking of basketball teams in the Big TenRanking of basketball teams in the Big Ten

discussion overview: measurement i) reliability of measures i) reliability of measures ii) construct...

Documents