lecture overview: measurement 1) reliability of measures 1) reliability of measures 2) construct...

28
Lecture Overview: Lecture Overview: Measurement Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3) Measurement scales

Post on 18-Dec-2015

239 views

Category:

Documents


4 download

TRANSCRIPT

Page 1: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Lecture Overview: Lecture Overview: MeasurementMeasurement

1) Reliability of Measures1) Reliability of Measures

2) Construct Validity2) Construct Validity

3) Measurement scales3) Measurement scales

Page 2: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

1) Reliability of Measures1) Reliability of Measures ReliabilityReliability

– The consistency or stability of a measure The consistency or stability of a measure Assessing a restaurant’s foodAssessing a restaurant’s food

Three important variablesThree important variables– How many testers? (Observers)How many testers? (Observers)

Interrater reliabilityInterrater reliability– How many different entrees? (Observations)How many different entrees? (Observations)

Internal consistencyInternal consistency– How many times? (Occasions)How many times? (Occasions)

Test-retestTest-retest

Page 3: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Interrater ReliabilityInterrater Reliability

The degree to which The degree to which independent raters independent raters agree on an observationagree on an observation

Have two (or more) Have two (or more) judges rate the same judges rate the same peoplepeople

Trained and Trained and independent raters, independent raters, using a coding schemeusing a coding scheme

Page 4: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Observer 1 Observer 2

Complain about injection

-2 3

First negative comment

0 1

Second negative comment

-2 2

Rip up questionnaire -2 3

Interrater ReliabilityInterrater Reliability

Page 5: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Observer 1 Observer 2

Complain about injection

22 22

First negative comment

00 00

Second negative comment

-2-2 -2-2

Rip up questionnaire 22 33

Interrater ReliabilityInterrater Reliability

Page 6: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Internal ConsistencyInternal Consistency

Internal consistencyInternal consistency – the degree to – the degree to which all specific items of a measure which all specific items of a measure behave the same waybehave the same way

Measure the same people with Measure the same people with multiple itemsmultiple items– Different questions in a surveyDifferent questions in a survey– Different behaviors in observationDifferent behaviors in observation

Page 7: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

ExtraversionExtraversion

1 2 3 4 5

Not at all

true

Very true

1.I am outgoing. ____

2.I am friendly. ____

3.I am talkative. ____

4.I am gregarious.____

Page 8: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Internal consistencyInternal consistency

Split-half reliabilitySplit-half reliability – correlation of – correlation of scores on one half of the test with scores on one half of the test with scores on the other halfscores on the other half

Cronbach’s alphaCronbach’s alpha – the average of all – the average of all possible correlations between itemspossible correlations between items

Page 9: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

‘‘One of these things just One of these things just doesn’t belong’doesn’t belong’

One of these things is not like the others, One of these things is not like the others, One of these things just doesn't belongOne of these things just doesn't belong

Student 1 Student 2 Student 3

Ques 1 Ques 1 (Chpt 12)(Chpt 12)

1010 22 99

Ques 2 Ques 2 (Chpt 12)(Chpt 12)

99 33 88

Ques 3 Ques 3 (Chpt 3)(Chpt 3)

22 66 11

Ques 4 Ques 4 (Chpt 12)(Chpt 12)

1010 22 99

Page 10: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Test-Retest ReliabilityTest-Retest Reliability The degree to which a measure correlates The degree to which a measure correlates

positively with itself over timepositively with itself over time– Consistency of the measure over timeConsistency of the measure over time

Measure the same people at two (or more) Measure the same people at two (or more) points in timepoints in time

Desirable for stable traits, but not for transient Desirable for stable traits, but not for transient statesstates

Page 11: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

The “More is Better Rule”The “More is Better Rule”

Reliability is likely to increase as we Reliability is likely to increase as we increase the number of…increase the number of…– Observers (or raters)Observers (or raters)– Observations (or items)Observations (or items)– OccasionsOccasions

Measurement error will average outMeasurement error will average out

Page 12: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

2) Construct Validity 2) Construct Validity

How well an How well an operational definition operational definition represents the represents the construct of interestconstruct of interest

The degree to which The degree to which the construct can be the construct can be inferred from the inferred from the operational definition operational definition of that constructof that construct

Page 13: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Indicators of Construct Indicators of Construct ValidityValidity

Face validityFace validity Criterion validityCriterion validity

– Predictive validityPredictive validity– Concurrent validityConcurrent validity– Convergent validityConvergent validity– Discriminant validityDiscriminant validity

Page 14: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Face ValidityFace Validity

Face validityFace validity – Does the measure – Does the measure appear to measure the construct of appear to measure the construct of interest?interest?– Does the measure “on the face of it” Does the measure “on the face of it”

look like what it’s supposed to look like what it’s supposed to measure?measure?

Not necessary or sufficient for a Not necessary or sufficient for a good measuregood measure

Page 15: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Predictive ValidityPredictive Validity

Predictive validityPredictive validity – Is the measure – Is the measure associated with variables it should associated with variables it should theoretically predict?theoretically predict?

LSAT – Law school performanceLSAT – Law school performance Self-esteem – DepressionSelf-esteem – Depression Shyness – Social anxiety Shyness – Social anxiety

Page 16: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Concurrent ValidityConcurrent Validity

Concurrent validityConcurrent validity – Does the – Does the measure differ between groups it measure differ between groups it ought to differ between?ought to differ between?– Also called “known groups validity”Also called “known groups validity”

E.g., clinically depressed versus non-E.g., clinically depressed versus non-depressed groupsdepressed groups

Page 17: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Convergent ValidityConvergent Validity

Convergent validityConvergent validity – Is the – Is the measure associated with other measure associated with other established measures of the same established measures of the same construct?construct?

Self-report - ObservationsSelf-report - Observations Physiological measure - Self-reportPhysiological measure - Self-report Self-report 1 – Self-report 2Self-report 1 – Self-report 2

Page 18: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Discriminant ValidityDiscriminant Validity

Discriminant validityDiscriminant validity – Is the – Is the measure NOT associated with measure NOT associated with measures of other constructs?measures of other constructs?

Self-esteem scores not associated Self-esteem scores not associated with locus of control scoreswith locus of control scores

Problem solving knowledge not Problem solving knowledge not associated with factual knowledgeassociated with factual knowledge

Page 19: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Measurement Reliability & Measurement Reliability & ValidityValidity

ReliabilityReliability: Is the measure consistent?: Is the measure consistent? ValidityValidity: Does the measure : Does the measure

adequately reflect the construct of adequately reflect the construct of interest?interest?

Reliable and Valid Reliable, not Valid Not Reliable, not Valid

Page 20: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Relationship between Relationship between Reliability and ValidityReliability and Validity

Can be reliable but not validCan be reliable but not valid To be valid it must be reliable

– But reliability is not sole condition for validity

Both reliability and validity are necessary for accurate measurement in a research study.

Page 21: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Measurement Scales Measurement Scales

Nominal scalesNominal scales Ordinal scalesOrdinal scales Interval scalesInterval scales Ratio scalesRatio scales

Page 22: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Nominal ScalesNominal Scales AKA Categorical scalesAKA Categorical scales No numerical/quantitative properties. No numerical/quantitative properties.

Categories or group simply differ from Categories or group simply differ from one anotherone another

Examples:Examples:– Men or womenMen or women– Right or left handedRight or left handed– Catholic, Protestant, Jewish, Hindu, Catholic, Protestant, Jewish, Hindu,

Buddhist…Buddhist…– Numbers on basketball jerseysNumbers on basketball jerseys– Zip codesZip codes

Page 23: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Ordinal ScalesOrdinal Scales

Allow us to rank order the levels of Allow us to rank order the levels of the variables being studiedthe variables being studied

ExamplesExamples– Social classSocial class

lower class, working class, middle class, lower class, working class, middle class, and upper classand upper class

– College football standingsCollege football standings– Letterman’s Top TenLetterman’s Top Ten

Page 24: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Top Ten Bush Goals For His Second Top Ten Bush Goals For His Second TermTerm

10. Fewer idiotic remarks; more hilarious pratfalls.10. Fewer idiotic remarks; more hilarious pratfalls.

9. Add mother Barbara to Mount Rushmore.9. Add mother Barbara to Mount Rushmore.

8. Combine Nebraska and Kansas into new state: Nebransas.8. Combine Nebraska and Kansas into new state: Nebransas.

7. Spice up boring state dinners with tasty fish sticks!7. Spice up boring state dinners with tasty fish sticks!

6. Improve communication skills from poor to fair.6. Improve communication skills from poor to fair.

5. Catch up on his "Smokey And The Bandit" collection.5. Catch up on his "Smokey And The Bandit" collection.

4. Get Ray Stevens to write some funny lyrics for "Hail To The 4. Get Ray Stevens to write some funny lyrics for "Hail To The Chief"Chief"

3. Ride every roller coaster in the country.3. Ride every roller coaster in the country.

2. Install remote-activated button in Oval Office so he can blow 2. Install remote-activated button in Oval Office so he can blow stuff up right from his desk!stuff up right from his desk!

1. Begin vote-rigging process for Jeb's White House run in 2008.1. Begin vote-rigging process for Jeb's White House run in 2008.

Page 25: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Interval ScalesInterval Scales

The difference between the numbers The difference between the numbers on the scale is meaningfulon the scale is meaningful

Scores separated by equal intervalsScores separated by equal intervals ExamplesExamples

– Temperature (Fahrenheit or Celsius)Temperature (Fahrenheit or Celsius)– Scores on personality measureScores on personality measure

Page 26: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Ratio ScalesRatio Scales

Scores separated by Scores separated by equal intervals and equal intervals and there is an absolute there is an absolute zerozero

ExamplesExamples– LengthLength– Weight Weight – TimeTime– Number of responsesNumber of responses

Page 27: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

LevelLevel

QualitativeInfo

Has inherent order

‘more to less’

EqualIntervals

Has zero point

Nominal XX

Ordinal XX XX

Interval XX XX XX

Ratio XX XX XX XX

Scales of MeasurementScales of Measurement

Page 28: Lecture Overview: Measurement 1) Reliability of Measures 1) Reliability of Measures 2) Construct Validity 2) Construct Validity 3) Measurement scales 3)

Concept Check Concept Check

Which scale of measurement best Which scale of measurement best describes the following:describes the following:– Telephone numbersTelephone numbers– Distances from Budapest to cities in the USDistances from Budapest to cities in the US– Scores on an extraversion personality Scores on an extraversion personality

assessmentassessment– Ranking of basketball teams in the Big TenRanking of basketball teams in the Big Ten