copyright © allyn & bacon (2007) data and the nature of measurement graziano and raulin...

Copyright © Allyn & Bacon (2007)Copyright © Allyn & Bacon (2007)

Data and the Nature Data and the Nature of Measurementof Measurement

Graziano and RaulinGraziano and RaulinResearch Methods: Chapter 4Research Methods: Chapter 4This multimedia product and its contents are protected under copyright law. The This multimedia product and its contents are protected under copyright law. The following are prohibited by law: (1) Any public performance or display, including following are prohibited by law: (1) Any public performance or display, including transmission of any image over a network; (2) Preparation of any derivative transmission of any image over a network; (2) Preparation of any derivative work, including the extraction, in whole or in part, of any images; (3) Any rental, work, including the extraction, in whole or in part, of any images; (3) Any rental, lease, or lending of the program.lease, or lending of the program.


Research VariablesResearch Variables

VariableVariable: Any characteristic that can : Any characteristic that can take on more than one valuetake on more than one value– Examples: speed, level of hostility, Examples: speed, level of hostility,

accuracy of feedback, reaction timeaccuracy of feedback, reaction time Research is the study of the Research is the study of the

relationship among variablesrelationship among variables– Therefore, there must be at least two Therefore, there must be at least two

variables in a research study (or there is variables in a research study (or there is no relationship to study)no relationship to study)


Measuring VariablesMeasuring Variables

MeasurementMeasurement: Assigning : Assigning numbers to indicate the level of a numbers to indicate the level of a variablevariable– Sometimes the number assignment Sometimes the number assignment

is easy to understand (e.g., time is easy to understand (e.g., time measured in seconds)measured in seconds)

– Sometimes it is more arbitrary (e.g., Sometimes it is more arbitrary (e.g., 1 for male and 2 for female)1 for male and 2 for female)


Scales of MeasurementScales of Measurement

Based on how closely the Based on how closely the measurement scale matches the real measurement scale matches the real number systemnumber system

Scales of Measurement (Stevens, Scales of Measurement (Stevens, 1946, 1957)1946, 1957)– NominalNominal– OrdinalOrdinal– IntervalInterval– RatioRatio


Nominal ScalesNominal Scales

Naming scaleNaming scale– Each number reflects a categoryEach number reflects a category– ExamplesExamples: diagnostic categories, : diagnostic categories,

political affiliationspolitical affiliations Produces nominal or categorical Produces nominal or categorical

datadata Mathematical propertiesMathematical properties

– IdentityIdentity


Ordinal ScalesOrdinal Scales

Scale indicating rank orderScale indicating rank order– Reflects the order, but not the Reflects the order, but not the

amount amount ExampleExample: order of finish in : order of finish in a race, class rankingsa race, class rankings

Produces ordered dataProduces ordered data Mathematical propertiesMathematical properties

– Identity Identity – MagnitudeMagnitude


Interval ScalesInterval Scales

Scale with equal intervalsScale with equal intervals– The scale indicates amount, but with no The scale indicates amount, but with no

zero pointzero point– ExamplesExamples: temperature on the Celsius : temperature on the Celsius

scale, most psychological testsscale, most psychological tests Produces score dataProduces score data Mathematical propertiesMathematical properties

– IdentityIdentity– MagnitudeMagnitude– Equal intervalsEqual intervals


Ratio ScalesRatio Scales

Scale that fits the number system wellScale that fits the number system well– Includes equal intervals and a true zero Includes equal intervals and a true zero – ExamplesExamples: time, distance, frequency : time, distance, frequency

Produces score dataProduces score data Mathematical properties Mathematical properties

– IdentityIdentity– MagnitudeMagnitude– Equal intervalsEqual intervals– True zeroTrue zero


Psychological TestsPsychological Tests

Most psychological test fall Most psychological test fall somewhere between an ordinal and somewhere between an ordinal and a ratio scalea ratio scale– Ordinal in that the distance between Ordinal in that the distance between

scores may not be equalscores may not be equal– Ratio in that one can view the test Ratio in that one can view the test

score as the number of correct itemsscore as the number of correct items Norm is to assume such tests Norm is to assume such tests

represent an interval scalerepresent an interval scale


Measurement ErrorMeasurement Error

Decreases the accuracy of Decreases the accuracy of measurementmeasurement

Possible sources of measurement Possible sources of measurement errorerror– Response set biasesResponse set biases– Inconsistent measurement proceduresInconsistent measurement procedures– Sloppy proceduresSloppy procedures– Unreliable measuresUnreliable measures


Operational DefinitionsOperational Definitions

Specific procedures for measuring Specific procedures for measuring and/or manipulating a variableand/or manipulating a variable

Every variable should be Every variable should be operationally definedoperationally defined

The more careful and complete The more careful and complete the operational definition, the the operational definition, the more precise the measurement of more precise the measurement of the variablethe variable


ReliabilityReliability

The consistency of measurementThe consistency of measurement Consistency can be conceptualized in Consistency can be conceptualized in

different waysdifferent ways– Therefore, there are different types of Therefore, there are different types of

reliabilityreliability Usually measured with a correlationUsually measured with a correlation

– Covered in Chapter 5Covered in Chapter 5– Sensitive to the consistency of rank Sensitive to the consistency of rank

orderings of participantsorderings of participants


Types of ReliabilityTypes of Reliability

Interrater reliabilityInterrater reliability: degree of : degree of agreement between two independent agreement between two independent ratersraters

Test-retest reliabilityTest-retest reliability: degree of : degree of consistency over timeconsistency over time

Internal consistency reliabilityInternal consistency reliability: : degree to which the items of a degree to which the items of a measure all measure the same thingmeasure all measure the same thing


Perfect ReliabilityPerfect Reliability

Reliability is a Reliability is a measure of measure of consistency.consistency.

Perfect reliability Perfect reliability means that the means that the scores are scores are perfectly perfectly consistent.consistent.


Good ReliabilityGood Reliability

Reliability Reliability deteriorates deteriorates when the when the consistency is consistency is lost.lost.

Here the rank Here the rank orderings are still orderings are still reasonably reasonably consistent.consistent.


Fair ReliabilityFair Reliability

As reliability As reliability deteriorates deteriorates further, the rank further, the rank orderings from orderings from the two testings the two testings are less similar.are less similar.


Poor ReliabilityPoor Reliability

When you reach When you reach the point where the point where the rank the rank orderings have orderings have no relationship to no relationship to one another, one another, your reliability your reliability (i.e., consistency) (i.e., consistency) is poor.is poor.


Effective RangeEffective Range

The range in which a measure gives an The range in which a measure gives an accurate indication of the level of the accurate indication of the level of the variablevariable

Critical to select measures with an Critical to select measures with an effective range appropriate foreffective range appropriate for– Your sample Your sample – Your studyYour study– ExampleExample: Using a calculus test to measure : Using a calculus test to measure

math skills in second graders would NOT math skills in second graders would NOT workwork


Scale Attenuation Scale Attenuation EffectsEffects Results from a restriction of the Results from a restriction of the

range of the measurerange of the measure– Therefore, people above or below the Therefore, people above or below the

effective range are not measured effective range are not measured accuratelyaccurately

Two types of scale attenuation Two types of scale attenuation effectseffects– Floor effectsFloor effects– Ceiling effectsCeiling effects


Illustrating These Illustrating These EffectsEffects SituationSituation

– Assume that we Assume that we are measuring are measuring the weight of 9 the weight of 9 men using a men using a standard standard bathroom scalebathroom scale

Floor EffectFloor Effect– The needle is The needle is

stuck so that it stuck so that it never reads never reads below 175below 175

Ceiling EffectCeiling Effect– The needle is The needle is

stuck so that it stuck so that it never reads never reads above 200above 200


Weights for 9 MenWeights for 9 Men

0

50

100

150

200

250

300

1 2 3 4 5 6 7 8 9

Individuals

Weig

ht


Floor EffectFloor Effect

Range on the bottom of the scale is Range on the bottom of the scale is insufficient to measure people who insufficient to measure people who score near the bottomscore near the bottom

If the scale needle was stuck so that it If the scale needle was stuck so that it never read below 175 poundsnever read below 175 pounds– There would be a floor at 175There would be a floor at 175

Error due to this floor effect is shown Error due to this floor effect is shown in dark blue in the next slidein dark blue in the next slide– Weights recorded for these people are too Weights recorded for these people are too

highhigh


Floor EffectFloor Effect

0

50

100

150

200

250

300

1 2 3 4 5 6 7 8 9

Individuals

Weig

ht


Ceiling EffectCeiling Effect

The range on the top of the scale is The range on the top of the scale is insufficient to measure people who insufficient to measure people who score near the topscore near the top

If the scale could not read above 200 If the scale could not read above 200 poundspounds– There would be a ceiling at 200 (everyone There would be a ceiling at 200 (everyone

above 200 would be reported as weighing above 200 would be reported as weighing 200)200)

Error due to this ceiling effect is shown Error due to this ceiling effect is shown in dark blue in the next slidein dark blue in the next slide


Ceiling EffectCeiling Effect

0

50

100

150

200

250

300

1 2 3 4 5 6 7 8 9

Individuals

Weig

ht


Source of These Source of These EffectsEffects Result of psychological scale with Result of psychological scale with

an insufficient range to measure an insufficient range to measure the range of performance in the the range of performance in the samplesample– Example: a measure of memory that Example: a measure of memory that

is either too easy or too hardis either too easy or too hard


ValidityValidity

A scale is valid if it measures what it is A scale is valid if it measures what it is supposed to measuresupposed to measure

Validity also refers to how well a scale Validity also refers to how well a scale predicts other variablespredicts other variables– Example: An IQ test is likely to be a valid Example: An IQ test is likely to be a valid

predictor of grades in school.predictor of grades in school.– When used this way, the scale is called the When used this way, the scale is called the

predictor measurepredictor measure and the measure and the measure predicted is called the predicted is called the criterioncriterion


Measuring ValidityMeasuring Validity

Validity is usually measured with a Validity is usually measured with a correlationcorrelation– The correlation is between The correlation is between

The measure andThe measure and A SPECIFIED criterion measureA SPECIFIED criterion measure

– Always list the criterion when reporting the Always list the criterion when reporting the level of validity level of validity

For example, the IQ test is a valid predictor of For example, the IQ test is a valid predictor of school grades, but not a valid predictor of school grades, but not a valid predictor of athletic ability.athletic ability.


Perfect ValidityPerfect Validity

Validity is the Validity is the degree to which degree to which one measure one measure predicts another.predicts another.

With perfect With perfect validity, the rank validity, the rank orderings on the orderings on the predictor and predictor and criterion criterion measures are measures are identical.identical.


Good ValidityGood Validity

If the rank If the rank orderings from orderings from the predictor and the predictor and criterion criterion measures are measures are similar, you have similar, you have good validity.good validity.


Fair ValidityFair Validity

But the less the But the less the rank orderings rank orderings from the from the predictor and predictor and criterion criterion measures agree, measures agree, the lower the the lower the validity.validity.


Poor ValidityPoor Validity

When you reach When you reach the point that the the point that the rank orderings of rank orderings of predictor and predictor and criterion criterion measures are measures are unrelated, you unrelated, you have essentially have essentially zero validity.zero validity.


Objective Objective MeasurementMeasurement The hallmark of scienceThe hallmark of science

– Objective measures produce the same Objective measures produce the same result no matter who does the result no matter who does the measuringmeasuring

– Therefore, scientific principles will apply Therefore, scientific principles will apply no matter who tests these principlesno matter who tests these principles

Objective measures reduce biases Objective measures reduce biases that could distort results that could distort results (see Chapters 8 (see Chapters 8 and 9)and 9)


SummarySummary

Measuring variables is central to Measuring variables is central to researchresearch

Several scales of measurement existSeveral scales of measurement exist Reduce measurement error with Reduce measurement error with

carefully developed operational carefully developed operational definitionsdefinitions

Enhance reliability and validity of your Enhance reliability and validity of your measuresmeasures

The goal is to produce objective, The goal is to produce objective, accurate measures of your variablesaccurate measures of your variables

copyright © allyn & bacon (2007) data and the nature of measurement graziano and raulin...

Documents