20150218130208kuliah7_validrelible

8/9/2019 20150218130208kuliah7_validrelible

1/43

© 2006 The McGraw-Hill Companies, Inc. All rightsMcGraw-Hill

Validity and ReliabilityValidity and Reliability


2/43


Validity and ReliabilityValidity and ReliabilityChapter EightChapter Eight


3/43


ValidityValidity

ValidityValidit

y has been defned as reerring to thehas been defned as reerring to the

appropriateness, correctness, meaningulness,appropriateness, correctness, meaningulness,

and useulness o the specifc inerencesand useulness o the specifc inerences

researchers make based on the data they collect.researchers make based on the data they collect.

It is the most important idea to consider whenIt is the most important idea to consider when

preparing or selecting an instrument.preparing or selecting an instrument.

ValidationValidation is the process o collecting andis the process o collecting and

analyzing evidence to support such inerences.analyzing evidence to support such inerences.


4/43


Evidence of ValidityEvidence of Validity

here are ! types o evidence a researcher here are ! types o evidence a researchermight collectmight collect Content"related evidence o validityContent"related evidence o validity

Content and ormat o the instrumentContent and ormat o the instrument

Criterion"related evidence o validityCriterion"related evidence o validity #elationship between scores obtained using the#elationship between scores obtained using the

instrument and scores obtainedinstrument and scores obtained

Construct"related evidence o validityConstruct"related evidence o validity $sychological construct being measured by the$sychological construct being measured by the

instrumentinstrument


5/43


Illustration of Types of Evidence of Validity (Figure 8.1)Illustration of Types of Evidence of Validity (Figure 8.1)


6/43


Contentrelated EvidenceContentrelated Evidence

% key element is the ade&uacy o the% key element is the ade&uacy o thesampling o the domain it is supposed tosampling o the domain it is supposed torepresent.represent.

he other aspect o content validation is the he other aspect o content validation is the

ormat o the instrument.ormat o the instrument. %ttempts to obtain evidence that the items%ttempts to obtain evidence that the items

measure what they are supposed to measuremeasure what they are supposed to measuretypiy the process o content"relatedtypiy the process o content"related

evidence.evidence.


7/43


Criterionrelated EvidenceCriterionrelated Evidence

% criterion is a second test presumed to% criterion is a second test presumed tomeasure the same variable.measure the same variable.

here are two orms o criterion"related here are two orms o criterion"relatedvalidity'validity'

()() $redictive validity' time interval elapses between$redictive validity' time interval elapses betweenadministering the instrument and obtaining criterionadministering the instrument and obtaining criterionscoresscores

*)*) Concurrent validity' instrument data and criterionConcurrent validity' instrument data and criteriondata are gathered and compared at the same timedata are gathered and compared at the same time

% Correlation Coe+cient r) indicates the% Correlation Coe+cient r) indicates thedegree o relationship that e-ists between thedegree o relationship that e-ists between thescores o individuals obtained by twoscores o individuals obtained by twoinstruments.instruments.


8/43


Constructrelated EvidenceConstructrelated Evidence

Considered the broadest o the threeConsidered the broadest o the threecategories.categories.

here is no single piece o evidence that here is no single piece o evidence that

satisfes construct"related validity.satisfes construct"related validity. #esearchers attempt to collect a variety o#esearchers attempt to collect a variety o

types o evidence, including both content"types o evidence, including both content"related and criterion"related evidence.related and criterion"related evidence.

he more evidence researchers have rom he more evidence researchers have romdierent sources, the more confdent theydierent sources, the more confdent theybecome about the interpretation o thebecome about the interpretation o theinstrument.instrument.


9/43© 2006 The McGraw-Hill Companies, Inc. All rightsMcGraw-Hill

ReliabilityReliability

#eers to the consistency o scores or#eers to the consistency o scores or

answers provided by an instrument.answers provided by an instrument.

/cores obtained can be considered reliable/cores obtained can be considered reliable

but not valid.but not valid. %n instrument should be reliable and valid%n instrument should be reliable and valid

0igure 1.*), depending on the conte-t in0igure 1.*), depending on the conte-t in

which an instrument is used.which an instrument is used.



Reliability and ValidityReliability and Validity (Figure 8.!)(Figure 8.!)



Reliability of "easure#entReliability of "easure#ent (Figure 8.$)(Figure 8.$)



Errors of "easure#entErrors of "easure#ent

2ecause errors o measurement are2ecause errors o measurement are

always present to some degree, variationalways present to some degree, variation

in test scores are common.in test scores are common.

his is due to' his is due to' 3ierences in motivation3ierences in motivation

EnergyEnergy

%n-iety%n-iety

3ierent testing situation3ierent testing situation



Reliability CoefficientReliability Coefficient

E-presses a relationship between scoresE-presses a relationship between scores

o the same instrument at two dierento the same instrument at two dierent

times or parts o the instrument.times or parts o the instrument.

he ! best known methods are' he ! best known methods are' est"retest est"retest

E&uivalent orms methodE&uivalent orms method

Internal consistency methodInternal consistency method



TestRetest "et%odTestRetest "et%od

Involves administering the same test twice toInvolves administering the same test twice tothe same group ater a certain time interval hasthe same group ater a certain time interval haselapsed.elapsed.

% reliability coe+cient is calculated to indicate% reliability coe+cient is calculated to indicate

the relationship between the two sets o scores.the relationship between the two sets o scores. #eliability coe+cients are aected by the lapse#eliability coe+cients are aected by the lapse

o time between the administrations o the test.o time between the administrations o the test. %n appropriate time interval should be selected.%n appropriate time interval should be selected. In Educational #esearch, scores collected over aIn Educational #esearch, scores collected over a

two"month period is considered su+cienttwo"month period is considered su+cientevidence o test"retest reliability.evidence o test"retest reliability.



E&uivalentFor#s "et%odE&uivalentFor#s "et%od

wo dierent but e&uivalent alternate or wo dierent but e&uivalent alternate orparallel) orms o an instrument areparallel) orms o an instrument areadministered to the same group during theadministered to the same group during thesame time period.same time period.

% reliability coe+cient is then calculated% reliability coe+cient is then calculatedbetween the two sets o scores.between the two sets o scores.

It is possible to combine the test"retest andIt is possible to combine the test"retest ande&uivalent"orms methods by giving twoe&uivalent"orms methods by giving two

dierent orms o testing with a time intervaldierent orms o testing with a time intervalbetween the two administrations.between the two administrations.



InternalConsistency "et%odsInternalConsistency "et%ods

here are several internal"consistency methods that here are several internal"consistency methods thatre&uire only one administration o an instrument.re&uire only one administration o an instrument. /plit"hal $rocedure/plit"hal $rocedure' involves scoring two halves o a' involves scoring two halves o a

test separately or each sub4ect and calculating thetest separately or each sub4ect and calculating thecorrelation coe+cient between the two scores.correlation coe+cient between the two scores.

5uder"#ichardson %pproaches5uder"#ichardson %pproaches' 5#*6 and 5#*()' 5#*6 and 5#*()re&uires ! pieces o inormation're&uires ! pieces o inormation' 7umber o items on the test7umber o items on the test he mean he mean he standard deviation he standard deviation

Considered the most re&uent method or determiningConsidered the most re&uent method or determining

internal consistencyinternal consistency %lpha Coe+cient%lpha Coe+cient' a general orm o the 5#*6 used to' a general orm o the 5#*6 used to

calculate the reliability o items that are not scoredcalculate the reliability o items that are not scoredright vs. wrong.right vs. wrong.



'tandard Error of "easure#ent'tandard Error of "easure#ent

%n inde- that shows the e-tent to which%n inde- that shows the e-tent to whicha measurement would vary undera measurement would vary underchanged circumstances.changed circumstances.

here are many possible standard errors here are many possible standard errorsor scores given.or scores given. %lso known as%lso known as measurement errormeasurement error, a, a

range o scores that show the amount orange o scores that show the amount o

error which can be e-pected. %ppendi-error which can be e-pected. %ppendi-3)3)



'coring gree#ent'coring gree#ent

/coring agreement re&uires a demonstration that/coring agreement re&uires a demonstration thatindependent scorers can achieve satisactoryindependent scorers can achieve satisactory

agreement in their scoring.agreement in their scoring.

Instruments that use direct observations are highlyInstruments that use direct observations are highly

vulnerable to observer dierences.vulnerable to observer dierences. 8hat is desired is a correlation o at least .968hat is desired is a correlation o at least .96

among scorers as an acceptable level oamong scorers as an acceptable level o

agreement.agreement.



Internal ValidityInternal Validity



Internal ValidityInternal ValidityChapter 7ineChapter 7ine



%at is Internal Validity*%at is Internal Validity*

Internal ValidityInternal Validity is when observed dierences onis when observed dierences onthe dependent variable are directly related to thethe dependent variable are directly related to theindependent variable, and not due to some otherindependent variable, and not due to some otherunintended variable.unintended variable.

In other words, any relationship observedIn other words, any relationship observedbetween two or more variables should bebetween two or more variables should beunambiguous as to what it means rather thanunambiguous as to what it means rather thanbeing due to :something else;.being due to :something else;.

he :something else; could be' he :something else; could be' %ge%ge %bility%bility ypes o materials used ypes o materials used



T%reats to Internal ValidityT%reats to Internal Validity

/ub4ect/ub4ect

CharacteristicsCharacteristics

istory



"ortality T%reat to Internal Validity"ortality T%reat to Internal Validity (Figure +.1)(Figure +.1)



,ocation "ig%t "a-e a ifference,ocation "ig%t "a-e a ifference(Figure +.!)(Figure +.!)



'ub/ect C%aracteristics'ub/ect C%aracteristics

he selection o people may result in dierences, he selection o people may result in dierences,either between individuals or groups, that areeither between individuals or groups, that arerelated to the variables being studied.related to the variables being studied.

his reers to a ?selection bias@ or sub4ect his reers to a ?selection bias@ or sub4ect

characteristics threat.characteristics threat. I not controlled, these variables may :e-plain away;I not controlled, these variables may :e-plain away;

whatever dierences are ound in the study.whatever dierences are ound in the study. here are techni&ues used to either e&ualize the here are techni&ues used to either e&ualize the

dierences or control these variables.dierences or control these variables.



"ortality"ortality

It is common to lose sub4ects as a studyIt is common to lose sub4ects as a studyprogressesprogresses

his is known as :mortality threat;. his is known as :mortality threat;. =oss o sub4ects limits generalizability and=oss o sub4ects limits generalizability and

can introduce bias.can introduce bias.


27/43


,ocation,ocation

he particular locations where data is he particular locations where data is

collected may create dierent results orcollected may create dierent results or

e-planations known as ?location [email protected] known as ?location threat@.

he best way to control or this is to keep he best way to control or this is to keepthe location consistent or all sub4ects.the location consistent or all sub4ects.

I this is not possible, the researcher shouldI this is not possible, the researcher should

ensure that dierent locations do not avorensure that dierent locations do not avor

or 4eopardize the hypothesis.or 4eopardize the hypothesis.


28/43


Instru#entationInstru#entation

he way instruments are used may he way instruments are used may

constitute a threat to the internal validityconstitute a threat to the internal validity

o a study.o a study.

/ome e-amples are as ollows'/ome e-amples are as ollows' Instrument decayInstrument decay

3ata Collector Characteristics3ata Collector Characteristics

3ata Collector 2ias3ata Collector 2ias

I t t I t t


29/43


Instru#ent ecayInstru#ent ecay (Figure +.$)(Figure +.$)


30/43


ata Collector C%aracteristics T%reat ata Collector C%aracteristics T%reat(Figure +.0)(Figure +.0)


31/43


TestingTesting

% :testing threat; is when eorts are% :testing threat; is when eorts are

achieved by the sub4ects due to :practice;achieved by the sub4ects due to :practice;

i.e., pretest, post"test designed study)i.e., pretest, post"test designed study)

%n interaction also could cause this by%n interaction also could cause this bytaking the test and being more aware o ataking the test and being more aware o a

possible interaction, allowing the sub4ectspossible interaction, allowing the sub4ects

to be more responsive towards theto be more responsive towards the

treatment.treatment.


32/43


Testing T%reat to Internal Validity Testing T%reat to Internal Validity(Figure +.)(Figure +.)


33/43


2istory2istory

%% history threathistory threat is when an unoreseenis when an unoreseen

event occurs during the course o theevent occurs during the course o the

study.study.

#esearchers need to be alert to any#esearchers need to be alert to anypossibilities o inAuences that may occurpossibilities o inAuences that may occur

during the course o the study.during the course o the study.


34/43


2istory T%reat to Internal Validity 2istory T%reat to Internal Validity(Figure +.3)(Figure +.3)


35/43


"aturation"aturation

Change during an intervention may be due toChange during an intervention may be due toactors associated with the passing o timeactors associated with the passing o timerather than the intervention.rather than the intervention.

/tudents could change over the course o a/tudents could change over the course o a

study. his is known as a


36/43


Could "aturation be at or- 2ere*Could "aturation be at or- 2ere*(Figure +.4)(Figure +.4)


37/43


ttitude of 'ub/ects ttitude of 'ub/ects

he way sub4ects view a study and their he way sub4ects view a study and theirparticipation can be considered a threat toparticipation can be considered a threat tointernal validity, a.k.a. the ?>awthorne [email protected] validity, a.k.a. the ?>awthorne eect@.

/ub4ects may perorm better based upon a/ub4ects may perorm better based upon aeeling o receiving special attention.eeling o receiving special attention.

he opposite may occur, with sub4ects receiving he opposite may occur, with sub4ects receivingno treatment at all, resulting in poorno treatment at all, resulting in poorperormances.perormances.

% remedy to this would be to provide both% remedy to this would be to provide both

groups with comparable treatments or to makegroups with comparable treatments or to makethe treatment a regular part o the study.the treatment a regular part o the study.

T% ttit d f ' b/ t C " - iffT%e ttitude of 'ub/ects Can "a-e a ifference


38/43


T%e ttitude of 'ub/ects Can "a-e a ifferenceT%e ttitude of 'ub/ects Can "a-e a ifference(Figure +.8)(Figure +.8)


39/43


RegressionRegression

%% regressionregression threat is possible whenthreat is possible when

change is studied in a group that haschange is studied in a group that has

e-treme low or high perormance in thee-treme low or high perormance in the

pre"intervention stage.pre"intervention stage. %s with the%s with the maturationmaturation threat, this can bethreat, this can be

controlled by the use o an e&uivalentcontrolled by the use o an e&uivalent

control or comparison group.control or comparison group.

R i R It 2 d


40/43


Regression Rears Its 2eadRegression Rears Its 2ead(Figure +.+)(Figure +.+)


41/43


I#ple#entationI#ple#entation

he e-perimental group may be treated in he e-perimental group may be treated in

ways that are unintended, giving them anways that are unintended, giving them an

advantage.advantage.

his is known as an his is known as an implementationimplementation threat.threat. his can occur in two ways' his can occur in two ways'

()() 8hen dierent individuals are assigned to8hen dierent individuals are assigned to

implement dierent methods, and theseimplement dierent methods, and these

individuals dier in ways related to the outcomeindividuals dier in ways related to the outcome*)*) 8hen some individuals have a personal bias in8hen some individuals have a personal bias in

avor o one method over the otheravor o one method over the other


42/43


2o5 to "ini#i6e T%reats to2o5 to "ini#i6e T%reats to

Internal ValidityInternal Validity

here are our alternatives a researcher here are our alternatives a researchercan use to reduce threats to internalcan use to reduce threats to internalvalidity'validity'

()() /tandardize the conditions under which the/tandardize the conditions under which thestudy occursstudy occurs

*)*) btain more inormation on the sub4ects obtain more inormation on the sub4ects othe studythe study

!)!) btain more inormation on the details o thebtain more inormation on the details o thestudystudy

D)D) Choose an appropriate designChoose an appropriate design

Illustration of T%reats to Internal ValidityIllustration of T%reats to Internal Validity


43/43

Illustration of T%reats to Internal ValidityIllustration of T%reats to Internal Validity(Figure +.17)(Figure +.17)

20150218130208kuliah7_validrelible

Documents