chapter 5 understanding, calculating, and evaluating reliability and objectivity
TRANSCRIPT
Chapter 5Chapter 5
Understanding, Calculating, Understanding, Calculating, and Evaluating Reliability and Evaluating Reliability
and Objectivityand Objectivity
Methods for Evaluating Reliability Methods for Evaluating Reliability and Their Calculationsand Their Calculations
TestTest–Retest–Retest
• Most straightforward way to determine Most straightforward way to determine reliabilityreliability
• Must have:Must have:– No major changes in the construct being No major changes in the construct being
measuredmeasured– Sufficient recovery time between Sufficient recovery time between
measurementsmeasurements
Evaluating TestEvaluating Test–Retest Reliability–Retest Reliability
• Reliability coefficient: Reliability coefficient: – A ratio that shows the relationship between A ratio that shows the relationship between
two measurements, indicating the consistency two measurements, indicating the consistency (or reliability) between them. (or reliability) between them.
• Intraclass correlation:Intraclass correlation:– A statistical technique used to compute the A statistical technique used to compute the
reliability coefficient to assess the relationship reliability coefficient to assess the relationship between measures of the same class as in a between measures of the same class as in a testtest–retest study.–retest study.
Calculating the Intraclass CorrelationCalculating the Intraclass Correlation
R=[(SSa/R=[(SSa/nn––1) 1) – – (SSw/((SSw/(n*n*(k(k––1)))]/ (SSa/1)))]/ (SSa/nn––1)1)SSa=(ΣTSSa=(ΣT22//kk) ) – – ((ΣX) ((ΣX) 22//nknk))SSw= ΣXSSw= ΣX2 2 – – (ΣT(ΣT22/k)/k)
Where R is the intraclass reliabilityWhere R is the intraclass reliabilityΣ represents the sumΣ represents the sumN N = the number of test subjects= the number of test subjectskk = the number of trials for each person (usually = the number of trials for each person (usually
two)two)ΣTΣT2 2 = the sum of all the squared total scores for = the sum of all the squared total scores for
each personeach personΣXΣX22 = the sum of all the scores of everyone tested = the sum of all the scores of everyone tested
An Alternative for An Alternative for Calculating the Intraclass Correlation Calculating the Intraclass Correlation
• Try this website: Try this website:
http://department.obg.cuhk.edu.hk
and go the Statistics Tool Box link.and go the Statistics Tool Box link.
Results of a Calculation Results of a Calculation of Intraclass Reliabilityof Intraclass Reliability
Evaluating Reliability with Evaluating Reliability with a Single Test Administrationa Single Test Administration
• Split-half reliability:Split-half reliability:– Compare one half Compare one half
of a test with the of a test with the other halfother half
– Spearman-Brown Spearman-Brown Prophecy FormulaProphecy Formula
• Internal Internal consistency consistency reliability:reliability:– Average all possible Average all possible
split-half estimatessplit-half estimates– Cronbach’s alphaCronbach’s alpha
Evaluating the Reliability Evaluating the Reliability of Criterion-Referenced Measurementsof Criterion-Referenced Measurements
• Calculate percentage of agreement between the Calculate percentage of agreement between the test and the retest.test and the retest.
Percentage of Agreement= [(Cboth + NCboth) / (Cboth + Percentage of Agreement= [(Cboth + NCboth) / (Cboth + NCboth + C/NC + NC/C)] NCboth + C/NC + NC/C)] ** 100 100
Where Cboth = people scored as competent in both Trials 1 Where Cboth = people scored as competent in both Trials 1 and 2and 2
NCboth = people scored as not competent in both trialsNCboth = people scored as not competent in both trials NC/C = people scored as not competent in Trial 1 but NC/C = people scored as not competent in Trial 1 but
competent in Trial 2competent in Trial 2 C/NC = people scored as competent in Trial 1 but not C/NC = people scored as competent in Trial 1 but not
competent in Trial 2competent in Trial 2
Example Diagram for Evaluating the Example Diagram for Evaluating the Reliability of a Criterion-Referenced Reliability of a Criterion-Referenced
MeasurementMeasurement
Standard Error of MeasurementStandard Error of Measurement
• Defined: Defined: – An estimation of the error inherent in any An estimation of the error inherent in any
individual’s test score.individual’s test score.
SEM = SD * SEM = SD * √1–r√1–rrcrc
where SD = the standard deviation for the testwhere SD = the standard deviation for the test
rrrc rc = the reliability coefficient for the test= the reliability coefficient for the test
Increasing ReliabilityIncreasing Reliability
• Repeat a measurement several Repeat a measurement several times—times—– To improve both validity and reliabilityTo improve both validity and reliability– To discover and minimize errorsTo discover and minimize errors– To average out the errorsTo average out the errors
Methods for Evaluating ObjectivityMethods for Evaluating Objectivity
Calculating ObjectivityCalculating Objectivity
• Objectivity can be considered a special Objectivity can be considered a special case of reliability.case of reliability.– Inter-rater reliabilityInter-rater reliability
• Most techniques used to evaluate reliability Most techniques used to evaluate reliability can be used to evaluate objectivity.can be used to evaluate objectivity.
Calculating Objectivity Calculating Objectivity of Different Types of Measuresof Different Types of Measures
• For continuous measures:For continuous measures:– Intraclass correlationIntraclass correlation
• For discrete measures:For discrete measures:– Calculate the percent agreement Calculate the percent agreement
between test administratorsbetween test administrators
Validity, Reliability, and ObjectivityValidity, Reliability, and Objectivity
• It is possible to have high reliability or It is possible to have high reliability or objectivity without high validity.objectivity without high validity.
• Good reliability or objectivity will always Good reliability or objectivity will always be present with a valid measurement.be present with a valid measurement.
• Good reliability and objectivity do not Good reliability and objectivity do not establish good validity; they simply establish good validity; they simply suggest that a measurement suggest that a measurement maymay be be valid.valid.
Your ViewpointYour Viewpoint
• Can you think of any times in your life Can you think of any times in your life when you have had to evaluate the when you have had to evaluate the reliability or objectivity of something or reliability or objectivity of something or someone?someone?
• What did you do with the results of this What did you do with the results of this evaluation? Did it cause you to make any evaluation? Did it cause you to make any changes in your daily routines or change changes in your daily routines or change your mind about a decision?your mind about a decision?