calculations of reliability we are interested in calculating the icc –first step: conduct a...
TRANSCRIPT
Calculations of ReliabilityCalculations of Reliability
We are interested in calculating the ICCWe are interested in calculating the ICC– First step:First step:
Conduct a single-factor, within-subjects (repeated Conduct a single-factor, within-subjects (repeated measures) ANOVAmeasures) ANOVA
– This is an inferential test for systematic errorThis is an inferential test for systematic error– All subsequent equations are derived from the ANOVA All subsequent equations are derived from the ANOVA
tabletable
Trial A1Trial A1 Trial A2Trial A2 Trial B1Trial B1 Trial B2Trial B2
146146 140140 -6-6 166166 160160 - 6- 6
148148 152152 + 4+ 4 168168 172172 + 4+ 4
170170 152152 - 18- 18 160160 142142 - 18- 18
9090 9999 + 9+ 9 150150 159159 + 9+ 9
157157 145145 - 12- 12 147147 135135 - 12- 12
156156 153153 + 3+ 3 146146 143143 - 3- 3
176176 167167 - 9- 9 156156 147147 - 9- 9
205205 218218 + 13+ 13 155155 168168 + 13+ 13
156 156 ++ 33 33 153 153 ++ 33 33 156 156 ++ 8 8 153 153 ++ 13 13
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:1.1. Arrange the raw data (X) into tabular form, Arrange the raw data (X) into tabular form,
placing the data for subjects in rows (R), and placing the data for subjects in rows (R), and repeated measures in columns (C).repeated measures in columns (C).
Sub # Trial A1 (Trial A1)2 Trial A2 (Trial A2)2 ΣR (ΣR)2
1 146 1402 148 1523 170 1524 90 995 157 1456 156 1537 176 1678 205 218
ΣC
Mean ΣΣR Σ(ΣR)2
ΣXT = ΣΣR
ΣX2
Σ(ΣR)2
Nk
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:2.2. Square each value = (Trial A1)Square each value = (Trial A1)22
3.3. Calculate the row totals (Calculate the row totals (ΣΣR) using the R) using the original scoresoriginal scores
4.4. Calculate the column totals (Calculate the column totals (ΣΣC) using the C) using the original scores original scores
5.5. Calculate the grand total (Calculate the grand total (ΣΣXXTT) or () or (ΣΣΣΣR) – R) –
same thing.same thing.
Sub # Trial A1 (Trial A1)2 Trial A2 (Trial A2)2 ΣR (ΣR)2
1 146 21316 140 19600 2862 148 21904 152 23104 3003 170 28900 152 23104 3224 90 8100 99 9801 1895 157 24649 145 21025 3026 156 24336 153 23409 3097 176 30976 167 27889 3438 205 42025 218 47524 423
ΣC 1248 1226
Mean ΣΣR Σ(ΣR)2
ΣXT = ΣΣR
ΣX2
Σ(ΣR)2
Nk
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:6.6. Sum the row totals = Sum the row totals = ΣΣΣΣRR
– This is also the “total sum” of all original scores.This is also the “total sum” of all original scores.
– Also = Also = ΣΣXXTT
7.7. Square each row total = (Square each row total = (ΣΣR)R)22
8.8. Sum the squares of the row totals = Sum the squares of the row totals = ΣΣ((ΣΣR)R)22
Sub # Trial A1 (Trial A1)2 Trial A2 (Trial A2)2 ΣR (ΣR)2
1 146 21316 140 19600 286 817962 148 21904 152 23104 300 900003 170 28900 152 23104 322 1036844 90 8100 99 9801 189 357215 157 24649 145 21025 302 912046 156 24336 153 23409 309 954817 176 30976 167 27889 343 1176498 205 42025 218 47524 423 178929
ΣC 1248 1226 2474 794464
Mean 156 25275.75 153.25 24432 ΣΣR Σ(ΣR)2
ΣXT 2474 = ΣΣR
ΣX2
Σ(ΣR)2
Nk
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:9.9. Compute the mean values for each column.Compute the mean values for each column.
Sub # Trial A1 (Trial A1)2 Trial A2 (Trial A2)2 ΣR (ΣR)2
1 146 21316 140 19600 286 817962 148 21904 152 23104 300 900003 170 28900 152 23104 322 1036844 90 8100 99 9801 189 357215 157 24649 145 21025 302 912046 156 24336 153 23409 309 954817 176 30976 167 27889 343 1176498 205 42025 218 47524 423 178929
ΣC 1248 1226 2474 794464
Mean 156 25275.75 153.25 24432 ΣΣR Σ(ΣR)2
ΣXT 2474 = ΣΣR
ΣX2
Σ(ΣR)2
Nk
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:10.10. Sum the squares of columns = Sum the squares of columns = ΣΣ(Trial A1)(Trial A1)22
11.11. Sum the sum of squared columns = Sum the sum of squared columns = ΣΣ((ΣΣ(Trial A1)(Trial A1)22))
– This is also referred to as This is also referred to as ΣΣXX22. .
12.12. ΣΣ((ΣΣR)R)22 was calculated in step 8. was calculated in step 8.
13.13. N = the number of subjects.N = the number of subjects.
14.14. k = the number of trials.k = the number of trials.
Sub # Trial A1 (Trial A1)2 Trial A2 (Trial A2)2 ΣR (ΣR)2
1 146 21316 140 19600 286 817962 148 21904 152 23104 300 900003 170 28900 152 23104 322 1036844 90 8100 99 9801 189 357215 157 24649 145 21025 302 912046 156 24336 153 23409 309 954817 176 30976 167 27889 343 1176498 205 42025 218 47524 423 178929
ΣC 1248 202206 1226 195456 2474 794464
Mean 156 44934.667 153.25 24432 ΣΣR Σ(ΣR)2
ΣXT 2474 = ΣΣR
ΣX2 397662
Σ(ΣR)2 794464N 8k 2
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:15.15. Compute the sum of squares between columns (SSCompute the sum of squares between columns (SSCC), which is ), which is
the variability due to the repeated-measures treatment effect.the variability due to the repeated-measures treatment effect.– In this case, SSIn this case, SSCC is “systematic variability.” is “systematic variability.”
))((
)()()( 222
21
kN
X
N
CCSS T
C
)2)(8(
)2474(
8
)1226()1248( 222
CSS
25.30CSS
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:16.16. Compute the sum of squares between rows (SSCompute the sum of squares between rows (SSRR), ),
which is the variability due to differences among which is the variability due to differences among subjects.subjects.
))((
)()( 22
kN
X
k
RSS T
R
)2)(8(
)2474(
2
794464 2
RSS
75.14689RSS
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:17.17. Calculate the total sum of squares (SSCalculate the total sum of squares (SSTT), which is the ), which is the
variability due to subjects (rows), treatment variability due to subjects (rows), treatment (columns), and unexplained residual variability (columns), and unexplained residual variability (error).(error).
))((
)( 22
kN
XXSS T
T
25.382542397662 TSS
75.15119TSS
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:18.18. Calculate the total sum of squares due to error (SSCalculate the total sum of squares due to error (SSEE), ),
which is the unexplained variability due to error. This which is the unexplained variability due to error. This will be used in the denominator for the F ratio.will be used in the denominator for the F ratio.
75.1468925.3075.15119 ESS
RCTE SSSSSSSS
75.399ESS
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:19.19. Calculate the degrees of freedom for each source of Calculate the degrees of freedom for each source of
variance (dfvariance (dfCC, df, dfRR, df, dfEE, and df, and dfTT).).
1kdfC
)1)(1( NkdfE
1NdfR
1))(( kNdfT
12 Cdf
)18)(12( Edf
18 Rdf
1)2)(8( Tdf
1Cdf
7Edf
7Rdf
15Tdf
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:20.20. Construct an ANOVA table:Construct an ANOVA table:
Source df SS MS F p (sig.)
Between Subjects 7 14,689.75
Within Subjects 8 430
Trials 1 30.25
Error 7 399.75
Total 15 15,119.75
dfRdfC dfE dfT SSRSSC SSE SST
Between Subjects = rows
Trials = columns
Within Subjects = Trials + Error
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:21.21. Calculate the mean square for each source of Calculate the mean square for each source of
variance (MSvariance (MSCC, MS, MSRR, and MS, and MSEE).).
C
CC df
SSMS
E
EE df
SSMS
R
RR df
SSMS
25.30CMS
12.57EMS
54.2098RMS
Source df SS MS F p (sig.)
Between Subjects 7 14,689.75 2098.54
Within Subjects 8 430 53.75
Trials 1 30.25 30.25
Error 7 399.75 57.11
Total 15 15,119.75
Repeated Measures ANOVARepeated Measures ANOVA
Steps for calculation:Steps for calculation:22.22. Calculate the F ratio for the treatment effect Calculate the F ratio for the treatment effect
(columns, F(columns, FCC).).
E
CsubjectsWithin MS
MSF
E
RsubjectsBetween MS
MSF
53.0 subjectsWithinF
75.36 subjectsBetweenF
Source df SS MS F p (sig.)
Between Subjects 7 14,689.75 2098.54 36.75
Within Subjects 8 430 53.75
Trials 1 30.25 30.25 0.53
Error 7 399.75 57.11
Total 15 15,119.75
Repeated Measures ANOVARepeated Measures ANOVA
Determining the Significance of F:Determining the Significance of F:– Use the F Distribution Critical Values table.Use the F Distribution Critical Values table.
dfdfCC = df = dfB B – columns across the top– columns across the top
dfdfEE = df = dfEE – rows down the side – rows down the side
– If your calculated F ratio is greater than the critical F If your calculated F ratio is greater than the critical F ratio, then reject the null hypothesis.ratio, then reject the null hypothesis.
There is a significant difference from Trial A1 to Trial A2There is a significant difference from Trial A1 to Trial A2
There is a significant systematic errorThere is a significant systematic error
– If your calculated F ratio is less than the critical F If your calculated F ratio is less than the critical F ratio, then accept the null hypothesis.ratio, then accept the null hypothesis.
There is no difference from Trial A1 to Trial A2There is no difference from Trial A1 to Trial A2
There is no systematic errorThere is no systematic error
Source df SS MS F p (sig.)
Between Subjects 7 14,689.75 2098.54 36.75 < 0.05
Within Subjects 8 430 53.75
Trials 1 30.25 30.25 0.53 > 0.05
Error 7 399.75 57.11
Total 15 15,119.75
Source df SS MS F p (sig.)
Between Subjects 7 14,689.75 2098.54 36.75 0.00005
Within Subjects 8 430 53.75
Trials 1 30.25 30.25 0.53 0.49035
Error 7 399.75 57.11
Total 15 15,119.75
Using ANOVA Table for ICCUsing ANOVA Table for ICC
2 sources of variability for ICC model 3,12 sources of variability for ICC model 3,1– Subjects (MSSubjects (MSSS))
Between-subjects variability (for calculating the ICC)Between-subjects variability (for calculating the ICC)
– Error (MSError (MSEE))
Random error (for calculating the ICC)Random error (for calculating the ICC)
ES
ES
MSkMS
MSMSICC
)1(1,3
ER
ER
MSkMS
MSMSICC
)1(1,3
Equation reported by Weir (2005) Same equation, but modified for our terminology (MSS = MSR).
Source df SS MS F p (sig.)
Between Subjects 7 14,689.75 2098.54 36.75 0.00005
Within Subjects 8 430 53.75
Trials 1 30.25 30.25 0.53 0.49035
Error 7 399.75 57.11
Total 15 15,119.75
MSE
MSR or MSS
Using ANOVA Table for ICCUsing ANOVA Table for ICC
Calculating the ICCCalculating the ICC3,13,1::
ER
ER
MSkMS
MSMSICC
)1(1,3
11.57)12(54.2098
11.5754.20981,3
ICC
947.01,3 ICC
Interpreting the ICCInterpreting the ICC
If ICC = 0.95If ICC = 0.95– 95% of the observed score variance is due to true 95% of the observed score variance is due to true
score variancescore variance– 5% of the observed score variance is due to error5% of the observed score variance is due to error
2 factors for examining the magnitude of the ICC2 factors for examining the magnitude of the ICC– Which version of the ICC was used?Which version of the ICC was used?– Magnitude of the ICC depends on the between-Magnitude of the ICC depends on the between-
subjects variability in the datasubjects variability in the dataBecause of the relationship between the ICC magnitude and Because of the relationship between the ICC magnitude and between-subjects variability, standard error of measurement between-subjects variability, standard error of measurement values (SEM) should be included with the ICCvalues (SEM) should be included with the ICC
Implications of a Low ICCImplications of a Low ICC
Low reliabilityLow reliabilityReal differencesReal differences– Argument to include SEM valuesArgument to include SEM values
Type I vs. Type II errorType I vs. Type II error– Type I error is rejecting HType I error is rejecting H00 when there was no effect when there was no effect
(i.e., H(i.e., H00 = 0) = 0)– Type II error is failing to reject the HType II error is failing to reject the H00 when there is an when there is an
effect (i.e., Heffect (i.e., H00 ≠ 0)≠ 0)
A low ICC means that more subjects will be A low ICC means that more subjects will be necessary to overcome the increased necessary to overcome the increased percentage of the observed score variance due percentage of the observed score variance due to error.to error.
Standard Error of MeasurementStandard Error of Measurement
ICC ICC relative measure of reliability relative measure of reliability– No unitsNo units
SEM SEM absolute index of reliability absolute index of reliability– Same units as the measurement of interestSame units as the measurement of interest
The SEM is the standard error in The SEM is the standard error in estimating observed scores from true estimating observed scores from true scores.scores.
Calculating the SEMCalculating the SEM
Calculating the SEMCalculating the SEM3,13,1::
EMSSEM
11.57SEM
56.7SEM
SEMSEM
We can report SEM values in addition to We can report SEM values in addition to the ICC values and the results of the the ICC values and the results of the ANOVAANOVA
We can calculate the minimum difference We can calculate the minimum difference (MD) that can be considered “real” (MD) that can be considered “real” between scoresbetween scores
Minimum DifferenceMinimum Difference
The SEM can be used to determine the The SEM can be used to determine the minimum difference (MD) to be considered minimum difference (MD) to be considered “real” and can be calculated as follows:“real” and can be calculated as follows:
296.1SEMMD
296.156.7MD
95.20MD
Example ProblemExample Problem
Now use your skills (by hand) to calculate Now use your skills (by hand) to calculate a repeated measures ANOVA, ICCa repeated measures ANOVA, ICC3,13,1, ,
SEMSEM3,13,1, and MD, and MD3,13,1 for Trials B1 and B2. for Trials B1 and B2.
– Report your results.Report your results.– Compare your results to Trials A1 and A2.Compare your results to Trials A1 and A2.
What is the primary difference?What is the primary difference?
Trial A1Trial A1 Trial A2Trial A2 Trial B1Trial B1 Trial B2Trial B2
146146 140140 -6-6 166166 160160 - 6- 6
148148 152152 + 4+ 4 168168 172172 + 4+ 4
170170 152152 - 18- 18 160160 142142 - 18- 18
9090 9999 + 9+ 9 150150 159159 + 9+ 9
157157 145145 - 12- 12 147147 135135 - 12- 12
156156 153153 + 3+ 3 146146 143143 - 3- 3
176176 167167 - 9- 9 156156 147147 - 9- 9
205205 218218 + 13+ 13 155155 168168 + 13+ 13
156 156 ++ 33 33 153 153 ++ 33 33 156 156 ++ 8 8 153 153 ++ 13 13
Using the Reliability Worksheet Using the Reliability Worksheet OnlineOnline
Go to the course website and download Go to the course website and download the Reliability.xls worksheet.the Reliability.xls worksheet.– Calculate the ANOVA, ICC, SEM, and MD Calculate the ANOVA, ICC, SEM, and MD
values for both Trials A1 and A2 and Trials B1 values for both Trials A1 and A2 and Trials B1 and B2 and compare your results.and B2 and compare your results.