Download - Screening and Diagnostic
-
8/21/2019 Screening and Diagnostic
1/82
Copyright 2008, The Johns Hopkins University and Sukon Kanchanaraksa. All rights reserved. Use of these
materials permitted only in accordance with license rights granted. Materials provided “AS IS”; no representations orwarranties provided. User assumes all responsibility for use, and all liability related thereto, and must independently
review all materials for accuracy and efficacy. May contain materials owned by others. User is responsible forobtaining permissions for use from third parties as needed.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike License. Your use of thismaterial constitutes acceptance of that license and the conditions of use of materials on this site.
http://creativecommons.org/licenses/by-nc-sa/2.5/http://creativecommons.org/licenses/by-nc-sa/2.5/
-
8/21/2019 Screening and Diagnostic
2/82
Evaluation of Diagnostic and Screening Tests:
Validity and Reliability
Sukon Kanchanaraksa, PhDJohns Hopkins University
-
8/21/2019 Screening and Diagnostic
3/82
Section A
Sensitivity and Specificity
-
8/21/2019 Screening and Diagnostic
4/82
4
Correctly Classifying Individuals by Disease Status
Tests are used in medical diagnosis, screening, and research
How well is a subject classified into disease or non-diseasegroup?
−
Ideally, all subjects who have the disease should beclassified as “having the disease”
and vice versa
−
Practically, the ability to classify individuals into thecorrect disease status depends on the accuracy of thetests, among other things
-
8/21/2019 Screening and Diagnostic
5/82
5
Diagnostic Test and Screening Test
A diagnostic test is used to determine the presence orabsence of a disease when a subject shows signs or
symptoms of the disease A screening test identifies asymptomatic individuals who
may have the disease
The diagnostic test is performed after a positive screeningtest to establish a definitive diagnosis
-
8/21/2019 Screening and Diagnostic
6/826
Some Common Screening Tests
Pap smear for cervical dysplasia or cervical cancer
Fasting blood cholesterol for heart disease
Fasting blood sugar for diabetes
Blood pressure for hypertension
Mammography for breast cancer
PSA test for prostate cancer
Fecal occult blood for colon cancer
Ocular pressure for glaucoma
PKU test for phenolketonuria in newborns
TSH for hypothyroid and hyperthyroid
-
8/21/2019 Screening and Diagnostic
7/827
Variation in Biologic Values
Many test results have a continuous scale (are continuousvariables)
Distribution of biologic measurements in humans may or maynot permit easy separation of diseased from non-diseasedindividuals, based upon the value of the measurement
-
8/21/2019 Screening and Diagnostic
8/828
N u m b e r o f
s u b j e c t s
0
5
10
15
20
25
3 9 15 21 27Diameter of induration (mm)
Distribution of Tuberculin Reactions
Source: Edwards et al, WHO Monograph 12, 1953
-
8/21/2019 Screening and Diagnostic
9/829
N u m b e r o f m e n
0
5
10
15
20
25
80 90 100 110 120 130 140 150 160 170 180 190 200
Systolic blood pressure in mm of mercury
Source: Data of W.W. Holland
Distribution of Systolic Blood Pressures:
744 Employed White Males, Ages 40–64
-
8/21/2019 Screening and Diagnostic
10/8210
Validity
Validity is the ability of a test to indicate which individualshave the disease and which do not
-
8/21/2019 Screening and Diagnostic
11/8211
Sensitivity and Specificity
Sensitivity
−
The ability of the test to identify correctly those who have
the disease
Specificity
−
The ability of the test to identify correctly those who do
not have
the disease
-
8/21/2019 Screening and Diagnostic
12/82
12
Determining the Sensitivity, Specificity of a New Test
Must know the correct disease status prior to calculation
Gold standard test is the best test available
−
It is often invasive or expensive
A new test is, for example, a new screening test or a lessexpensive diagnostic test
Use a 2 x 2 table to compare the performance of the new testto the gold standard test
-
8/21/2019 Screening and Diagnostic
13/82
13
Disease
Gold Standard Test
+ –
a+c(All people with
disease)
b+d(All people
without
disease)
-
8/21/2019 Screening and Diagnostic
14/82
14
Disease
New test(True positives)
(True negatives)
Comparison of Disease Status:
Gold Standard Test and New Test
+ –
+
a b
– c d
-
8/21/2019 Screening and Diagnostic
15/82
15
Sensitivity is the ability of the test to identify correctly thosewho have the disease (a) from all individuals with the disease
(a+c)
Sensitivity is a fixed characteristic of the test
sensitivity= aa+c
= truepositivesdisease+
=Pr(T+ |D+)
Sensitivity
-
8/21/2019 Screening and Diagnostic
16/82
16
Specificity is the ability of the test to identify correctly thosewho do not have the disease (d) from all individuals free from
the disease (b+d)
Specificity is also a fixed characteristic of the test
Specificity
sensitivity = db + d
=true negatives
disease −
= Pr(T− | D−)
-
8/21/2019 Screening and Diagnostic
17/82
17
True Characteristics in Population Screening
Results Disease No Disease
Total
Positive 80 100 180
Negative 20 800 820
Total 100 900 1,000
Applying Concept of Sensitivity and Specificity to a
Screening Test
Assume a population of 1,000 people
100 have a disease
900 do not have the disease
A screening test is used to identify the 100 people with thedisease
The results of the screening appears in this table
-
8/21/2019 Screening and Diagnostic
18/82
18
Calculating Sensitivity and Specificity
True Characteristics in Population ScreeningResults Disease No Disease
Total
Positive 80 100 180
Negative 20 800 820
Total 100 900 1,000
Sensitivity
= Specificity
= 800/900 = 89%80/100 = 80%
-
8/21/2019 Screening and Diagnostic
19/82
19
True Characteristics in Population ScreeningResults Disease No Disease
Total
Positive 80 100 180
Negative 20 800 820
Total 100 900 1,000
Evaluating Validity
Sensitivity
= 80/100 = 80% Specificity
= 800/900 = 89%
-
8/21/2019 Screening and Diagnostic
20/82
20
Examining the Effect of Changing Cut-Points
Example: type II diabetes mellitus
−
Highly prevalent in the older, especially obese, U.S.
population−
Diagnosis requires oral glucose tolerance test
−
Subjects drink a glucose solution, and blood is drawn at
intervals for measurement of glucose−
Screening test is fasting plasma glucose
Easier, faster, more convenient, and less expensive
http://en.wikipedia.org/wiki/Oral_glucose_tolerance_testhttp://care.diabetesjournals.org/cgi/content/full/25/suppl_1/s21
-
8/21/2019 Screening and Diagnostic
21/82
21
Blood
sugar
Low
High20 diabetics
20 non-diabetics
20 diabetics
20 non-diabetics
Diabetics Non-diabetics
Concept of Sensitivity and Specificity
-
8/21/2019 Screening and Diagnostic
22/82
22
Subjects arescreened using
fasting plasmaglucose with a low(blood sugar) cut-
point
Subjects arescreened using
fasting plasmaglucose with a low(blood sugar) cut-
pointBlood
sugar
Low
High
Diabetics Non-diabetics
Concept of Sensitivity and Specificity
Diabetics Non-Diabetics
+ 17 14
– 3 6
20 20
Sens=85% Spec=30%
-
8/21/2019 Screening and Diagnostic
23/82
23
Subjects arescreened using
fasting plasmaglucose with a highcut-point
Subjects arescreened using
fasting plasmaglucose with a highcut-point
Blood
sugar
Low
High
Diabetics Non-diabetics
Concept of Sensitivity and Specificity
Diabetics Non-Diabetics
+ 5 2
– 15 18
20 20
Sens=25% Spec=90%
-
8/21/2019 Screening and Diagnostic
24/82
24
In a typicalpopulation, there is
no line separatingthe two groups, andthe subjects are
mixed
In a typicalpopulation, there is
no line separatingthe two groups, andthe subjects aremixed
Blood
sugar
Low
High
Diabetics Non-diabetics
Concept of Sensitivity and Specificity
-
8/21/2019 Screening and Diagnostic
25/82
25
In a typicalpopulation, there is
no line separatingthe two groups, andthe subjects are
mixed
In a typicalpopulation, there is
no line separatingthe two groups, andthe subjects aremixed
Blood
sugar
Low
High
Diabetics Non-diabetics
Concept of Sensitivity and Specificity
-
8/21/2019 Screening and Diagnostic
26/82
26
In fact, there is nocolor or label
In fact, there is nocolor or label
Blood
sugar
Low
High
Diabetics Non-diabetics
Concept of Sensitivity and Specificity
-
8/21/2019 Screening and Diagnostic
27/82
27
A screening testusing a high cut-
point will treat thebottom box asnormal and will
identify the 7subjects above theline as havingdiabetes
A screening testusing a high cut-
point will treat thebottom box asnormal and willidentify the 7subjects above theline as havingdiabetes
Blood
sugar
Low
High
Diabetics Non-diabetics
Concept of Sensitivity and Specificity
-
8/21/2019 Screening and Diagnostic
28/82
28
A screening testusing a high cut-
point will treat thebottom box asnormal and will
identify the 7subjects above theline as havingdiabetes;
But a low cut-pointwill result inidentifying 31
subjects as havingdiabetes
A screening testusing a high cut-
point will treat thebottom box asnormal and willidentify the 7subjects above theline as havingdiabetes;
But a low cut-pointwill result inidentifying 31
subjects as havingdiabetes
Blood
sugar
Low
High
Diabetics Non-diabetics
Concept of Sensitivity and Specificity
-
8/21/2019 Screening and Diagnostic
29/82
29
Lessons Learned
Different cut-points yield different sensitivities andspecificities
The cut-point determines how many subjects will beconsidered as having the disease
The cut-point that identifies more true negatives will also
identify more false negatives The cut-point that identifies more true positives will also
identify more false positives
-
8/21/2019 Screening and Diagnostic
30/82
30
Where to Draw the Cut-Point
If the diagnostic (confirmatory) test is expensive or invasive:
−
Minimize false positives
or−
Use a cut-point with high specificity
If the penalty for missing a case is high (e.g., the disease is
fatal and treatment exists, or disease easily spreads):−
Maximize true positives
That is, use a cut-point with high sensitivity
Balance severity of false positives against false negatives
-
8/21/2019 Screening and Diagnostic
31/82
31
Disease
Test(True positives)
(True negatives)
Behind the Test Results
(False positives)
(False negatives)
+ –
+
a b
– c d
-
8/21/2019 Screening and Diagnostic
32/82
32
Review
Fill in the missing cells and calculate sensitivity and specificityfor this example
True Characteristics in Population ScreeningResults Disease No Disease
Total
Positive 240Negative 600
Total 300 700 1,000
-
8/21/2019 Screening and Diagnostic
33/82
Section B
Multiple Testing
-
8/21/2019 Screening and Diagnostic
34/82
34
Use of Multiple Tests
Commonly done in medical practice
Choices depend on cost, invasiveness, volume of test,
presence and capability of lab infrastructure, urgency, etc. Can be done sequentially or simultaneously
-
8/21/2019 Screening and Diagnostic
35/82
35
Sequential Testing (Two-Stage Screening)
After the first (screening) test was conducted, those whotested positive were brought back for the second test to
further reduce false positives Consequently, the overall process will increase specificity but
with reduced sensitivity
Example of a Two Stage Screening Program:
-
8/21/2019 Screening and Diagnostic
36/82
36
Test 1 (blood sugar), assume:
−
Disease prevalence = 5%, population = 10,000
−
Sensitivity = 70%, specificity = 80%
−
Screen positives
from the first test
Diabetes
T e s t r e
s u l t s
350
150
1900
7600
+ –
+
–
70% x 500=350
80% x 9500=7600
9500–7600=1900
500–350=150
5005% x 10,000 = 500 9500
2250
7750
10,000
Example of a Two-Stage Screening Program:
Test 1 (Blood Sugar)
Example of a Two Stage Screening Program:
-
8/21/2019 Screening and Diagnostic
37/82
37
Diabetes
T e s t r e s u l t s
350
150
1900
7600
+ –
+
–
500 9500
2250
7750
10,000
Example of a Two-Stage Screening Program:
Test 2 (Glucose Tolerance Test)
Test 1 (blood sugar)
−
Sensitivity = 70%
−
Specificity = 80%
Test 2 (glucose
tolerance test)
−
Sensitivity = 90%−
Specificity = 90%
350 1900 2250
Diabetes
T e s t r e s
u l t s
+ –
+
–
315
35
190
1710
505
1745
Example of a Two Stage Screening Program:
-
8/21/2019 Screening and Diagnostic
38/82
38
Diabetes
T e s t r e s u l t s
350
150
1900
7600
+ –
+
–
500 9500
2250
7750
10,000
Example of a Two-Stage Screening Program:
Test 2 (Glucose Tolerance Test)
Test 1 (blood sugar)
−
Sensitivity = 70%
−
Specificity = 80%
Test 2 (glucose
tolerance test)
−
Sensitivity = 90%−
Specificity = 90%
350 1900 2250
Net sensitivity =
New specificity =
315500
= 63%
7600+1710
9500=98%
Diabetes
T e s t r e s
u l t s
+ –
+
–
315
35
190
1710
505
1745
Two Stage Screening: Re Screen the Positives from the
-
8/21/2019 Screening and Diagnostic
39/82
39
Two-Stage Screening: Re-Screen the Positives from the
First Test
Subject is disease positive when test positive in both tests
Subject is disease negative when test negative in either test
A I B
A U B
Net Sensitivity in a Two-Stage Screening when Test + in
-
8/21/2019 Screening and Diagnostic
40/82
40
Net Sensitivity in a Two-Stage Screening when Test + in
the First Test Are Re-Screened
The multiplication rule of probability is
When events are independent (two tests are independent), then
thus
The multiplication rule of probability is
When events are independent (two tests are independent), then
thus
P(AIB) = P(B)∗P(A|B)
) A (P=)B| A (P
P(A I B) =
P(A) ∗
P(B) Net sensitivity = Sensitivity 1 x Sensitivity 2
Net Specificity in a Two-Stage Screening when Test + in the
-
8/21/2019 Screening and Diagnostic
41/82
41
Use addition rule of probabilityUse addition rule of probability
Net Specificity in a Two-Stage Screening when Test + in the
First Test Are Re-Screened
Net specificity = Spec1 + Spec2 – (Spec1 x Spec2)
P(AU
B)=
P(A)+
P(B)−
P(AI
B)
-
8/21/2019 Screening and Diagnostic
42/82
42
Other Two-Stage Screening
Screen the negatives from the first test to identify any missedtrue positives from the first test
−
Net sensitivity and net specificity calculation followssimilar but different logical algorithms
−
What is the net effect of testing the negatives from the
first test? Find more true positives => net sensitivity will be
higher than sensitivity from the individual tests
Also find more false positives => net specificity willbe lower than specificity from the individual tests
-
8/21/2019 Screening and Diagnostic
43/82
43
Simultaneous Testing
When two (or more) tests are conducted in parallel
The goal is to maximize the probability that subjects with the
disease (true positives) are identified (increase sensitivity) Consequently, more false positives are also identified
(decrease specificity)
-
8/21/2019 Screening and Diagnostic
44/82
44
Simultaneous Testing: Calculate Net Sensitivity
When two tests are used simultaneously, disease positives aredefined as those who test positive by either one test or by
both tests We use the addition rule of probability to calculate the net
sensitivity
Net sensitivity = sens 1 + sens 2 – (sens 1 x sens 2)
P(AUB) = P(A)+ P(B)−P(AIB)
-
8/21/2019 Screening and Diagnostic
45/82
45
When two tests are used simultaneously, disease negativesare defined as those who test negative by both tests
We use the multiplication rule of probability to calculate thenet specificity
Net specificity = specificity test 1 x specificity test 2
Simultaneous Testing: Calculate Net Specificity
P(A I B) = P(A) ∗ P(B)
-
8/21/2019 Screening and Diagnostic
46/82
46
Net sensitivity
= sens 1 + sens 2 –
sens1 x sens 2
= 80% + 90% –
(80% x 90%)
= 98%
Net sensitivity
= sens 1 + sens 2 –
sens1 x sens 2
= 80% + 90% –
(80% x 90%)
= 98%
Net specificity
= spec 1 x spec 2
= 60% x 90%
= 54%
Net specificity
= spec 1 x spec 2
= 60% x 90%
= 54%
Example of a Simultaneous Testing
In a population of 1000, the prevalence of disease is 20%
Two tests (A and B) are used at the same time
Test A has sensitivity of 80% and specificity of 60% Test B has sensitivity of 90% and specificity of 90%
Calculate net sensitivity and net specificity from using Test A
and Test B simultaneously
-
8/21/2019 Screening and Diagnostic
47/82
47
Net Gain and Net Loss
In simultaneous testing, there is a net gain in sensitivity but anet loss in specificity, when compared to either of the tests
used In sequential testing when positives from the first test are re-
tested, there is a net loss in sensitivity but a net gain in
specificity, compared to either of the tests used
-
8/21/2019 Screening and Diagnostic
48/82
48
Review
Test A is known to have the following characteristics:
−
Sensitivity of 80%
−
Specificity of 90%−
Cost of $15 per test
Suppose the following:
−
Test A is used in a population of 10,000 to identifyindividuals who have the disease
−
The prevalence of the disease is 5%
What are the net sensitivity, net specificity, and cost perpositive case when: (1) Test A is used twice simultaneouslyand when (2) a single Test A is used first, and individuals who
test positive with Test A are tested again with Test A(sequentially)
-
8/21/2019 Screening and Diagnostic
49/82
Section C
Predictive Values
-
8/21/2019 Screening and Diagnostic
50/82
50
Predictive Values
Positive predictive value (PPV)
−
The proportion of patients who test positive who actually
have the disease Negative predictive value (NPV)
−
The proportion of patients who test negative who are
actually free of the disease Note: PPV and NPV are not fixed characteristics of the test
-
8/21/2019 Screening and Diagnostic
51/82
51
Another Interpretation of PPV
If a person tests positive, what is the probability that he or shehas the disease?
(And if that person tests negative, what is the probability thathe or she does not have the disease?)
-
8/21/2019 Screening and Diagnostic
52/82
52
Disease
Test(True positives)
(True negatives)
Behind the Test Results
(False positives)
(False negatives)
+ –
+
a b
– c d
-
8/21/2019 Screening and Diagnostic
53/82
53
Disease
Test
What the Test Shows
+ –
+
a + b
– c + d
(All people with positive results)
(All people with negative results)
-
8/21/2019 Screening and Diagnostic
54/82
54
=a
a+ b
= True Positives Test +
= P(D+ | T+)
=d
c + d=
True Negatives
Test – =
P(D– | T–)
Predictive Value
Positive predictive value
Negative predictive value
-
8/21/2019 Screening and Diagnostic
55/82
55
True Characteristics in Population Screening
Results
Disease No Disease
Total
Positive 80 100 180
Negative 20 800 820
Total 100 900 1,000
Applying Concept of Predictive Values to Screening Test
Assume a population of 1,000 people
100 have a disease
900 do not have the disease A screening test is used to identify the 100 people with the
disease
The results of the screening appear in this table
-
8/21/2019 Screening and Diagnostic
56/82
56
Calculating Predictive Values
True Characteristics in Population
ScreeningResults Disease No Disease
Total
Positive 80 100 180
Negative 20 800 820 Total 100 900 1,000
Positive predictive value
=80/180 = 44%
Negative predictive value
= 800/820 = 98%
-
8/21/2019 Screening and Diagnostic
57/82
57
Calculating Predictive Values
True Characteristics in Population
ScreeningResults Disease No Disease
Total
Positive 80 100 180
Negative 20 800 820 Total 100 900 1,000
-
8/21/2019 Screening and Diagnostic
58/82
58
PPV Primarily Depends On …
The prevalence of the disease in the population tested, andthe test itself (sensitivity and specificity)
−
In general, it depends more on the specificity (and less onthe sensitivity) of the test (if the disease prevalence is low)
-
8/21/2019 Screening and Diagnostic
59/82
59
PPV Formula
For those who are interested
Use Bayes’ theorem
PPV = sensitivity x prevalence
(sensitivity x prevalence) + (1- specificity) x (1- prevalence)
NPV = specificity x (1– prevalence)
[(specificity x (1– prevalence)] + [(1- sensitivity) x prevalence]
-
8/21/2019 Screening and Diagnostic
60/82
60
Calculation of PPV and NPV
Construct the table and use the definition to guide thecalculation of PPV and NPV
[Or, use the formula] In a multiple testing situation, PPV and NPV are calculated for
each test (not for the combined test)
-
8/21/2019 Screening and Diagnostic
61/82
61
Relationship of Disease Prevalence to Predictive Value
Example: Sensitivity = 99%; Specificity = 95%
DiseasePrevalence TestResults Sick Not Sick Totals PositivePredictive Value
+ 99 495 594
– 1 9,405 9,4061% Totals 100 9,900 10,000
99
594 =17%
+ 495 475 970
– 5 9,025 9,3035%
Totals 500 9,500 10,000
495970
=51%
-
8/21/2019 Screening and Diagnostic
62/82
62
Prevalence of Disease
0
10
20
3040
50
60
70
8090
100
0 20 40 60 80 100
Prev alence o f Disease (%)
P r e d i c t e d V a l u e ( % )
Positive Test Negative Test
For test with
95% sensitivity95% specificity
For test with
95% sensitivity95% specificity
Adapted from Mausner JS, Kramer S.Epidemiology: an Introductory Text. Philadelphia, WB Saunders 1985, p221.
-
8/21/2019 Screening and Diagnostic
63/82
63
So If a Person Tests Positive …
The probability that he or she has the disease depends on theprevalence of the disease in the population tested and the
validity of the test (sensitivity and specificity) In general, specificity has more impact on predictive values
-
8/21/2019 Screening and Diagnostic
64/82
64
Prevalence = 50%
Sensitivity = 50%Specificity = 50%
Disease+ –
+
–
Test
250 250
250250
500
500
The Relationship of Specificity to Predictive Value
PPV =250
500
=50%
500 500 1,000
-
8/21/2019 Screening and Diagnostic
65/82
65
Prevalence = 20%
Sensitivity = 50%
Specificity = 50%
Disease+ –
+
–
Test
100 400
200 800 1,000
The Relationship of Specificity to Predictive Value
PPV =100
500
=20%
100 400
500
500
Change prevalenceChange prevalence
-
8/21/2019 Screening and Diagnostic
66/82
66
Prevalence = 20%
Sensitivity = 90%
Specificity = 50%
Disease+ –
+
–
Test
180 400
800 1,000
The Relationship of Specificity to Predictive Value
PPV =180
580
=31%
400
580
420
Change sensitivityChange sensitivity20
200
-
8/21/2019 Screening and Diagnostic
67/82
67
The Relationship of Specificity to Predictive Value
Prevalence = 20%
Sensitivity = 50%
Specificity = 90%
Disease+ –
+
–
Test
10080
200 800 1,000
PPV =100
180
=56%
100 720
180
820
Restore sensitivityChange specificityRestore sensitivityChange specificity
More impact than
changing sensitivityMore impact than
changing sensitivity
Age-Specific Breast Cancer Incidence Rates U.S., All Races
-
8/21/2019 Screening and Diagnostic
68/82
68
Rates per 100,000 Population of the Specified Five-year Age Group
I n c i d e
n c e
R a t e
0
50
100150
200
250
300
25–29 35–39 45–49 55–59 65–69 75–79
Five-Year Age
Group
(SEER 1984-88)
Source: SEER Program.
Results of First Screening Mammography by Age Group
-
8/21/2019 Screening and Diagnostic
69/82
69
— UCSF Mobile Mammography Program
Age (Years)Cancer
DetectedNo CancerDetected
TotalAbnormal
PositivePredictive
Value30–39 9 273 282 3%
40–49 26 571 597 4%
50–59 30 297 327 9%
60–69 46 230 276 17%
70 26 108 134 19%
PPV of First Screening Mammography
-
8/21/2019 Screening and Diagnostic
70/82
70
by Age and Family History of Breast Cancer
Age (Years)Women without a FamilyHistory of Breast Cancer
Women with a FamilyHistory of Breast Cancer
30–39 3% 4%
40–49 4% 13%
50–59 9% 22%
60–69 17% 14%
70 19% 24%
-
8/21/2019 Screening and Diagnostic
71/82
71
Age < 50 Years >
50 Years
Positive predictive value 4% 14%
Consequence of Different PPVs
Source: Kerlikowske et al. (1993). JAMA, 270:2444–2450.
-
8/21/2019 Screening and Diagnostic
72/82
Section D
Reliability (Repeatability)
-
8/21/2019 Screening and Diagnostic
73/82
73
Reproducibility, Repeatability, Reliability
Reproducibility, repeatability, reliability all mean that theresults of a test or measure are identical or closely similar each
time it is conducted Because of variation in laboratory procedures, observers, or
changing conditions of test subjects (such as time, location), a
test may not consistently yield the same result whenrepeated
Different types of variation
−
Intra-subject variation−
Intra-observer variation
−
Inter-observer variation
-
8/21/2019 Screening and Diagnostic
74/82
74
Intra-Subject Variation
Intra-subject variation is a variation in the results of a testconducted over (a short period of) time on the same
individual The difference is due to the changes (such as physiological,
environmental, etc.) occurring to that individual over that
time period
l d d d
-
8/21/2019 Screening and Diagnostic
75/82
75
Variation in Blood Pressure Readings: A 24-Hour Period
Blood Pressure(mm Hg)
Female27 Years Old
Female62 Years Old
Male33 Years Old
Basal 110/70 132/82 152/109
Lowest hour 86/47 102/61 123/78
Highest hour 126/79 172/94 153/107
Casual 108/64 155/93 157/109
I Ob d I Ob V
-
8/21/2019 Screening and Diagnostic
76/82
76
Inter-Observer and Intra-Observer Variation
Inter-observer variation is a variation in the result of a testdue to multiple observers examining the result (inter =
between) Intra-observer variation is a variation in the result of a test
due to the same observer examining the result at different
times (intra = within) The difference is due to the extent to which observer(s) agree
or disagree when interpreting the same test result
Agreement between Two Observers
(O T Ob i )
-
8/21/2019 Screening and Diagnostic
77/82
77
(Or Two Observations)
A perfect agreement occurs when:
−
b=0
−
c=0
Observer 1
Positive Negative
Observer 2
Positive a b
Negative c d
P A
-
8/21/2019 Screening and Diagnostic
78/82
78
Overall Percent Agreement = a + d
a + b + c + dx 100
Percent Positive Agreement =
a
a + b + cx 100
Note: This is a conditional probability
Percent Agreement
E l
-
8/21/2019 Screening and Diagnostic
79/82
79
Radiologist A
Positive Negative
Radiologist B
Positive
4 5
Negative 2 6
Overall Percent Agreement =
4 + 6
4 + 5 + 2 + 6x 100 = 58.8%
Percent Positive Agreement = 4
4 + 5 + 2x 100 = 36.4%
Example
Observer or Instrument Variation:
O ll P t A t
-
8/21/2019 Screening and Diagnostic
80/82
80
Reading #1
Reading #2 Abnormal Suspect Doubtful Normal
Abnormal A B C D
Suspect E F G H
Doubtful I J K L
Normal M N O P
Percent agreement=A+F+K +P
Total x100
Overall Percent Agreement
+
+
+
O t f T t R lt
-
8/21/2019 Screening and Diagnostic
81/82
81
Outcome of Test Results
Valid but not reliableValid but not reliable
Valid and reliableValid and reliable
Reliable but not validReliable but not valid
Test results
Test results
Test results
True value
R i
-
8/21/2019 Screening and Diagnostic
82/82
Review
Define
−
Overall percent agreement
−
Percent positive agreement Contrast overall percent agreement and percent positive
agreement