STATISTICAL EVALUATION STATISTICAL EVALUATION OF DIAGNOSTIC TESTSOF DIAGNOSTIC TESTS

Describing the performance of a new diagnostic testDescribing the performance of a new diagnostic test

Physicians are often faced with the task of evaluation

the merit of a new diagnostic test. An adequate critical

appraisal of a new test requires a working knowledge

of the properties of diagnostic tests and the

mathematical relationships between them.

The gold standard test: Assessing a new diagnostic

test begins with the identification of a group of

patients known to have the disorder of interest, using

an accepted reference test known as the gold

standard.

Limitations:

1) The gold standard is often the most risky,

technically difficult, expensive, or impractical of

available diagnostic options.

2) For some conditions, no gold standard is

available.

The basic idea of diagnostic test interpretation is to

calculate the probability a patient has a disease under

consideration given a certain test result.  A 2 by 2 table

can be used for this purpose. Be sure to label the table

with the test results on the left side and the disease

status on top as shown here: 

Test Disease

Present Absent

Positive True Positive False Positive

Negative False Negative True Negative

The sensitivity of a diagnostic test is the

probability that a diseased individual will have a

positive test result. Sensitivity is the true positive

rate (TPR) of the test.

Sensitivity = P(T+|D+)=TPR

= TP / (TP+FN)

diseased all

test positive withdiseased

The specificity of a diagnostic test is the

probability that a disease-free individual will have a

negative test result. Specificity is the true negative

rate (TNR) of the test.

Specificity=P(T-|D-) = TNR

=TN / (TN + FP).

free-disease all

test negative withfree-disease

False-positive rate: The likelihood that a nondiseased patient has an abnormal test result.

FPR = P(T+|D-)=

= FP / (FP+TN)

free-diseased all

test positive withfree-disease

False-negative rate: The likelihood that a diseased patient has a normal test result.

FNR = P(T-|D+)=

= FN / (FN+TP)

diseased all

test negative withdiseased

Pretest Probability is the estimated likelihood of

disease before the test is done.

It is the same thing as prior probability and is often

estimated. If a defined population of patients is being

evaluated, the pretest probability is equal to the

prevalence of disease in the population. It is the

proportion of total patients who have the disease.

P(D+) = (TP+FN) / (TP+FP+TN+FN)

Sensitivity and specificity describe how well the test

discriminates between patients with and without

disease. They address a different question than we

want answered when evaluating a patient, however.

What we usually want to know is: given a certain test

result, what is the probability of disease? This is the

predictive value of the test.

Predictive value of a positive test is the proportion

of patients with positive tests who have disease.

PVP=P(D+|T+) = TP / (TP+FP)

This is the same thing as posttest probability of

disease given a positive test. It measures how well

the test rules in disease.

Predictive value of a negative test is the

proportion of patients with negative tests who do not

have disease. In probability notation:

PVN = P(D-|T-) = TN / (TN+FN)

It measures how well the test rules out disease. This

is posttest probability of non-disease given a

negative test.

Evaluating a 2 by 2 table is simple if you are methodical in your approach. 

Test Disease

Present Absent

Positive TP FP Total positive

Negative FN TN Total negative

Total with disease

Total with- out disease

Grand total

Bayes’ Rule MethodBayes’ Rule MethodBayes’ rule is a mathematical formula that may be used as an alternative to the back calculation method for obtaining unknown conditional probabilities such as PVP or PVN from known conditional probabilities such as sensitivity and specificity.

FPRDpTPRDp

TPRDpTDPPVP

))(1()(

)()(

FNRDpTNRDp

TNRDpTDPPVN

))(1()(

)()(

)()()()(

)()()(

ABPAPABPAP

ABPAPBAP

General form of Bayes’ rule is

Using Bayes’ rule, PVP and PVN are defined as

Example The following table summarizes results of a study to evaluate the dexamethasone suppression test (DST) as a diagnostic test for major depression. The study compared results on the DST to those obtained using the gold standard procedure (routine psychiatric assessment and structured interview) in 368 psychiatric patients.

1. What is the prevalence of major depression in the study group?

2. For the DST, determine

a-Sensitivity and specificity

b-False positive rate (FPR) and false negative rate (FNR)

c-Predictive value positive (PVP) and predictive value negative (PVN)

DST

Result

Depression Total

+ -

+ 84 5 89

- 131 148 279

Total 215 153 368

Sensitivity = P(T+|D+)=TPR=TP/(TP+FN)=84/215=0.391

Specificity=P(T-|D-)=TNR=TN / (TN + FP)=148/153=0.967

FPR = P(T+|D-)=FP/(FP+TN)=5/153=0.033

FNR = P(T-|D+)=FN/(FN+TP)=131/215=0.609

PVN = P(D-|T-) = TN / (TN+FN)=148/279=0.53

PVP=P(D+|T+) = TP / (TP+FP)=84/89=0.944

P(D+) =215/368 =0.584

FNR=1-Sensitivity=1-0.391=0.609

FPR=1-Specificity=1-0.967=0.033

ROC (Receiver Operating Characteristic ) CURVEROC (Receiver Operating Characteristic ) CURVE

The ROC Curve is a graphic representation of the

relationship between sensitivity and specificity for a

diagnostic test. It provides a simple tool for applying the

predictive value method to the choice of a positivity

criterion.

ROC Curve is constructed by plottting the true positive rate

(sensitivity) against the false positive rate (1-specificty) for

several choices of the positivity criterion.

ROC Curve

Diagonal segments are produced by ties.

1 - Specificity

1,00,75,50,250,00

Sensitivity

1,00

,75

,50

,25

0,00

Disease-free Diseased

Test negative Test positive

TP

FP

FN

TN

2 1

1

11

ix

z

2

22

ix

z

Positivity criterion

xi

Example: One of the parameters which are evaluated for the diagnosis of CHD, is the value of “HDL/Total Cholesterol”. Consider a population consisting of 67 patients with CHD, 93 patients without CHD. The result of HDL/Total Cholesterol values of these two groups of patients are as follows.

CHD+

Hdl/Total Cholestrol

CHD-

Hdl/Total Cholestrol

0,29

0,26

0,39

0,16

.

.

.

0,25

0,36

0,30

0,20

.

.

.

To construct the ROC Curve, we should find sensitivity and specificity for each cut off point. We have two alternatives to find these characteristics.

• Cross tables

• Normal Curve

Descriptive Statistics

HDL/Total Cholestrol

,2926 ,066 ,16 ,52,2301 ,048 ,06 ,34

GROUPCHD-CHD+

Mean SD Min Max

If HDL/Total Cholestrol is less than or equal to 0,26, we classify this group into diseased.

64 15 79

68,8% 22,4% 49,4%

29 52 81

31,2% 77,6% 50,6%

93 67 160

100,0% 100,0% 100,0%

Count

Count

Count

0,26>

0,26<=

RATIO

Total

- +

CHD

Total

SensitivitySpecificity

Best cutoff point

Cutoff TPR FPR0,000 0,000 0,0000,093 0,015 0,0000,129 0,030 0,0000,142 0,045 0,0000,156 0,060 0,0000,158 0,075 0,0000,162 0,075 0,0110,168 0,104 0,0110,171 0,119 0,0110,173 0,119 0,0220,175 0,119 0,032

. . .

. . .

. . .0.26 0.78 0.31

. . .

. . .0,393 1,000 0,9350,402 1,000 0,9460,407 1,000 0,9570,420 1,000 0,9680,446 1,000 0,9780,493 1,000 0,9891,000 1,000 1,000

92 59 151

98,9% 88% 94%

1 8 9

1,1% 12% 5,6%

93 67 160

100% 100% 100%

0,171<

0,171>=

RATIO

Total

- +

CHD

Total

Let cutoff=0,171

ROC Curve

1 - Seçicilik

1,0,9,8,7,6,5,4,3,2,10,0

Se

ns

itiv

ity

1,0

,9

,8

,7

,6

,5

,4

,3

,2

,1

0,0

1-Specificity

Cutoff=0.26

TPR=0.78

FPR=0.31

TNR=0.69

FNR=0.22

Area Under the Curve

Test Result Variable(s): ORAN

,778 ,036 ,000 ,708 ,849Area Std. Error

aAsymptotic

Sig.b

Lower Bound Upper Bound

Asymptotic 95% ConfidenceInterval

The test result variable(s): ORAN has at least one tie between thepositive actual state group and the negative actual state group. Statisticsmay be biased.

Under the nonparametric assumptiona.

Null hypothesis: true area = 0.5b.

If Patients with CHD and without CHD are normally distributed, we can easily find sensitivity and specificity from the area under these normal curves. Sensitivity and specificity are calculated for each different cotoff points

CHD+ CHD-

Cutoff=0,28

TP

FP

FN

TN

0,23 0,29

ZCHD+=(0.28-0.23)/0.048=1.04

TPR=0.5+0.3508=0.8508

FNR=1-TPR=0.1492

If we take cut off point=0.28, the characteristics of test are:

ZCHD-=(0.28-0,29)/0.066=-0.15

TNR=0.5+0.0596=0.5596

FPR=1-TNR=0.4404

Cutoff=0,28

0,23 0,29

CHD+ CHD-

Cutoff TPR FNR TNR FPR0,10 0,00 1,00 1,00 0,000,15 0,05 0,95 0,98 0,020,20 0,27 0,73 0,91 0,090,25 0,66 0,34 0,73 0,270,28 0,85 0,15 0,56 0,440,30 0,93 0,07 0,44 0,560,35 0,99 0,01 0,18 0,820,45 1,00 0,00 0,00 1,00

0,000,100,200,300,400,500,600,700,800,901,00

0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1

FPR

TP

R