ebm-diagnostic testing k. mae hla, md, mhs primary care faculty development fellowship november 13,...

EBM-Diagnostic Testing

K. Mae Hla, MD, MHS

Primary Care Faculty

Development Fellowship

November 13, 2010

Objectives

• Develop pre-test probabilities • Derive treatment thresholds• Appraise evidence about a diagnostic test

-validity, accuracy and applicability• Calculate the results of diagnostic tests

-sensitivity, specificity and likelihood ratios• Apply evidence to patient care decisions

The Diagnostic Process

• Working diagnosis- pretest probability• With each new finding/test we move

from the pre-test probability to a new post-test probability

• Clinicians estimate probability of disease using probabilistic, prognostic and pragmatic approaches

• Compare disease probabilities to two thresholds

Applying Diagnostic Tests

Example #1

8-year-old with fever, sore throat, swollen cervical glands and tonsillar exudates. No h/o cough.

You order a rapid strep test.

What’s your pretest probability of the patient having group A strep pharyngitis?

How much of a change would help you decide to treat, not treat or test further?

Treatment Thresholds

ZONE OF UNCERTAINTYNo Tx Tx

0% 100%

Probability of Strep Pharyngitis

5% 90%

Rapid Strep Test Results

Rapid Strep test result comes back negative

How does the rapid strep test result change the probability of the patient having or not having the disease?

A positive rapid strep test raises post test probability of strep pharyngitis to 85% in one study

A negative strep test decreases probability to 12 %

Pre-test Prob = 40%

LR+ =

LR- =

7.2

0.24

Treatment Thresholds

ZONE OF UNCERTAINTYNo Tx Tx

0% 12% 100%

Probability of strep when rapid strep test

is negative

X

5% 90%

Example #2

18-year-old female with ankle pain after a roller-blading accident. States unable to walk on her injured ankle. Exam demonstrates a slightly swollen ankle but no tenderness noted. Able to bear weight and take 4 full steps upon encouragement.

What is the probability of ankle fracture?

Do you need to order an ankle x-ray?

Ottowa Ankle rule

An ankle x-ray is only necessary if there is pain near the malleoli and any of the following findings are present:

inability to bear wt. both immediately and in the ED

bone tenderness at the posterior edge or tip of the malleolus

How accurate is the Ottowa ankle rule in ruling out ankle

fracture?

A prospective validation study in > 1000 pts presenting to the ED with ankle pain

Likelihood ratio + = 1.96

Likelihood ratio - = 0

Pre-test Prob = 10%

LR+ =

LR- =

1.96

0

Case 1• 75-year-old woman with a hemoglobin

of 10, MCV was 80 on routine checkup, a negative history and physical except osteoarthritis, and on no meds likely to suppress her marrow or cause a bleed

• Her probability of iron deficiency was 50%

• You want to avoid doing a bone marrow and order serum ferritin to diagnose iron deficiency anemia

Case 1• P: In an elderly symptomless woman with

mild anemia• I: how useful is serum ferritin • C:• O: in diagnosing iron deficiency anemia• T(ype of question): Diagnosis• T(ype of study): Prospective Cohort

*Diagnosis of Iron Deficiency Anemia in the Elderly (Guyatt, et al. Am J Med, 1990;(88):205-209

Three Main Questions

• Validity-Is this evidence about the accuracy of a diagnostic test valid?

• Results-Does this evidence show that this test can adequately distinguish patients who do and do not have the disorder?

• Applicability-How can I apply this valid, accurate diagnostic test to a specific patient?

Validity

• Measurement: was the gold standard measured independently?

• Representative: was the test evaluated in appropriate spectrum of patients?

• Ascertainment: was the reference test ascertained regardless of the diagnostic test?

• All patients in study should have both the diagnostic test in question (blood test, history, physical exam) and the gold/reference standard test (autopsy, bone marrow, biopsy, angiogram)

• Independent- test not part of gold standard, decision to perform gold standard should not depend on result of diagnostic test under study

• Blinding- reference test readers should be unaware of results to avoid bias if tests/gold standard have subjective component- x-rays, biopsy, slides

Validity- Measurement

Validity: Measurement

• Was there an independent blind comparison with a reference gold standard? • The gold standard was the bone marrow

aspirate results• All patients got the serum ferritin and

bone marrow done independently • Marrow aspirates and iron deficiency

status was determined by 2 hematologists unaware of the lab result

Validity: Representative

• Was the diagnostic test evaluated in an appropriate spectrum of patients?

• Examples: risk markers such as CEA were initially done in high risk patients

Validity: Representative

• Diagnostic uncertainty• Patients with mild as well as severe

symptoms• Patients with early as well as late

disease• Patients with other commonly

confused diagnoses

Study spectrum representative?

• Consecutive patients age 65 or older with anemia were recruited

• 36% of patients had iron deficiency anemia• 44% had anemia of chronic disease• 8% megaloblastic anemia• Patients with other commonly confused

disorders- different types of anemia and chronic medical conditions were included

Validity: Ascertainment

• Was the reference standard ascertained regardless of the diagnostic test result?

• Did all patients in the study both with and without iron deficiency anemia get the bone marrow done?

• Yes

• Patients with negative diagnostic test may not get the gold standard done if the latter is invasive

• How do we prove for sure that the ones with negative tests truly do not have the disease or vice versa?

• Other ways to establish reference test

Ascertainment-Continued

• In the Pioped study looking at the utility of V/Q scan in patients with suspected pulmonary embolism-all patients with negative V/Q scan did not get pulmonary angiogram

• Clinical followup in a year was the

additional reference standard to not miss patients with false negative VQ results

Results

• Does the test accurately distinguish between patients with and without the disorder?– Sensitivity and specificity– Likelihood ratios

DiseaseDisease

PresentPresent AbsentAbsent

TT

ee

ss

tt

PositivePositiveaa bb a+ba+b

NegativeNegativecc dd c+dc+d

a+ca+c b+db+d

DiseaseDisease

PresentPresent AbsentAbsent

TT

ee

ss

tt

PositivePositiveaa

TPTP

bb

FPFP

a+ba+b

NegativeNegativecc

FNFN

dd

TNTN

c+dc+d

a+ca+c b+db+d

DiseaseDisease

PresentPresent AbsentAbsent

TT

ee

ss

tt

PositivePositiveaa

TPTP

bb

FPFP

a+ba+b

NegativeNegativecc

FNFN

dd

TNTN

c+dc+d

a+ca+c

Sen=a/a+cSen=a/a+c

b+db+d

Sp=d/d+bSp=d/d+b

DiseaseDisease

PresentPresent AbsentAbsent

TT

ee

ss

tt

PositivePositiveaa

TPTP

bb

FPFP

a+ba+b

NegativeNegativecc

FNFN

dd

TNTN

c+dc+d

a+ca+c

Sen=a/a+cSen=a/a+c

Sp=d/d+bSp=d/d+b

b+db+d

““PID”PID”

““NIH”NIH”

Sensitive test-rules out the disease (SnNout)

• Test with high sensitivity (high TPR and very low false negative rate), negative test rules out the disease

• Examples: • loss of retinal vein pulsation in increased

intracranial pressure-the presence of pulsation (negative test) rules out IIP

• HIV antibody- negative test rules out HIV

Specific test – rules in the disease (SpPIN)

• Test with high specificity (high TNR, very low FPR)-positive test rules in the diagnosis

• Features of child with Down’s syndrome-very specific

• Presence of features (positive test) rules in the diagnosis

• Western blot confirmatory testing for HIV- high specificity: positive test rules in HIV disease

Likelihood RatioLikelihood Ratio

likelihood of the test result in patients with the diseaselikelihood of the same result in patients without diseaseLR =LR =

DiseaseDisease

PresentPresent AbsentAbsent

TT

ee

ss

tt

PositivePositiveaa

TPTP

bb

FPFP

a+ba+b

NegativeNegativecc

FNFN

dd

TNTN

c+dc+d

a+ca+cSen=a/a+cSen=a/a+c

b+db+dSp=d/d+bSp=d/d+b

(+)LR= (+)LR= + test result in pts with dz+ test result in pts with dz (-)LR= (-)LR= - test result in pts with dz- test result in pts with dz + test result in pts without dz - test result in pts without dz+ test result in pts without dz - test result in pts without dz

DiseaseDisease

PresentPresent AbsentAbsent

TT

ee

ss

tt

PositivePositiveaa

TPTP

bb

FPFP

a+ba+b

NegativeNegativecc

FNFN

dd

TNTN

c+dc+d

a+ca+cSen=a/a+cSen=a/a+c

b+db+dSp=d/d+bSp=d/d+b

(+)LR= (+)LR= + test result in pts with dz+ test result in pts with dz (-)LR= (-)LR= - test result in pts with dz- test result in pts with dz + test result in pts without dz - test result in pts without dz+ test result in pts without dz - test result in pts without dz

= Sn/1-Sp = 1-Sn/Sp= Sn/1-Sp = 1-Sn/Sp

Likelihood RatioLikelihood Ratio

probability of the test result in patients with the diseaseprobability of the same result in patients without diseaseLR =LR =

Pre-Test Probability

Pre-Test Odds

Post-Test Odds

Post-Test Probability

Odds = Probability/1-probability Probability = Odds/1 + Odds

What do all these numbers mean?!?

• L.R.s indicate by how much a given diagnostic test result will raise or lower the pre-test probability of the target disorder

• L.R. of 1 = post-test probability is same as pre-test probability

• L.R. > 1 increases the probability that the target disorder is present; the higher the L.R., the greater the increase

• L.R. < 1 decreases the probability of the target disorder; the smaller the L.R., the greater the decrease

Effects of different likelihood ratios

• >10 or <0.1 generate large and often conclusive changes from pre- to post-test probability

• 5-10 and 0.1-0.2 generate moderate shifts in pre- to post-test probability

• Depending on pre-test probability, change may or may not be large enough to influence Rx decision

Back to our patient

Our patient’s serum ferritin comes back at 40 mmol/L

How should we put all this together?

Iron Deficiency AnemiaIron Deficiency Anemia

PresentPresent AbsentAbsent

TT

ee

ss

tt

PositivePositive

<<4545

7070

aa

1515

bb

NegativeNegative

>45>45

cc

1515

dd

135135

a+ca+c b+db+d a+b+c+da+b+c+d

TotalsTotals 8585 150150 235235

Low ferritin (<45) in diagnosing Fe def anemia

Prevalence (study pre-test probability)

= 85/235= 36%

Sensitivity = True positive / all with disease

= a/a+c

= 70/85

= 82%

Specificity = True neg / all without disease

= d/b+d

= 135/150

= 90%

Low ferritin (<45) in diagnosing iron deficiency

anemia

L.R.+ = sensitivity/(1-specificity)

= 82%/10% = 8.2

L.R. - = (1-sens)/spec

= 18%/90% = 0.2

Simplifying Likelihood Simplifying Likelihood Ratio CalculationsRatio Calculations

Bone Marrow: Bone Marrow:

iron deficientiron deficient

Bone Marrow: Bone Marrow: normal ironnormal iron

Test Results:Test Results:

<< 45 45 7070 1515

>> 46 46 1515 135135

TotalsTotals 8585 150150

Calculating Likelihood Ratios at 45 cut point

Bone Marrow: Bone Marrow:

iron deficientiron deficient

Bone Marrow: Bone Marrow: normal ironnormal iron

Likelihood Likelihood RatiosRatios

Test Results:Test Results:

<< 45 45 7070

70/85=0.82470/85=0.824

1515

15/150=0.115/150=0.1

0.824/0.1=0.824/0.1=

8.248.24

>> 46 46 1515

15/85=0.17615/85=0.176

135135

135/150=0.9135/150=0.9

0.176/0.9=0.176/0.9=

0.1960.196

TotalsTotals 8585 150150

Pre-test Prob = 36%

LR+ =

LR- =

8.2

0.2

Applying the Test to the Patient

• Is the diagnostic test available, affordable, accurate and precise in our setting?

• Yes

Applicability (cont’d)

• Test needs to be available, affordable• Interpreted in competent, reproducible

fashion in clinical setting• Potential consequences should justify

the cost

Applicability (Cont’d)

• Are the study patients similar to our own?

• Are the results applicable to the patient in my practice?

• Will the patient be better off as a result of the test?

Applicability-study patients’ characteristics

• 235/259 patients had interpretable aspirates• Mean age 79.7, 46% men, 72% had no

medical diagnosis other than anemia• Early dementia 25• CHF 25• COPD 25• Rheumatoid arthritis 17• Osteoarthritis 14• Pneumonia 13

Can we generate a clinically sensible estimate of our

patient’s pre-test probability?

How can we estimate pre-test probability?

• Clinical experience• Regional or national prevalence statistics• Practice databases• Pretest probability observed in the study itself

• Studies of pre-test probabilities

Low = 3.6% (2-6)

Wells PS, Anderson DR, Rodger M, et al. Derivation of a simple clinical model to Wells PS, Anderson DR, Rodger M, et al. Derivation of a simple clinical model to categorize patients' probability of pulmonary embolism: increasing the model's categorize patients' probability of pulmonary embolism: increasing the model's utility with the SimpliRED D-dimer. Thromb Haemost 2000;83:416-420. utility with the SimpliRED D-dimer. Thromb Haemost 2000;83:416-420.

Risk of PE

Inter = 20%

(17-24)

High = 67%

(54-77)

Will the post-test probabilities affect our management and

help our patient?

• Could the test result move us across a test-treatment threshold?

• Would the patient be willing to undergo the test?

• Would it help the patient?

Pre-test Prob = 50%

LR+ =

LR- =

8.2

0.2

Treatment Thresholds

ZONE OF UNCERTAINTYNo Tx Tx

0% 100%

90% Probability of Fe def Anemia when

Ferritin is <45

10% 90%

x

Likelihood Ratios for 4 levels of Serum Ferritin

FerritinFerritin Fe def #Fe def # Not Fe defNot Fe def L.R.L.R.

<<1818 4747 22 41.4741.47

>18>18<<4545 2323 1313 3.123.12

>45>45<<100100 7 7 2727 0.460.46

>100>100 88 108108 0.130.13

Total Total 8585 150150

Calculating Likelihood Ratios

Bone Marrow: Bone Marrow:

iron deficientiron deficient

Bone Marrow: Bone Marrow: normal iron normal iron

Likelihood Likelihood RatiosRatios

Test Results:Test Results:

<< 18 18 4747

47/85=0.55347/85=0.553

22

2/150=0.0132/150=0.013

0.553/0.013=0.553/0.013=

42.542.5

19-4519-45 2323

23/85=0.27123/85=0.271

1313

13/150=0.08713/150=0.087

0.271/0.087=0.271/0.087=

3.113.11

46-10046-100 77

7/85=0.0827/85=0.082

2727

27/150=0.1827/150=0.18

0.082/0.18=0.082/0.18=

0.4560.456

>100>100 88

8/85=0.0948/85=0.094

108108

108/150=0.72108/150=0.72

0.094/0.72=0.094/0.72=

0.1310.131

TotalsTotals 8585 150150

Clinical Scenario 52 y.o. male admitted to Orthopedics 3 days 52 y.o. male admitted to Orthopedics 3 days

ago for a R femur fracture after falling from a ago for a R femur fracture after falling from a ladderladder

Underwent ORIF 2 days agoUnderwent ORIF 2 days ago Last night developed SOBLast night developed SOB No CP or coughNo CP or cough PE: Afeb 115/70 HR 110 RR 18 95% on 4LPE: Afeb 115/70 HR 110 RR 18 95% on 4L

– Otherwise unremarkable (Lungs clear, No Otherwise unremarkable (Lungs clear, No elevated JVP or RV heave, no lower ext swelling)elevated JVP or RV heave, no lower ext swelling)

Labs: EKG sinus tach, CXR clearLabs: EKG sinus tach, CXR clear D-dimer 0.5 mcg/mL (nl <0.6)D-dimer 0.5 mcg/mL (nl <0.6)

PICO• P: In a patient with acute onset of SOB,

hypoxia and sinus tachycardia after a major orthopedic surgery for femur fracture

• I: Does a negative D-dimer result• C:• O: Rule out pulmonary embolism• T(ype of question): Diagnosis• T(ype of study): Prospective Cohort

*De Monye W et al: The performance of two rapid d-dimer assays in 287 patients with clinically suspected pulmonary embolism. Thrombosis Res 2002;(107):283-286.

Low = 3.6% (2-6)

Wells PS, Anderson DR, Rodger M, et al. Derivation of a simple clinical model to Wells PS, Anderson DR, Rodger M, et al. Derivation of a simple clinical model to categorize patients' probability of pulmonary embolism: increasing the model's categorize patients' probability of pulmonary embolism: increasing the model's utility with the SimpliRED D-dimer. Thromb Haemost 2000;83:416-420. utility with the SimpliRED D-dimer. Thromb Haemost 2000;83:416-420.

Risk of PE

Inter = 20%

(17-24)

High = 67%

(54-77)

Treatment Thresholds

ZONE OF UNCERTAINTYNo Tx Tx

0% 20% 100%

Probability of Pulm Embolism

X

5% 90%

Sensitivity and SpecificitySensitivity and Specificity disease (+) disease (-)

Test (+) 74 77

Test (-) 16 120

Sensitivity (“Positive in disease”) = true pos / all disease = 74 / 90 = 82.2%

Specificity (“Negative in health”) = true neg/all disease free = 120 / 197 = 60.3%

Likelihood Ratios of D-Dimer TestLikelihood Ratios of D-Dimer Test

disease (+) disease (-)

Test (+) 74 77

Test (-) 16 120

Likelihood ratio (+) = (74 / 90) / (77/197) = 2.1

Likelihood ratio (-) = (16/90) / (120/197) = 0.29

LR+ =

LR- =

2.1

0.29

Pre-test Prob = 20%

Treatment Thresholds

ZONE OF UNCERTAINTYNo Tx Tx

0% 5% 100%

Probability of Pulm Embolism if

D-Dimer is negative

X

5% 90%

LR (using lower cut off 95% LR (using lower cut off 95% sensitive D-Dimer value)sensitive D-Dimer value)

disease (+) disease (-)

Test (+) 86 149

Test (-) 4 48

Likelihood ratio + test = (86/90) / (149/197) = 1.3

Likelihood ratio - test = (4/90) / (48/197) = 0.18

probability of the test result in patients with the diseaseprobability of the same result in patients without diseaseLR =LR =

LR+ =

LR- =

1.3

0.18

Pre-test Prob = 20%

Treatment Thresholds

ZONE OF UNCERTAINTYNo Tx Tx

0% 100%

3% Probability of PE if D-Dimer is

negative

3% 90%

x

Summary

• Develop pre-test probabilities • Steps in appraising a diagnostic test• Calculate and interpret sensitivity,

specificity and likelihood ratios• Evaluate how pre-test probability and

likelihood ratio results affect post-test probability of disease to influence further testing or treatment decisions