Core Conference Series: Numerical Concepts in Clinical Study Interpretation Type I and Type II error in Clinical Trials

“Truth” Clinical Benefit No Benefit

Trial Result

“Positive” “Correct” Type I error

“Negative” Type II error “Correct” The “p-value” = probability of type I error—the probability of finding benefit where there is no benefit. “α” The power = 1 - probability of type II error—the probability of finding no benefit when there is benefit. “1-β” The sample size a function of the study design, effect size, and acceptable type I and type II error. Sensitivity, Specificity, PPV, NPV, and Likelihood Ratios in Diagnostic Testing

“Truth” Disease + Disease -

Test Result

“Positive” True Positive

False Positive

“Negative” False Negative

True Negative

Sensitivity, specificity, and likelihood ratios are inherent to the test; PPV and NPV depend on specificity, sensitivity, and disease prevalence Relative Risk Difference vs. Absolute Risk Difference

N Alive

N Dead Mortality

Absolute Risk

Reduction

Relative Risk Reduction

Number Needed to

Treat Study 1 Treatment 80 20 0.20 0.05 1-(0.20/0.25)=0.20 1/ARR=20

N=200 Control 75 25 0.25

Study 2 Treatment 980 20 0.02 0.01 1-(0.02/0.03)=0.33 1/ARR=100

N=2000 Control 975 30 0.03 Greater relative risk reduction ≠ “better” treatment!

𝑆𝑒𝑛𝑠 =𝑇𝑃

𝑇𝑃 + 𝐹𝑁 𝑆𝑝𝑒𝑐 =𝑇𝑁

𝑇𝑁 + 𝐹𝑃

𝑃𝑃𝑉 =𝑇𝑃

𝑇𝑃 + 𝐹𝑃 𝑁𝑃𝑉 =𝑇𝑁

𝑇𝑁 + 𝐹𝑁

𝐿𝑅+=𝑠𝑒𝑛𝑠

1− 𝑠𝑝𝑒𝑐 𝐿𝑅−=1− 𝑠𝑒𝑛𝑠𝑠𝑝𝑒𝑐

Diagnostic Testing Example: DDimer for VTE—low risk population 1000 pts VTE

Yes No DDimer Positive 56 536 592

Negative 4 404 408 60 940 1000

Example: DDimer for VTE—high risk population 1000 pts VTE

Yes No

DDimer Positive 263 410 673 Negative 17 310 327

280 720 2000

Relative vs. Absolute Risk

N Total N Dead Mortality Absolute

Risk Reduction

Relative Risk Reduction

Number Needed to

Treat Prostate Cancer

Screening Screen 72,952 214 0.0030

Control 89,435 326 0.0036

Low VT for ARDS Low VT 432 134 0.31

Control 429 171 0.40

𝑆𝑒𝑛𝑠 =𝑇𝑃

𝑇𝑃 + 𝐹𝑁 = 𝑆𝑝𝑒𝑐 =𝑇𝑁

𝑇𝑁 + 𝐹𝑃 =

𝑃𝑃𝑉 =𝑇𝑃

𝑇𝑃 + 𝐹𝑃 = 𝑁𝑃𝑉 =𝑇𝑁

𝑇𝑁 + 𝐹𝑁 =

𝐿𝑅+=𝑠𝑒𝑛𝑠

1− 𝑠𝑝𝑒𝑐 = 𝐿𝑅−=1− 𝑠𝑒𝑛𝑠𝑠𝑝𝑒𝑐 =

𝑆𝑒𝑛𝑠 =𝑇𝑃

𝑇𝑃 + 𝐹𝑁 = 𝑆𝑝𝑒𝑐 =𝑇𝑁

𝑇𝑁 + 𝐹𝑃 =

𝑃𝑃𝑉 =𝑇𝑃

𝑇𝑃 + 𝐹𝑃 = 𝑁𝑃𝑉 =𝑇𝑁

𝑇𝑁 + 𝐹𝑁 =

𝐿𝑅+=𝑠𝑒𝑛𝑠

1− 𝑠𝑝𝑒𝑐 = 𝐿𝑅−=1− 𝑠𝑒𝑛𝑠𝑠𝑝𝑒𝑐 =

Diagnostic Testing Example: DDimer for VTE—low risk population 1000 pts VTE

Yes No DDimer Positive 56 536 592

Negative 4 404 408 60 940 1000

Example: DDimer for VTE—high risk population 1000 pts VTE

Yes No

DDimer Positive 263 410 673 Negative 17 310 327

280 720 2000

Relative vs. Absolute Risk

N Total N Dead Mortality Absolute

Risk Reduction

Relative Risk Reduction

Number Needed to

Treat Prostate Cancer

Screening Screen 72,952 214 0.0030 0.0006 1-(0.0030/0.0036)

=0.17 1/ARR=

1667

Control 89,435 326 0.0036

Low VT for ARDS Low VT 432 134 0.31 0.09 1-(0.31/0.40)=0.23 1/ARR=11

Control 429 171 0.40

𝑆𝑒𝑛𝑠 =𝑇𝑃

𝑇𝑃 + 𝐹𝑁 =5660 = 0.93 𝑆𝑝𝑒𝑐 =

𝑇𝑁𝑇𝑁 + 𝐹𝑃 =

404940 = 0.43

𝑃𝑃𝑉 =𝑇𝑃

𝑇𝑃 + 𝐹𝑃 =56592 = 0.09 𝑁𝑃𝑉 =

𝑇𝑁𝑇𝑁 + 𝐹𝑁 =

404408 = 0.99

𝐿𝑅+=𝑠𝑒𝑛𝑠

1− 𝑠𝑝𝑒𝑐 =0.93

1− 0.43 = 1.63 𝐿𝑅−=1− 𝑠𝑒𝑛𝑠𝑠𝑝𝑒𝑐 =

1− 0.930.43 = 0.16

𝑆𝑒𝑛𝑠 =𝑇𝑃

𝑇𝑃 + 𝐹𝑁 =263280 = ~0.93 𝑆𝑝𝑒𝑐 =

𝑇𝑁𝑇𝑁 + 𝐹𝑃 =

310720 = 0.43

𝑃𝑃𝑉 =𝑇𝑃

𝑇𝑃 + 𝐹𝑃 =263673 = 0.39 𝑁𝑃𝑉 =

𝑇𝑁𝑇𝑁 + 𝐹𝑁 =

310327 = 0.95

𝐿𝑅+=𝑠𝑒𝑛𝑠

1− 𝑠𝑝𝑒𝑐 =0.93

1− 0.43 = 1.63 𝐿𝑅−=1− 𝑠𝑒𝑛𝑠𝑠𝑝𝑒𝑐 =

1− 0.930.43 = 0.16