Design & Interpretation of Randomized Trials:A Clinician’s PerspectiveFrancis KL ChanDepartment of Medicine & TherapeuticsCUHK

Common problems of RCTs

Originality

Hypothesis

Allocation concealment & randomization

Evaluation of baseline data

“Intention-to-treat” analysis

Subgroup analysis

Is the study original?

Ground breaking research?

Does this work add to the literature in any way?

Bigger, longer?

More rigorous methodology?

Results add to a meta-analysis of previous studies?

Different population (age, sex, ethnic groups)?

Hypothesis & End pointMany RCTs did not explicitly state their study

hypotheses

“The aim of this study was to compare the efficacy of a new treatment with the standard treatment…”

Hypothesis 1:

Treatment A is superior to the standard treatment

Hypothesis 2:

Treatment A is equivalent to the standard treatment

Hypothesis?

Sample size estimation

None!

Failure to detect a difference

=Equivalence?

Superiority Trial

The new treatment (µN) is superior to the

standard treatment (µS) if the difference

exceeds by a clinically important amount ().

Test hypothesis (H): µN - µS>

Equivalence trial

The new treatment is equivalent to the

standard treatment if the maximal allowable

difference does not exceed by a clinically

important amount.

Equivalence trial

- 0 +

Equivalent

Difference

Not equivalent

Favors new treatment

Not equivalent

Favors standard

treatment

New agent is not inferior to the

standard

Assume non-inferiority if the lower limit of 95% CI is

less than –5%,N=904 per group!

Allocation concealment & randomization

Concealment of allocation

(investigators and patients not knowing the assigned

treatment before randomization)

Was treatment assigned by an independent staff?

What was the method of allocation concealment?

contact with central office

blinded packages

sealed (opaque) envelopes

Allocation concealment

NEJM, JAMA, Lancet, BMJ (N=50, Jul - Sep 97)

44%

56%

YesNo

Comparison of baseline data

Chan et al. Lancet 1997

Does P>0.05 indicate

comparability of treatment groups?

Baseline data

Azathioprine Placebo

Mean age 54.7 54.9

Serum bilirubin (mol/L) 37.2 30.9

Stage I disease % 14 12

Stage II disease % 44 43

Stage III disease % 15 15

Stage IV disease % 27 30

Christensen et al. Gastro 1985

Effect of azathioprine on the survival of patients with primary biliary cirrhosis

Baseline data

Azathioprine Placebo

Mean age 54.7 54.9

Serum bilirubin (mol/L) 37.2 30.9

Stage I disease % 14 12

Stage II disease % 44 43

Stage III disease % 15 15

Stage IV disease % 27 30

Christensen et al. Gastro 1985

Effect of azathioprine on the survival of patients with primary biliary cirrhosis

UnadjustedP=0.10

Adjusted for bilirubin

P=0.01

P=0.04

P=0.02

Columbus Investigators. NEJM 1997

Significant imbalance may not affect outcome

Comparison of baseline data

Non-significant imbalance may affect outcome

Significance tests for baseline differences are inappropriate.

Significance tests for baseline differences

Chan et al. Lancet 1997

INAPPROPRIA

TE

Significant imbalance may not affect outcome

Comparison of baseline data

Non-significant imbalance may affect outcome

Significance tests for baseline differences are inappropriate.

Table of baseline data should focus on factors affecting outcome.

45 baseline factors!

Significant imbalance may not affect outcome

Comparison of baseline data

Non-significant imbalance may affect outcome

Significance tests for baseline differences are inappropriate.

Table of baseline data should focus on factors affecting outcome.

Analysis adjusted for baseline factors that are known to strongly influence the outcome (Covariate-adjusted analysis).

Analysis of covariance for a quantitative outcome

Logistic regression for a binary response

Cox’s-proportional hazard model for time-to-event data

“Intention-To-Treat” Analysis

“…results were analyzed according to the ITT principle.”

Question:

How were missing outcomes/ protocol violators

handled in the so called “ITT” analysis?

“Intention-To-Treat” Analysis

Endpt

Savage et al. NEJM 1997

Minimize missing response on primary outcome

Recommendations for ITT Analysis

Follow up subjects who withdraw early

Investigate & report the effect of missing response

Report all deviations and missing response

Subgroup AnalysisRandomised trial of home-based psychosocial nursing intervention for patients recovering from myocardial infarction. Frasure-Smith et al. Lancet 1997

“…The poor overall outcome for women, and the possible harmful impact of the intervention on women, underlie the need for…”

Effect of antenatal dexamethasone administration on the prevention of respiratory distress syndrome.

Am J Obstet Gynecol 1981;141:276-87.

Subgroup Analysis

Steroid Placebo P

Pre-ecclampsia 21.2%

(7/33)

27.3%

(9/33)

0.57

No pre-ecclampsia 7.9%

(21/267)

14.1%

(37/262)

0.021

Effect of antenatal dexamethasone administration on the prevention of respiratory distress syndrome.

Am J Obstet Gynecol 1981;141:276-87.

Subgroup Analysis

Steroid Placebo P

Pre-ecclampsia 21.2%

(7/33)

27.3%

(9/33)

0.57

No pre-ecclampsia 7.9%

(21/267)

14.1%

(37/262)

0.021

Difference 6.1%

Effect of antenatal dexamethasone administration on the prevention of respiratory distress syndrome.

Am J Obstet Gynecol 1981;141:276-87.

Subgroup Analysis

Steroid Placebo P

Pre-ecclampsia 21.2%

(7/33)

27.3%

(9/33)

0.57

No pre-ecclampsia 7.9%

(21/267)

14.1%

(37/262)

0.021

Difference 6.1%

Difference 6.2%

P value depends on effect size & SE

Evaluation of Subgroup Analysis

Tests of interaction (assess whether a treatment

effect differs between subgroups) rather than

subgroup P values

Diff in Subgroup A – Diff in Subgroup B

SE of the above Diff Z =

Trial of vitamin D supplements in pregnancy to prevent infant hypocalcemia. BMJ 1980;281:11-4.

Infant plasma calcium (mg/100ml)

Mean SEM P

Vit D 9.20 0.085 Bottle-fed

Placebo 8.78 0.076

0.0006

Vit D 9.79 0.146 Breast-fed Placebo 9.64 0.125

0.4

Interaction TestDifference = 0.42

Difference = 0.15

0.42 – 0.15 = 0.27

SE of this Diff = 0.22

Z = Diff / SE = 1.23

P = 0.2No evidence th

at the effe

ct of V

it D is

different

between bottle

-fed and breast-fe

d infants

General points regarding subgroup analysis

Emphasis should remain on overall comparison

More convincing if confined to a limited number of pre-specified subgroup hypothesis

Rely on interaction tests, not P values

View subgroup findings as exploratory (to be confirmed in subsequent trials)