Why we don’t like

protocol violations

Yuko Y. Palesch, PhDMedical University of South Carolina

Humans are the worst experimental units.

Because…

They are very heterogeneous

Because… They are very heterogeneous

They need to be respected and treated fairly and with dignity

Because… They are very heterogeneous

They need to be respected and treated fairly and with dignity

They have a mind of their own

In addition, those of us who design and conduct the trial

are also humans

To Err = Human

Some Errors in ProTECT Eligibility / Randomization

TBI mimic (e.g., alcohol intoxication) randomized (??%)

Study drug not given within 4 hours (16%)

Study drug administration Wrong drug given / crossovers (<1%) Infusion interruptions (12%) Taper errors (30%)

Missing data Lost to follow-up (<1%) Outcomes assessed beyond the time window (13%)

Data quality and timeliness Incorrect and/or late data entries

But why do we care about these errors?

the ITT Analysis

ITT stands for… (choose one)

1. Incentive to Treat

2. Insure to Treat

3. Intent to Treat

4. International Treatment Trial

5. Incredibly Tedious Trial

6. Both 3 and 5

Scenario 1 (misdiagnosis)A patient, upon arrival in the ER was mis-diagnosed to have a TBI and was randomized and treated with the study drug. Shortly thereafter, it was discovered that she was just intoxicated, and terminated from the study. Should the subject be included in the ITT analysis of the primary outcome?

A. YesB. No

A. B.

0%0%

Scenario 2 (time delay)An eligible patient was randomized at 3.5 hours from injury, but the study drug was initiated at 5 hours. He died from his injury within two hours of study drug initiation. Should the subject be included in the ITT analysis of the primary outcome?

A. YesB. No

A. B.

0%0%

Scenario 3 (crossover)An eligible patient was randomized but, because of the study drug kit accounting error, the “wrong” study drug was administered. Nevertheless, the subject is followed through 6 months per the protocol. Should the subject be included in the ITT analysis of the primary outcome?

A. YesB. No

A. B.

0%0%

Scenario 4 (drug admin error) An eligible patient was randomized but, because of a variety of reasons, including infusion interruptions, only 1/2 of the study drug dose was administered. The subject was lost-to-follow- up at 3 months. Should the subject be included in the ITT analysis of the primary outcome?

A. YesB. No

A. B.

0%0%

Scenario 5 (non-EFIC Trial)An eligible patient is randomized and treated with study drug, but it is discovered upon site monitoring 2 months later that a signed and dated Informed Consent was not obtained prior to randomization.Should the subject be included in the ITT analysis of the primary outcome?

A. YesB. No

A. B.

0%0%

Scenario 6 (consent withdrawal)An eligible patient is randomized at 2 hours from injury and IC obtained from the LAR within the hour. But the family asks for DNR before the study drug is administered and withdraws consent. Should the subject be included in the ITT analysis of the primary outcome?

A. YesB. No

A. B.

0%0%

Definition of ITT Analysis

Analysis that includes all randomized patients in the groups to which they were randomly assigned, regardless of their adherence with the entry criteria, regardless of the treatment they actually received, and regardless of subsequent withdrawal from treatment or deviation from the protocol.

Fisher et al. In: Statistical issues in drug research and development. New York:Marcel Dekker, pp. 331-350, 1990.

Effect of errors on the statistical test and

interpretation of results

Primary outcome is dichotomous - good vs bad:

* Using alpha (two-sided) = 0.05 and power = 85%; accounting for two planned interim analysis.

GroupHypothesized % good outcome

N*

PRG 60% 462

PLC 50% 462

Tx effect ∆=10%

ProTECT Trial 1º Hypothesis

A Hypothetical Scenario

Subject Types 

PRG Group PLC Group

A B C D E F

How many?

% Good outcome among them

Col. A x Col. B

How many?

% Good outcome among them

Col. D x Col. B

Misdiagnose alcohol intoxication 1% 90% 0.009 1% 90% 0.009

Time to study drug initiation >4 hrs 6% 50% 0.03 6% 50% 0.03

Crossovers 1% 50% 0.005 1% 60% 0.006

Drug infusion errors 10% 50% 0.05 10% 50% 0.05

Remainder (i.e., protocol compliers)

82% 60% 0.492 82% 50% 0.41

Observed good outcome 

  

58.6%  

50.5%

Observed difference8.1% (instead of 10%)

Hence, dilution of treatment effect

Assume true PRG and PLC % good outcome are 60% and 50%, respectively

With N=924, we only have 67% power to detect a difference of 8.1%, even if the true PRG effect is 10% better than PLC.

OR we’d need an additional 482 subjects to keep the original 85% power (or a total of N=1,404)

(NOTE: Max total N planned in protocol = 1,140 to account for some dilution effect)

Therefore, because of the errors, we may fail to achieve statistical significance, and the ProTECT Trial will be deemed negative / neutral study, even if PRG is an effective treatment.

Dilution of Tx Effect

Scenario 4: Lost to follow-up

Scenario 6: Withdrawal of consent

Missing data

Need to impute (make up) data for ITT

Will also contribute to the dilution of the tx effect

May cause biased results, especially if Missing NOT At Random (MNAR)

Bias Effect

“…no statistical analyses can ever adequately adjust for missing data, despite many techniques that attempt to do so.”

DeMets DL. Journal of Internal Medicine 2004; 255:529–537

Bias Effect (cont’d)

So why do the ITT analysis? Why not do Per Protocol

analysis?

Problem with PP Analysis Assumes that the non-compliers do not

differ in their state of health from those who comply.

And that the decision to comply is not itself influenced by treatment.

If assumptions are incorrect, we have subset selection bias which causes increase in false positive errors.

In practice, EXTREMELY difficult to determine who belongs in PP analysis.

Anturane Infarction Trial Example

Anturane Placebo p-valuedeaths/total (%) deaths/total (%)

Randomized 74/813 (9.1%) 89/816 (10.9%) 0.20Eligible 64/775 (8.3%) 85/783 (10.9%) 0.07Ineligible 10/38 (26.3%) 4/33 (12.1%) 0.12

p-value for eligible vs ineligible

0.0001 0.92

Temple R, Pledger GW. The FDA's critique of the Anturane Reinfarction Trial. New Engl J Med 1980; 303: 1488–92.

Compared sulfinpyrazone vs placebo in post-heart attack patients. Original study results only used those deemed “eligible” post-hoc with p-value=0.07.

Results of re-analysis by the FDA:

Anturane Infarction Trial Example

Anturane Placebo p-valuedeaths/total (%) deaths/total (%)

Randomized 74/813 (9.1%) 89/816 (10.9%) 0.20Eligible 64/775 (8.3%) 85/783 (10.9%) 0.07Ineligible 10/38 (26.3%) 4/33 (12.1%) 0.12

p-value for eligible vs ineligible

0.0001 0.92

Temple R, Pledger GW. The FDA's critique of the Anturane Reinfarction Trial. New Engl J Med 1980; 303: 1488–92.

Compared sulfinpyrazone vs placebo in post-heart attack patients. Original study results only used those deemed “eligible” post-hoc with p-value=0.07.

Results of re-analysis by the FDA:

Conclusion

To ensure correct statistical inference from the study:

Minimize randomization and implementation errors.

Avoid missing data.

Enter data in a timely manner so that the interim analysis can be performed with complete and accurate data.