The potential role of mixed treatment comparisons

Deborah Caldwell

Tony Ades MRC HSRC

University of Bristol

Outline of presentation

• Indirect comparisons and mixed treatment comparisons (MTC).

• Potential concerns regarding use of indirect comparisons/ MTC.

• Hypothetical ‘simulation’ example of MTC evidence structure.

• MTC of NICE appraisal for thrombolysis• Addressing potential concerns • Should MTC be routine and future areas of

research.

Background

• For any given condition there are an array of possible interventions/ treatments.

• Treatment recommendations & decisions should be evidence based.

• Principle sources are systematic reviews of randomised controlled trials.

• Systematic reviews focus on pairwise, direct comparisons of treatments.

Indirect comparisons

• In absence of trials comparing treatments A versus B, an indirect estimate of odds ratio dAB is obtained from RCTs comparing A vs C and B vs

C: dAB = dAC – dBC

A B C

Indirect comparisons

• In absence of trials comparing treatments A versus B, an indirect estimate of odds ratio dAB is obtained from RCTs comparing A vs C and B vs

C: dAB = dAC – dBC

A B C

Indirect comparisons

• In absence of trials comparing treatments A versus B, an indirect estimate of odds ratio dAB is obtained from RCTs comparing A vs C and B vs

C: dAB = dAC – dBC

A B C

Mixed treatment comparisons

• Where there are 3 or more treatments, compared using direct and indirect evidence from several RCTs = mixed (multiple) treatment comparisons (MTC).

• MTC evidence structures are pervasive in Health Technology Assessments (HTA) – decisions between >5 treatments are commonplace.

• A unified, coherent analysis of multiple treatments can only be achieved by including the entire evidence structure of relevant RCTs.

Potential concerns about MTC

• Indirect comparisons produce relatively imprecise estimates of treatment effect

• They are not randomised comparisons

• Suffer the biases of observational studies (level 3 of EBM evidence hierarchies?)

• Direct and indirect evidence should be considered separately.

• Direct evidence should take precedence.

Contradictions in the ‘received wisdom’

• Why is lower level evidence used when level one is unavailable but it is irrelevant when it isn’t?

• What do we do when direct evidence is inconclusive but in combination with indirect is conclusive?

• If 5 treatments are all compared with each other does it make sense to separate the 10 direct pairwise comparisons from the 70 indirect?

Hypothetical evidence structure

dAB

dAC

dAD

dAE

dAF

A B C D E F

dBC = dAC - dAB

dBD = dAD - dAB

dBF = dAF - dAB

dBE = dAE - dAB

dCD = dAD - dAC

dCE = dAE - dAC

dCF = dAF - dAC

Objectives

• ‘Simulation’ exercise to explore benefit of increasing levels of complexity in MTC evidence structures.– To examine additional benefit of including

evidence routinely excluded from systematic reviews.

– To what extent different MTC evidence structures give increasing levels of precision.

– Address some of the concerns outlined.

Method

• Contrast estimates of posterior precision of log odds ratios (LOR) from

i. A standard pairwise meta-analysis

ii. Use of mixed treatment comparison analysis

• Compare estimates of posterior precisioni. LOR of treatment A vs. treatment B

ii. ‘Average’ precision - across all 15 possible treatment comparisons.

• Assumptions:• Equal amounts of information on each treatment

comparison.

Simulation results: precision of dAB

Pairwise MTC IncreaseAB, AC, AD, AE, AF 1 1.01 0.01Plus BC 1 1.51 0.51Plus BC, BD 1 2.00 1.00Plus BC, BD, CD 1 2.02 1.02Plus BC, BD, BE 1 2.50 1.50Plus BC, BD, BE, CD 1 2.52 1.52Plus BC, BD, BE, CD, CE 1 2.52 1.52Plus BC, BD, BE, BF 1 3.00 2.00Plus BC, BD, BE, BF, CD 1 2.94 1.94Plus BC, BD, BE, BF, CD, CE 1 3.02 2.02Plus BC, BD, BE, BF, CD, CE, CF, 1 3.02 2.02All 15 pairwise comparisons 1 2.98 1.98

Data ensemble, 1 trial each comparing:Precision dAB

• Precision of pairwise dAB = 1

• Precision of MTC dAB = 1.01

Hypothetical evidence structure

dAB

dAC

dAD

dAE

dAF

A B C D E F

Simulation results: precision of dAB

Pairwise MTC IncreaseAB, AC, AD, AE, AF 1 1.01 0.01Plus BC 1 1.51 0.51Plus BC, BD 1 2.00 1.00Plus BC, BD, CD 1 2.02 1.02Plus BC, BD, BE 1 2.50 1.50Plus BC, BD, BE, CD 1 2.52 1.52Plus BC, BD, BE, CD, CE 1 2.52 1.52Plus BC, BD, BE, BF 1 3.00 2.00Plus BC, BD, BE, BF, CD 1 2.94 1.94Plus BC, BD, BE, BF, CD, CE 1 3.02 2.02Plus BC, BD, BE, BF, CD, CE, CF, 1 3.02 2.02All 15 pairwise comparisons 1 2.98 1.98

Data ensemble, 1 trial each comparing:Precision dAB

• Additional data on a single indirect comparison increases precision by 0.51

5.1

)()precision()(Precision

5.0)(precision

11)Var()Var()(Var

IAB

DAB

MTCAB

IAB

BCACIAB

BCACIAB

dprecisiondd

d

ddd

ddd

Simulation results: precision of dAB

Pairwise MTC IncreaseAB, AC, AD, AE, AF 1 1.01 0.01Plus BC 1 1.51 0.51Plus BC, BD 1 2.00 1.00Plus BC, BD, CD 1 2.02 1.02Plus BC, BD, BE 1 2.50 1.50Plus BC, BD, BE, CD 1 2.52 1.52Plus BC, BD, BE, CD, CE 1 2.52 1.52Plus BC, BD, BE, BF 1 3.00 2.00Plus BC, BD, BE, BF, CD 1 2.94 1.94Plus BC, BD, BE, BF, CD, CE 1 3.02 2.02Plus BC, BD, BE, BF, CD, CE, CF, 1 3.02 2.02All 15 pairwise comparisons 1 2.98 1.98

Data ensemble, 1 trial each comparing:Precision dAB

• Each additional indirect treatment comparison increases precision in dAB by 0.5

Simulation results: precision of dAB

Pairwise MTC IncreaseAB, AC, AD, AE, AF 1 1.01 0.01Plus BC 1 1.51 0.51Plus BC, BD 1 2.00 1.00Plus BC, BD, CD 1 2.02 1.02Plus BC, BD, BE 1 2.50 1.50Plus BC, BD, BE, CD 1 2.52 1.52Plus BC, BD, BE, CD, CE 1 2.52 1.52Plus BC, BD, BE, BF 1 3.00 2.00Plus BC, BD, BE, BF, CD 1 2.94 1.94Plus BC, BD, BE, BF, CD, CE 1 3.02 2.02Plus BC, BD, BE, BF, CD, CE, CF, 1 3.02 2.02All 15 pairwise comparisons 1 2.98 1.98

Data ensemble, 1 trial each comparing:Precision dAB

• Is there value in ‘linking’ indirect comparisons? • ‘Linking’ comparison is treatment C vs D

Simulation results: precision of dAB

Pairwise MTC IncreaseAB, AC, AD, AE, AF 1 1.01 0.01Plus BC 1 1.51 0.51Plus BC, BD 1 2.00 1.00Plus BC, BD, CD 1 2.02 1.02Plus BC, BD, BE 1 2.50 1.50Plus BC, BD, BE, CD 1 2.52 1.52Plus BC, BD, BE, CD, CE 1 2.52 1.52Plus BC, BD, BE, BF 1 3.00 2.00Plus BC, BD, BE, BF, CD 1 2.94 1.94Plus BC, BD, BE, BF, CD, CE 1 3.02 2.02Plus BC, BD, BE, BF, CD, CE, CF, 1 3.02 2.02All 15 pairwise comparisons 1 2.98 1.98

Data ensemble, 1 trial each comparing:Precision dAB

• Adding ‘linking’ comparisons doesn’t increase precision of dAB estimate.

• Property of this particular evidence structure

Summary of simulation results: precision of dAB

• If all you believe is ‘direct’ data – precision dAB stays = 1

• Mixed Treatment Comparison analysis– Adding data on a single indirect comparison

increases precision by 0.51– Adding multiple indirect comparisons further

increases precision

– Equivalent to 2 extra trials on dAB comparison

Simulation results: ‘average’ precision

Simulation results: ‘average’ precision

• 5 pieces of data/ 15 possible treatment comparisons.

Pairwise MTC Equivalent # trialsAB, AC, AD, AE, AF 0.33 0.67 10Plus BC 0.40 0.84 13Plus BC, BD 0.47 1.05 16Plus BC, BD, CD 0.53 1.24 19Plus BC, BD, BE 0.53 1.27 19Plus BC, BD, BE, CD 0.60 1.49 22Plus BC, BD, BE, CD, CE 0.67 1.71 26Plus BC, BD, BE, BF 0.60 1.51 23Plus BC, BD, BE, BF, CD 0.67 1.74 26Plus BC, BD, BE, BF, CD, CE 0.73 2.00 30Plus BC, BD, BE, BF, CD, CE, CF, 0.80 2.26 34All 15 pairwise comparisons 1.00 3.00 45

Average precision over 15 pairwise comparisonsData ensemble, 1 trial each comparing:

Simulation results: ‘average’ precision

• Maximum ‘average’ pairwise precision = 1.• 15 pieces of data/ 15 possible treatment comparisons

Pairwise MTC Equivalent # trialsAB, AC, AD, AE, AF 0.33 0.67 10Plus BC 0.40 0.84 13Plus BC, BD 0.47 1.05 16Plus BC, BD, CD 0.53 1.24 19Plus BC, BD, BE 0.53 1.27 19Plus BC, BD, BE, CD 0.60 1.49 22Plus BC, BD, BE, CD, CE 0.67 1.71 26Plus BC, BD, BE, BF 0.60 1.51 23Plus BC, BD, BE, BF, CD 0.67 1.74 26Plus BC, BD, BE, BF, CD, CE 0.73 2.00 30Plus BC, BD, BE, BF, CD, CE, CF, 0.80 2.26 34All 15 pairwise comparisons 1.00 3.00 45

Average precision over 15 pairwise comparisonsData ensemble, 1 trial each comparing:

Simulation results: ‘average’ precision

• MTC allows us to say something about all 15 pairwise comparisons.

Pairwise MTC Equivalent # trialsAB, AC, AD, AE, AF 0.33 0.67 10Plus BC 0.40 0.84 13Plus BC, BD 0.47 1.05 16Plus BC, BD, CD 0.53 1.24 19Plus BC, BD, BE 0.53 1.27 19Plus BC, BD, BE, CD 0.60 1.49 22Plus BC, BD, BE, CD, CE 0.67 1.71 26Plus BC, BD, BE, BF 0.60 1.51 23Plus BC, BD, BE, BF, CD 0.67 1.74 26Plus BC, BD, BE, BF, CD, CE 0.73 2.00 30Plus BC, BD, BE, BF, CD, CE, CF, 0.80 2.26 34All 15 pairwise comparisons 1.00 3.00 45

Average precision over 15 pairwise comparisonsData ensemble, 1 trial each comparing:

Simulation results: ‘average’ precision

• Equivalent number of trials • 0.67*15 = 10 trials worth of data

Pairwise MTC Equivalent # trialsAB, AC, AD, AE, AF 0.33 0.67 10Plus BC 0.40 0.84 13Plus BC, BD 0.47 1.05 16Plus BC, BD, CD 0.53 1.24 19Plus BC, BD, BE 0.53 1.27 19Plus BC, BD, BE, CD 0.60 1.49 22Plus BC, BD, BE, CD, CE 0.67 1.71 26Plus BC, BD, BE, BF 0.60 1.51 23Plus BC, BD, BE, BF, CD 0.67 1.74 26Plus BC, BD, BE, BF, CD, CE 0.73 2.00 30Plus BC, BD, BE, BF, CD, CE, CF, 0.80 2.26 34All 15 pairwise comparisons 1.00 3.00 45

Average precision over 15 pairwise comparisonsData ensemble, 1 trial each comparing:

Early thrombolysis for acute myocardial infarction (AMI).

• Technology appraisal for National Institute for Clinical Excellence (Boland et al, 2003)

• Affects 274,000 people each year

• 50% die within 30 days of AMI.

• National Service Framework for heart disease states thrombolysis should be given within 60 minutes.

• Thrombolysis = pharmaceutical agents to dissolve blood clots.

Thrombolytic treatments

• Four treatments assessed: – Streptokinase (SK), – Tissue-plasminogen activator (t-PA), – Tenecteplase (TNK)– Reteplase (r-PA).

• Distinction made between accelerated t-PA and standard t-PA.

• SK + t-PA used in two trials

Thrombolysis conclusions(Boland et al, 2003)

• “Definitive (sic) conclusions on efficacy are that streptokinase is as effective as non-accelerated alteplase, that tenecteplase is as effective as accelerated alteplase, and that reteplase is at least as effective as streptokinase.

• “Some conclusions require interpretation of data, i.e. whether streptokinase is as effective as, or inferior to accelerated alteplase; and whether reteplase is as effective as accelerated alteplase or not.

• “Depending on these, two further questions on indirect comparisons arise, whether tenecteplase is superior to as streptokinase or not and whether reteplase is as effective as tenecteplase or not.”

What is needed?

• A single statistical analysis providing estimates for all the 15 pairwise comparisons, between 6 treatments.– Using classical or Bayesian statistical methods.

• An assessment of which of these treatments is most likely to be best.

• Method– Bayesian Markov chain Monte Carlo method

Were all relevant treatments included?

• Primary percutaneous transluminal coronary angioplasty (PCTA).

• Keeley et al meta-analysis of PCTA vs thrombolysis (22 RCTs)– PCTA is better than thrombolysis (OR 0.70 [0.58 –

0.85])– But surely the relevant comparison is the ‘best’

thrombolytic NOT the ‘average’ one?

• 7 treatments• 21 possible pairwise comparisons

Extended evidence structure

SK t-PA At-PA Sk+tPA r-PA TNK PCTA

Boland

8 x x1 x x x1 x x1 x x2 x x1 x x

Keeley

8 x x3 x x11 x x

Consistency of odds ratios and CIs: fixed effect analysis

1.00 0.86 0.96 0.95 0.52(0.94 - 1.06) (0.78 - 0.94) (0.87 - 1.05) (0.79 – 1.12) (0.36- 0.73)

0.99 0.86 0.96 0.90 0.86 0.63(0.94-1.06) (0.78 – 0.93) (0.87 –1.05) (0.80 – 1.01) (0.74 – 1.00) (0.52 – 0.77)

MTC

Direct data

PCTAt-PA Acc t-PA SK + t-PA r-PA TNK

Lumping vs splitting: Fixed effect analysis of At-PA versus PCTA

Keeley lumped analysis 0.79(t-pa & at-PA vs PCTA) (0.63 - 0.99)Our lumped analysis 0.79

(t-pa & at-PA vs PCTA) (0.63 - 0.98)

Direct data 0.81(at-PA vs PCTA) (0.64 – 1.02)

MTC analysis 0.74(at-PA vs PCTA) (0.61 – 0.89)

Lumping

Splitting

Breaking randomisation?

• NO! – There are statistically invalid methods.

– Our MTC analyses are based only on randomised comparisons.

– Lack of assumptions about baseline risks across studies.

– A weighted combination of valid estimates of treatment effect.

Generalisability• Key assumption in MTC is that relative

treatment effect of one treatment vs another is same across entire set of trials.

• Irrespective of which treatments were actually evaluated in each trial

– True odds ratio of A vs B trials is exactly the same as the A vs B odds ratio in the A vs C, B vs C trials. (fixed effect)

– Common distribution of treatment effects is the same across all sets of trials (random effects).

Generalisability• Helpful to consider which target population we

are making treatment recommendation for. – The type of patients in the previous A vs B trials?

OR– The kind of patients in ALL the MTC trials?

• Clinical and epidemiological judgement necessary

• Poor judgements may introduce heterogeneity

Should MTC be routine?

• A more appropriate question is can MTC analyses be avoided?– No real alternative in multi-treatment decision

making.

• Transparency– No need to lump treatments – No ‘under the table’ indirect comparisons

• MTC same assumptions as meta-analysis

Future areas of research

• What is the extra literature searching burden of MTC – how far should searches go?– Should we include discontinued treatments in the

evidence base? – Placebo controlled trials?

• Greater awareness of MTC by commissioners of research when ‘scoping’ HTAs– NICE Obesity appraisals– Thrombolysis & PCTA