The CLAHRC Yorkshire
& Humber
Illustrating uncertainty in
extrapolating evidence for cost-
effectiveness modelling
Laura Bojke
The CLAHRC Yorkshire
& Humber
Extrapolation project team
• Stephen Palmer, Andrea Manca, Ronan
Mahon, Miqdad Asaira (University of York)
• Alan Brennan (PI), John Stevens, Nick
Latimer, Paul Tappenden, Suzy Paisley,
Kate Ren (University of Sheffield)
• Keith Abrams (University of Leicester)
• Chris Jackson (University of Cambridge)
The CLAHRC Yorkshire
& Humber
• The need for extrapolation
• Extrapolation methods – Not extrapolation from surrogate outcomes
• Uncertainty in extrapolation
• Approaches to dealing with uncertainty in
extrapolation
– Examples
• Areas for further research
Structure of presentation
The CLAHRC Yorkshire
& Humber
“estimating beyond the original observation range”
• An appropriate time horizon for evaluation
– All (incremental) positive/negative effects observed
– Patients lifetime when there are mortality effects
• Evidence base falls short of this - censoring
– High costs of research
– Loss to follow up
– Early market entry
• Modelling to extrapolate short term outcomes
– Involves assumptions and may be data sparse
Impetus for extrapolation
The CLAHRC Yorkshire
& Humber
Evidence gap resulting from time
horizon mismatch
Just extrapolation? Temporal Uncertainty in Cost-effectiveness Decision Models. Mahon
R, Manca A, Bojke L, Palmer, S Jackson C, et al.
The CLAHRC Yorkshire
& Humber
When does the mismatch
matter? • Not just the time difference between observed
and unobserved
• Relates to data maturity
– Number of people that have experienced the event of
interest
• Earlier outcomes may be of more value
– Proportion of costs & QALYs attributable to observed
period
– Discounting
The CLAHRC Yorkshire
& Humber
Mature data
Mean overall survival gain with aflibercept plus FOLFIRI vs placebo plus FOLFIRI in
patients with previously treated metastatic colorectal cancer. F Joulain, I Proskorovsky,
C Allegra, J Tabernero, M Hoyle, S U Iqbal and E Van Cutsem.
The CLAHRC Yorkshire
& Humber
Immature data
The CLAHRC Yorkshire
& Humber
Extrapolating TTE parameters
• Assumptions about how trends will continue
– Progression free or overall survival in cancer trials
• Estimation involves: (1) modelling the observed
data (Kaplan-Meier); (2) modelling the
unobserved data period.
– Statistical models (often parametric) are fitted to
observed data
• Choice of model informed by goodness of fit parameters
– Fitted model is used to extrapolate the un-observed
period to determine the TTE
– Assume hazard trends will continue
The CLAHRC Yorkshire
& Humber
Choice of distribution
• Choice of parametric distribution to fit:
– Exponential, Weibull, log-normal etc.
• Some advocate use of exponential as the default
– Proportional hazards, constant hazards
• Different assumptions/ survival estimates
• Data limitations
– IPD or aggregate (constant hazard)
The CLAHRC Yorkshire
& Humber
Extrapolating non-TTE parameters
• Repeated measurement of individuals over time
– Genuinely discrete or genuinely continuous
• Similar principles to TTE
– Set of observations on one subject tends to be inter-
correlated
• Driven by data availability
– Aggregate = discretised (Markov models)
– IPD = patient experienced models, continuous
outcomes – regression, e.g. risk equations
The CLAHRC Yorkshire
& Humber
Extrapolating resources
use/costs & utilities
• Typically models employ a simplistic approach to
extrapolation of costs and utilities
– Homogeneous w.r.t both time and patients'
characteristics
– Follow the dynamics of associated TTE parameters
• Resource use/costs and utilities may be more
nuanced
– Adaptation to health state
– Uncertainty may not resolve over time/with further
evidence
The CLAHRC Yorkshire
& Humber
Uncertainty in extrapolation
• Others have termed this ‘temporal’ uncertainty
• Extent of uncertainty is difficult to determine • Uncertainty cannot be resolved (now), because we
cannot observe the future
• Efforts should be made to characterise any
uncertainty − Accurate estimates of long-term cost-effectiveness
− Better adoption decisions
− Assess the need for further research
− Input into the design of further research
− Delay decisions?
The CLAHRC Yorkshire
& Humber
Does uncertainty in extrapolation
matter?
Just extrapolation? Temporal Uncertainty in Cost-effectiveness Decision
Models. Mahon R, Manca A, Bojke L, Palmer, S Jackson C, et al.
The CLAHRC Yorkshire
& Humber
“primary source of extrapolation uncertainty
in decision model results is the choice of
survival projection function rather than
sampling variation”
Bagust & Beale, 2014 in response to Latimer,
2013
The CLAHRC Yorkshire
& Humber
How should uncertainty be
assessed?
• NICE recommends that any extrapolation should
be assessed by ‘‘both clinical and biological
plausibility of the inferred outcome as well as its
coherence with external data sources’’
– Does not suggest specific methods to do this
• Latimer, 2013 recommends an assessment of
plausibility
– Concerns may be context specific
The CLAHRC Yorkshire
& Humber
Model selection
• Assessment of goodness-of-fit for observed
– Log-cumulative hazard, residuals plots, AIC/BIC etc
– Suggested the focus should be on fit for stable portion
of data
• Lots of things that prevent good model fit
– More flexible model forms, generalised gamma
• A model that fits the sample data well may not
provide good long-term predictions
• Clinical plausibility
• Causally known associations
The CLAHRC Yorkshire
& Humber
Just scenarios?
• Where there is no evidence to form an opinion
about the plausibility of assumptions/do not want
to rely on this
– Deterministic sensitivity analysis
– Illustrate how conclusions may change if the evidence
were to change and possibly inform the need for
further validation
– Not easy for decision-makers to interpret
• May implicitly average different scenarios
The CLAHRC Yorkshire
& Humber
Model averaging
• Explore uncertainty in the true underlying
distribution for survival extrapolation
– Bayesian view of model averaging
– Likelihood used to measure fit of a model to observed
data (plausibility)
– Unlikely scenarios: down-weighted/excluded
– Recommended that models should ‘stake out the
corners in the model space’
• Implications for extrapolation uncertainty
– Model-averaged inferences will generally have
greater uncertainty
The CLAHRC Yorkshire
& Humber
Discrepancy parameters • Where there are many sources of uncertainty in a
model
– Strong, et al method to identify the most important
sources of structural uncertainty
– Discrepancy parameters are added to intermediate
outputs of the model, rather than model inputs.
• e.g. the years of life spent in a health state could be replaced
by LY + delta_LY, where the discrepancy parameter delta_LY
is represented as a probability distribution based on experts
priors
– PSA to quantify which of the discrepancy parameters
are associated with the greatest variations in net benefit
The CLAHRC Yorkshire
& Humber
Stability of survival extrapolation
• Negrin, et al, 2016 explore uncertainty about the
stability of extrapolated parameters over time
– Explore past behaviour of distributions
• Built intermediate data sets (4, 5, 6 and 7 year)
– Numerous survival models fitted to 8 year data for 2
hip prosthesis types separately (Charley, Spectron)
• Two additional distributions to reflect stability over past
behaviour (optimistic and sceptical)
• BMA(1) using BIC (1/6 weight to 6 distributions)
• BMA(2) questions the value of BIC to select or average
across models (1/3 weight assigned to optimistic, sceptical,
others)
The CLAHRC Yorkshire
& Humber
Negrin, Nam, Briggs. Bayesian solutions for handling uncertainty in
survival extrapolation. Medical Decision Making. 2016 forthcoming
The CLAHRC Yorkshire
& Humber
Using external sources of data
• Sources: administrative data, disease registries,
cohort studies
– Longitudinal data over an appropriate time horizon
with sufficient follow up points
– Cross sectional data with sufficient coverage of
patients
• Assumed relationship between internal and
external data (baseline and treatments)
• Combine using Bayesian estimation
The CLAHRC Yorkshire
& Humber
Elicitation
• Use formally elicited priors
• Represents current level of knowledge regarding
the uncertainty of interest
– Expressed quantitatively
• Uncertainty about how this can be implemented:
– Alternative survivor function?
– Weights for survivor functions, e.g. TIDI
– Synthesis with internal data?
– Could elicit at multiple time points – how?
The CLAHRC Yorkshire
& Humber
Elicitation (2)
• Complexity of elicitation task
– Who are the relevant experts
– How can we weight experts according to accuracy
– Complex parameters difficult to elicit
– Time requirements
– Synthesis
• Eliciting uncertainty about priors
– Needed to fully characterise uncertainty and inform
long term follow up
The CLAHRC Yorkshire
& Humber
Case study: Extrapolation project
• Cost-effectiveness of biologics for PsA
– Probabilistic cohort model
– Time horizon = 40 years (3-month cycles)
– Initial response using PsARC @ 3-months
– Associated HAQ gain
– Assumptions that biologics halt progression
– Constant rate of progression on biologics (mean = 0)
• Different ways in which this data can be utilised in the model
The CLAHRC Yorkshire
& Humber
1) Initial HAQ gain
due to treatment
2)Long term HAQ
trajectory while on
treatment
3)Rebound of HAQ
once withdrawn
from treatment
4)Long term HAQ
trajectory post
withdrawal from
treatment
0
0.5
1
1.5
2
2.5
3
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40
HA
Q S
core
Time (Years)
2. HAQ on treatment
1. Initial
HAQ
Gain
Natural History
3. HAQ
Rebound
4. HAQ Post-
Withdrawal
Treatment Arm
Issues regarding HAQ
trajectory
The CLAHRC Yorkshire
& Humber
Impact of uncertainties in PsA
model
Cumulative NMB over time base case 3-month HAQ change of 0.
Within
trial period
Breakeven
point
etanercept
The CLAHRC Yorkshire
& Humber
Exploring long term HAQ trajectory
for treatment responders
• Key driver for cost-effectiveness.
• Assumed mean of 0 + expert elicitation exercise
(SD)
• But don’t know:
– Constant HAQ gradient?
– Independent HAQ gradient at each 3-month model
cycle?
• Limited by lack of external data.
The CLAHRC Yorkshire
& Humber
Impact of different HAQ trajectories
The CLAHRC Yorkshire
& Humber
Constant HAQ gradient over
time • Randomly draw HAQ change from a normal
distribution (mean = 0 and SD = 0.022)
• Apply this as the constant 3-month HAQ score
assuming perfect serial correlation.
• Process is repeated for each iteration of the
model
The CLAHRC Yorkshire
& Humber
Cumulative NMB over time – constant HAQ gradient over time.
Breakeven
point
etanercept
Within
trial period
The CLAHRC Yorkshire
& Humber
Independent HAQ gradient at
each cycle • Previously perfectly serially correlated HAQ
trajectories
• Independently sampled HAQ scores for each
cycle
• Repeatedly randomly draw HAQ changes from a
normal distribution with mean 0 and standard
deviation 0.022
The CLAHRC Yorkshire
& Humber
Cumulative NMB over time - independent HAQ gradient at each 3-month model cycle.
Breakeven
point
etanercept
The CLAHRC Yorkshire
& Humber
Comparing the two scenarios
Constant HAQ
gradient over
time
Independent
HAQ gradient
at each cycle
Palliative Care Probability
cost-effective
(£20,000 per
QALY)
0.30 0.38
Etanercept 0.31 0.39
Infliximab 0.04 0.00
Adalimumab 0.18 0.17
Golimumab 0.18 0.07
EVPI (population) £768 mil £378 mil
The CLAHRC Yorkshire
& Humber
Conclusions • Models may be well characterised over the
observed period, but there may be considerable
uncertainty regarding behaviour over time.
• None of the methods described can fully
describe the extent of extrapolation uncertainty.
– Statistical models for extrapolation cannot be fully
validated.
• Focus on which influence conclusions: adoption
decision, decision uncertainty and VOI.
– Degree of effort should be proportional to the
influence of uncertainty.
The CLAHRC Yorkshire
& Humber
Areas for further research • Ability to validate for ‘crucial’ uncertainties:
– Internal/external validity
– Use of external evidence/experts priors
• How to generate experts priors for extrapolation
– Which parameters to elicit
• Link to OIR/delayed decisions
– Break even point important
• Discounting out the problem
The CLAHRC Yorkshire
& Humber
(some) References • Methods of Extrapolating RCT Evidence for Use in Economic Evaluation
Models, MRC report.
• Extrapolating Survival from Randomized Trials Using External Data: A
Review of Methods. Jackson C, et al. Medical Decision Making, 2016.
• Survival Analysis and Extrapolation: Modeling of Time-to-Event Clinical Trial
Data for Economic Evaluation: An Alternative Approach. Medical Decision
Making. Bagust A & Beale S. 2014.
• Davies C, Briggs A, Lorgelly P, et al. The “hazards” of extrapolating survival
curves. Medical Decision Making. 2013; 33 (3).
• Negrin, Nam, Briggs. Bayesian solutions for handling uncertainty in survival
extrapolation. Medical Decision Making. 2016 forthcoming.
• NICE DSU Technical support document 14: Survival analysis for economic
evaluations alongside clinical trials: Extrapolation with patient-level data.
Latimer N. 2011.