Economic evaluations alongside clinical trials: what they contribute, how they are performed and their

limitations

Dr. Stavros Petrou

Presentation to Nottingham CTU

19th June 2008

What is economic evaluation?

• Premise: scarce (health care) resources• Aim: to maximise health gain with the available

resources• Method: compare cost and consequences of

interventions• Balance: about costs and consequences, inputs

and outputs• Economic evaluation: explicit criteria for making

choices.

Definition of economic evaluation

• Definition of economic evaluation:

“The comparative analysis of alternative courses of action in terms of both their costs and their consequences” (Drummond et al, 1997)

• Requires:– a comparison of two or more alternatives– examination of both costs and

consequences

Types of economic evaluation

NOCosting studyYESIs effectiveness of interventions equal?

YESNOCost minimization study

Can all outcomes be valued in monetary terms ( e.g. willingness to pay)?

YESCost benefit analysis

NO

Can outcomes be measured as quality adjusted life years? YES

NOCost-effectiveness analysis

Cost-utility analysis

Is there good evidence on effectiveness of interventions being compared?

Economic evaluation alongside trials

Two independent groups

Control group Treatment group

Patient (Cost,

Effect)

1 ( CC1, EC

1 )

2 ( CC2, EC

2 )

3 ( CC3, EC

3 )

.

.

nC ( CCn, EC

n )

Mean: ( CC, EC )

Patient (Cost,

Effect)

1 ( CT1, ET

1 )

2 ( CT2, ET

2 )

3 ( CT3, ET

3 )

.

.

nT ( CTn, ET

n )

Mean: ( CT, ET )

CT - CC

ET - EC

Economic evaluation alongside trials

Two independent groups

Control group Treatment group

Mean: ( CC, EC ) Mean: ( CT, ET )

Incremental cost-effectiveness ratio

The cost-effectiveness plane

C

New treatmentmore costly

New treatment more effective

New treatmentless effective

New treatmentless costly

NENW

SW SE

Existing treatmentdominates

New treatment dominates

New treatment less costlybut less effective

New treatment more effectivebut more costly

New treatment cost-effective

New treatmentcost-ineffective

The cost-effectiveness plane

C

New treatmentmore costly

New treatment more effective

New treatmentless effective

New treatmentless costly

NENW

SW SE

Maximum acceptable ICER

NICE: method of operation

• Preferred measure of cost-effectiveness:– Quality-adjusted life year (QALY)– Alternatives - e.g. cost per life year gained -

acceptable

• No absolute threshold for level of acceptability:– no empirical basis for setting a value– may in some circumstances want to ignore threshold– A set threshold implies efficiency has absolute priority

over other objectives (e.g. fairness)– many technology suppliers are monopolies; a

threshold would discourage price competition

Cost per QALY results from NICE

Those in bold: rejected

Source: N Devlin, D Parkin. Does NICE have a cost-effectiveness threshold and what other factors influence its decisions? A binary choice analysis. Health Economics 2004: 13(5):437-52.

Why QALYs as a measure of outcome?

• To use cost-effectiveness as a guide to decision-making, we need to compare the c-e of different uses of resources

• Therefore we need an effectiveness measure that can be used in a wide range of settings:

• Life-years gained– but only where survival is main outcome

• Quality adjusted life years (QALYs)– Composite of survival and quality of life

Time in years t

01 2 3 4 5 6

1.0

Health profile with intervention

Health profilewithout intervention

QALYs gained

Qualityof life

valuationu

The EuroQol EQ-5DThe following questions are designed to tell us about your state of health today. Please look at each group of questions, and then tick the statement which best describes youown health state today. You should make five ticks in all, one for each group.

MobilityI have no problems walking about ____I have some problems walking about ____I am confined to bed ____

Self-careI have no problems with self-care ____I have some problems washing or dressing myself ____I am unable to wash or dress myself ____Usual activitiesI have no problems performing my usual activities ____I have some problems with performing my usual activities ____I am unable to perform my usual activities ____Pain/discomfortI have no pain or discomfort ____I have moderate pain or discomfort ____I have extreme pain or discomfort ____Anxiety/depressionI am not anxious or depressed ____I am moderately anxious or depressed ____I am extremely anxious or depressed ____

Tariffs for the EuroQol EQ-5D Coefficient

Constant 0.081

Mobility

- Some problems 0.069

- Confined to bed 0.314

Self care

- Some problems 0.104

- Unable to wash/dress 0.214

Usual activities

- Some problems 0.036

- Unable to perform 0.094

Pain/discomfort

- Moderate 0.123

- Extreme 0.386

Anxiety/depression

- Moderate 0.071

- Extreme 0.236

N3 (Level 3 at least once) 0.269

Why randomised trials?

• Most treatments do not have large effects; reliably detecting moderate effects requires studies that simultaneously avoid:– moderate bias

• proper randomisation• intention-to-treat analysis• avoidance of inappropriate sub-group analysis

– moderate random error• adequate size

Why economic assessment in clinical trials?

• Many health economists advocate models using lots of data sources: trial, non-trial, summary data etc

• But...issues of bias and random error also affect incremental resource use and health outcomes

• And, trials provide patient-level data, useful for:– Dealing with patient heterogeneity– Examining covariance, e.g. between costs and outcomes – Building and validating models, eg to extrapolate

• Trials allow prospective measurement of resources & outcomes of interest

• Incremental cost of economic evaluation alongside trials is low

UK Collaborative ECMO Trial

• Pragmatic RCT• 185 mature (35 weeks, 2kgs) infants with

severe respiratory failure (ox. index 40) • Infants recruited from 55 centres in 1993-5• Randomisation to ECMO: 1 of 5 specialist

centres, cannulated, ECMO support• Randomisation to CM: conventional care• Outcomes: survival without severe disability up

to 7 years of age

Design and analytical issues

• Costs: measurement and valuation

• Consequences: measurement and valuation

• Analytical issues: within and beyond RCTs

Perspectives and types of costs • Direct costs

– Health care system– Other care inputs, e.g. social services– Patient, family, carer expenses

• Informal care costs– Opportunity cost of unpaid informal care

• Indirect costs– Time off work, reduced productivity– Early retirement– Premature mortality

• Transfer payments– Payments such as social security benefits that

redistribute output with no exchange of goods or services

Three elements of cost

• Resource use (cost generating event)– a day in hospital– a GP consultation

• Unit cost– cost per in-patient day / per hospitalisation– cost per GP consultation / per GP minute

• Cost– the product of resource use and unit costs

Measurement of resource use

• All significant resource inputs during first 7 years of life• Trial data collection forms:

– transport (mode, distances)– duration and intensity of neonatal care

• Observational research for infant death• Postal questionnaires, validated by information from

hospital records:– post-discharge hospital and social service

utilisation• GP records:

– community service utilisation

Valuation of resource use

• Unit costs employed to value resource use

• Neonatal care – top down methodology

• Readmissions - Reference cost schedules

• Community service utilisation - Published cost data (previous studies, PSSRU, etc.)

• Drugs – BNF

• £, 2002-3 prices

Time horizon and discounting

• Should extend far enough into the future to capture all costs and consequences of interventions being evaluated

• ECMO Study – time horizon initially reflected that of RCT

• Costs (and consequences) occurring beyond the first year of life must be reduced to present values

• Rationale for discounting- Time preference- Opportunity cost – market basis

Mean costs and mean cost differences

Cost category ECMO (n=93) CM (n=92)

Mean (SD) Mean (SD) Meandifference

P value

Death 434 (665) 846 (714) -412 <0.0001

Transport 2147 (3080) 296 (777) 1851 <0.0001

Initial hospitalisation 22996 (21618) 6143 (13144) 16853 <0.0001

Hospital readmissions 1518 (3868) 1127 (3063) 391 0.447

Outpatient hospital care

970 (1592) 496 (1008) 474 0.017

Community care 1976 (2882) 1247 (2647) 792 0.075

Other costs 229 (1255) 74 (182) 155 0.241

Total costs 30270 (24380) 10229 (18356) 20041 <0.0001

Source: Petrou S, Bischof M, Bennett C, Elbourne D, Field D, McNally H. Cost-effectiveness of neonatal ECMO based on seven year results from the UK Collaborative ECMO Trial. Pediatrics 2006; 117(5): 1640-1649.

Bootstrap mean cost differences

Cost category Mean cost difference

(ECMO – CM)

Bootstrap mean cost difference (95% CI)

Death -412 -406 (-614, -209)

Transport 1851 1849 (1251, 2497)

Initial hospitalisation 16853 16,826 (11,775, 22,044)

Hospital readmissions 391 371 (-603, 1334)

Outpatient hospital care 474 481 (97, 875)

Community care 792 730 (-62, 1549)

Other costs 155 156 (-13, 443)

Total costs 20041 20057 (13690, 26318)

Source: Petrou S, Bischof M, Bennett C, Elbourne D, Field D, McNally H. Cost-effectiveness of neonatal ECMO based on seven year results from the UK Collaborative ECMO Trial. Pediatrics 2006; 117(5): 1640-1649.

Measurement of outcomes at 7 years

• Survival period → life years gained

• Neurodevelopmental assessments performed by developmental psychologist across 6 domains:– cognitive ability - behaviour– neuromotor skills - hearing– general health - vision

• Each domain defined as normal, impaired or mild, moderate or severely disabled

• Overall status defined by highest degree of impairment or disability in any domain

→ disability-free life years gained

• Limitations of QALYs in childhood context

ECMO(n=93)

CM(n=92)

Deaths

Before discharge 28 (30.1%) 54 (58.7%)

Between discharge and one year 2 (2.2%) 0 (0.0%)

Between one years and four years 1 (1.1%) 0 (0.0%)

Between four years and seven years 0 (0.0%) 0 (0.0%)

Total 31 (33.3%) 54 (58.7%)

Loss to follow-up

Between discharge and one year 1 (1.1%) 1 (1.1%)

Between one year and four years 2 (2.2%) 2 (2.2%)

Between four years and seven years 3 (3.2%) 1 (1.1%)

Total 6 (6.5%) 4 (4.3%)

Final assessment at seven years of age

Severe disability 3 (3.2%) 0 (0.0%)

Moderate disability 9 (9.7%) 6 (6.5%)

Mild disability 13 (14.0%) 11 (12.0%)

Impairment only 21 (22.6%) 15 (16.3%)

No abnormal signs or disability 10 (10.8%) 2 (2.2%)

Known survivors with no disability 31/56 (55.4%) 17/34 (50.0%)

Cost-effectiveness of neonatal ECMO

Incremental cost-effectiveness ratio = C / E

= £13,385 per life year gained

or

= £23,566 per disability-free life year gained

NB: Natural or physical unit of outcome, which ignores full range of consequences.

Cost-effectiveness plane, life years gained

-£40,000

-£30,000

-£20,000

-£10,000

£0

£10,000

£20,000

£30,000

£40,000

-4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0

Incremental life years gained

Incr

emen

tal c

osts

ECMO more effective, more costly

ECMO more effective, less costly

ECMO less effective, less costly

ECMO less effective, more costly

maximum ICER

Cost-effectiveness plane, disability-free life years gained

-£40,000

-£30,000

-£20,000

-£10,000

£0

£10,000

£20,000

£30,000

£40,000

-4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0

Incremental disability-free life years gained

Incr

emen

tal c

osts

ECMO more effective, more costly

ECMO more effective, less costly

ECMO less effective, less costly

ECMO less effective, more costly

maximum ICER

Cost-effectiveness acceptability curves, probability that neonatal ECMO is cost-effective after seven years

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

£0 £10,000 £20,000 £30,000 £40,000 £50,000

Willingness to pay threshold (£)

Prob

abili

ty co

st-e

ffect

ive

Life year gained Disability-free life year gained

0.69

0.98

Mean net benefits of neonatal ECMO

Mean net benefits = Rc.E - C

Rc Life year gained Disability-free life year gained

Mean net benefit

95% CI Mean net benefit

95% CI

0 -20,130 (-26,027, -13,522) -19,949 (-25,997, -13,375)

10,000 -5,299 (-14,475, 3,750) -11,387 (-20,394, -244)

20,000 9,532 (-6,691, 25,351) -2,825 (-18,377, 13,804)

30,000 24,362 (1,474, 47,314) 5,737 (-16,245, 29,393)

40,000 39,193 (8,926, 69,513) 14,299 (-14,863, 44,781)

50,000 54,024 (15,496, 91,567) 22,861 (-13,246, 60,319)

Are trial-based economic evaluations sufficient?

• Under the right circumstances, Yes

• Like clinical evidence, economic evidence can stand alone or be synthesised

• Requirements: reasonable comparators, adopts final outcomes, collects data on a sufficiently broad set of services, adequately powered, sufficiently long follow-up, representative patient population

• Problems: Use of inappropriate statistical tests, lack of power, failure to handle missing data, lack of intention-to-treat analysis, follow-up too short, lack of transferability

Was the ECMO trial-based economic evaluation sufficient?

• Reasonable comparator • Representative patient population • Intention to treat analysis • Adequately powered • Appropriate perspective • Sufficiently long follow-up x• Adopts final outcomes ?• Appropriate statistical tests • Handled missing data

Longer-term cost-effectiveness

• Simple decision-analytic model• Assumptions:

– restricted to first 18 years of life– survivors to 7 years survive to 18 years– disability status at age 7 does not vary– excess annual costs during years 4-7 continue

during years 8-18

ICER: £7344 per life year gained £11802 per disability-free life year

gained

Vehicles for economic evaluation• Prospective collection of data alongside randomised

controlled trials– Least subject to bias, control over instruments, low incremental

cost. May need to supplement.

• Prospective collection of data alongside non-randomised studies– More subject to possible bias, control over instruments, low cost.

• Retrospective analysis of available data– Low control over design and data, subject to bias.

• Modelling study– Data from different sources, combined in decision-analytic

models. Hard to validate. Useful and often unavoidable adjunct to trials.

Limitations of trials as a vehicle for economic evaluation

Trial limitations

Inappropriate or partial comparisons

More than one trial

Partial measurement

Unrepresentative practice

Intermediate outcomes

Limited follow-up

No trials

NICE Examples

Temozolomide (recurrent malignant glioma)

Drugs for Alzheimer’s

Riluzole (resource use)

Glycoproteins

Beta interferon (MS)

Implantable cardioverter defibrillators

Liquid-based cytology

Modelling

• Hence modelling aims to address all these questions and can be used instead of or as a complement to trial evidence– Structure the economic question– Extrapolate beyond observed data– Links intermediate and final endpoints– Generalizes results to other settings/patient groups– Synthesises evidence and can create head-to-head

comparisons where RCTs don’t exist– Can indicate the need for further research

Models should:

• Represent a simplification of the real world• Encourage decision-makers to be explicit• Reflect current clinical practice and use

appropriate comparators• Be based on the best quality data available• Cover the appropriate time period• Include sensitivity analysis to explore uncertainty

of data inputs and model structure• Be transparent and reproducible• Have internal and external validity

The realities of research funding• A large proportion of funding for economic

evaluation is attached to trials• Three options available to analysts:

– Satisfy expectation of standard trial-based economic evaluation and risk misleading results

– Refuse to collaborate

– Pragmatic collaboration:• Seek opportunities to use modelling to inform trial

design• Ensure sufficient budget for analysis which includes

synthesis and modelling

A pragmatic way forward?

Synthesis andmodelling with

updatedevidence

Setting ofresearchpriorities

Primaryresearch

(e.g. RCTs)

Identifydecisionproblems

Synthesis and modelling

givenavailableevidence

Suggested checklist for assessing economic evaluationsSuggested checklist for assessing economic evaluations

1)Was question well-defined?

2)Were alternatives clearly described?

3)What evidence on effectiveness was used?

4)Were resources measured and valued fully & appropriately?

5)Was discounting necessary and was it performed?

6)Were incremental costs and outcomes analysed?

7)Was an adequate sensitivity analysis performed?

8)Are the results adequate to inform purchasing?

9)Are the conclusions justified?

10) Are the results applicable to the local population? Source: Drummond MF, O’Brien, B, Stoddart GL, Torrance GW. Methods for the economic evaluation of

health care programmes. 2nd edition. Oxford: Oxford University Press, 1997.

Drummond MF, Jefferson T, et al. Guidelines for authors and peer reviewers of economic submissions to the BMJ. BMJ 1996; 313:275-83.