1

MULTI-CRITERIA DECISION

ANALYSIS FOR HEALTHCARE

DECISION MAKING

Maarten IJzerman, Nancy Devlin, Praveen Thokala and

Kevin Marsh on behalf of the ISPOR MCDA Task Force

November 10, 2014

2

Vakaramoko Diaby , Kaitryn Campbell , Ron Goeree - Multi-criteria decision analysis (MCDA) in health care:

A bibliometric analysis, Operations Research for Health Care Volume 2, Issues 2013 20 – 24

http://dx.doi.org/10.1016/j.orhc.2013.03.001

4

What decisions were MCDAs designed to

support?

Source: Marsh et al (2014)

3

To develop guidance for outcomes researchers

and decision makers on the use and application

of MCDA in healthcare decision making

The task force will:

To provide a common definition for MCDA in health

care decision making

To develop emerging good practices for conducting

MCDA to aid health care decision making

Co-Chairs:

Maarten J. IJzerman, University of Twente, Netherlands

Kevin Marsh, Evidera, London

Nancy Devlin, Office of Health Economics, London

Praveen Thokala, University of Sheffield, Sheffield

4

Rob Baltussen, Radboud University Medical Center

Meindert Boysen, National Institute for Health and

Clinical Excellence

Zoltan Kalo, Eotvos Lorand University, Budapest

Thomas Lonngren, NDA group AB, UK and Sweden

Filip Mussen, Jansen Pharmaceutical, Antwerp

Stuart Peacock, British Columbia Cancer Agency,

Vancouver, Canada

John Watkins, Premera Blue Cross, USA

Solicit input from the ISPOR membership regarding

our work and choices made

Identify potential reviewers for draft taskforce reports

5

Maarten IJzerman: Introduction

Nancy Devlin: 1. What do we mean by

MCDA?

Praveen Thokala: 2. Diversity of MCDA

techniques

Kevin Marsh: 3. Which MCDA approach is

best for different kinds of decisions?

Nancy Devlin

Office of Health Economics

6

One of the first tasks for the Taskforce is to establish a working

definition of MCDA.

Not straightforward: different researchers use the term MCDA to

mean quite different things.

How broad should our definition be? e.g.

“Any approach to making decisions that involve multiple criteria”: In

principle, includes purely deliberative decision-making processes.

What kinds of uses of MCDA are we interested in? e.g.,

“Any application that entails consideration of multiple criteria” : In

principle, could include methods for valuing QoL.

We need to define MCDA in a way that is clear, and enables the

Taskforce to focus its efforts where it can add most value.

As generally understood, MCDA

Comprises a broad set of methodological approaches,

stemming from operations research.

Decomposes complex decision problems, where there are

many factors to be taken into account (‘multiple criteria’)

by using a set of relevant criteria.

Provides a way of structuring such decisions, and aims to

help the decision-maker be clear about what criteria are

relevant and the relative importance of each in their

decisions.

Generally entails being explicit about both the criteria and

the weights.

Facilitates transparent and consistent decisions.

7

Belton and Stewart

“An umbrella term to describe a collection of formal

approaches which seek to take explicit account of multiple

criteria in helping individuals or groups explore decisions

that matter”

Keeney and Raiffa

“An extension of decision theory that covers any decision

with multiple objectives. A methodology for appraising

options on individual, often conflicting criteria, and

combining them into one overall appraisal”

13

14

0 10 20 30 40

Yes

No

% of studies

8

15

0% 20% 40% 60% 80% 100%

Support decisions / decision-makers

Valuation of interventions

Elicitation of decision makersvalues

Elicitation of preferences

Deal with uncertainty

% of studies

Yes

No

We propose to focus on:

methods designed to evaluate the options available to

health care decision makers by accounting for all relevant

value criteria, and which explicitly defines, measures and

weights those criteria.

We will not include purely deliberative processes

how these methods can be used at ‘real’ decision points:

that is, where there is direct involvement of a decision

maker; a complete set of factors to be taken into account;

and a ‘real’ decision to be made.

Excludes stated preferences methods, other than where

those are used to weight decision criteria.

9

ISPOR Taskforces on health state utilities, DCE methods,

etc: important to avoid duplication of effort

The goal of PROs, QoL utilities and QALYs is not to make

a decision per se, but to measure health. This provides

one, very important source of evidence to decision makers,

but the aim of using those methods is not to make a

decision in itself, but rather to generate evidence.

While MAU constitutes a type of MCDA, participants in

TTO, DCE etc. are making hypothetical choices – they are

not making ‘real’ decisions.

Stated preference methods may be relevant to weighting

decision criteria: our focus will be on best practise in using

those methods in that specific context, building on existing

best practise.

A range of definitions of MCDA may be found in the

literature.

We have proposed (what we hope is!) a very clear,

focussed definition, which will direct our efforts to the use

of MCDA techniques to aid and structure real health care

decisions. Your feedback is welcome.

10

Praveen Thokala

University of Sheffield

Objective

Criteria

Measure performance

Performance matrix

Weights

Scoring

Decision

How these are done differentiates the MCDA methods

Aggregation

11

Value measurement models

- weighted sum approach

- PBMA, AHP, MAUT, etc

Outranking

- direct comparison of alternatives

- ELECTRE, PROMETHEE, etc

Goal programming

- multi-objective optimisation, LP, etc

Fully quantified methods

The total score for each alternative using the weighted sum model by combining the

scores for each intervention on each criterion

weights for each criterion

V(Ai) = ∑ wj*aij

where wj denotes the relative weight of importance of the criterion Cj and aij is the performance value of alternative Ai when it is evaluated in terms of criterion Cj.

12

Stakeholder expert views and mission

statements of the relevant decision makers

e.g. national/local directives

• Key stakeholders – e.g. o Clinicians

o Patients

• Key national stakeholders – e.g. o Policy

o Legislation

o NICE

• Elicitation of stakeholder values (e.g. focus

groups or surveys) in other situations

• Decision makers should construct or validate

criteria

Direct rating

Likert, visual analogue scales (VAS)

Swing weighting

Analytic Hierarchy Process (AHP)

Indirect methods

Discrete choice experiments

(DCEs)/Conjoint analysis

Increasing complexity

13

Visual Analogue Scale

Likert Scale

Assign highest

weighting to the

criterion which the

decisions maker

considers will lead to

the most important

change in outcomes,

from worst to best

case, for the available

alternatives.

Other weightings are

compared to this and

ranked accordingly. “How big is the difference, and how much do you care about it?”

Zafiropoulos, Nikolaos and Phillips, Lawrence D. and Pignatti, Francesco and Luria, Xavier (2012) Evaluating benefit-risk: an Agency perspective. Regulatory

rapporteur, 9 (6). pp. 5-8. ISSN 1742-8955

Swing Weights

This swing

was judged to

be larger…

…and this one

was judged to

be 60% as

much.

Swing

weights

express

the

relevance

of the

criteria

14

AHP – Pairwise Comparisons

SAATY T. 1977. A scaling method for priorities in hierarchical structures. Journal of

mathematical psychology, 15(3): 234–281.

SAATY T. 1980. The Analytic Hierarchy Process. New York, McGraw-Hill.

• Make pairwise

comparisons of

attributed and

alternatives

• Ratio scale

• Transform the

comparisons into

weights and check

the consistency of

the comparisons

Scale of relative

importance

Understand the relative importance of the different criteria using stated preferences on hypothetical scenarios

* http://help.matrixknowledge.com

15

Different methods e.g. Direct rating

Category estimation

Developing the form of value function (i.e.

importance of different levels of criteria)

e.g. bisection methods and indifference

methods

Intrinsically linked to the choice of the

weighting approach

Increasing complexity

Direct rating/Category estimation method

Direct rating:

1) Rank the alternatives

2) Give 100 points to the best alternative

3) Give 0 points to the worst alternative

4) Rate the remaining alternatives between 0 and 100

Category estimation

assign values to “a small number of categories” in a similar

manner as in the direct rating method:

Give 100 points to the best category

Give 0 points to the worst category

Rate the remaining categories between 0 and 10 Category Poor Satisfactory Good

Salary range Less than £1500 £1500-2500 More than £2500

16

• Define the value function by assessing the form of

the function or by curve drawing

• Needs input from the stakeholders

• Values for different alternatives can be read from the

value curve

Value

Level of an attribute

Objective

Criteria

Measure performance

Performance matrix

Weights

Scoring

Decision

How these are done differentiates the MCDA methods

Aggregation

17

Kevin Marsh

Evidera

Objective

To propose a framework that can help researchers and

decision makers distinguish and select between MCDA

approaches

Overview

Summary of existing typologies

Proposed synthesis of this literature for discussion

Illustration

Typology of approaches

Characterizing different decisions

18

The current MCDA literature

Includes many studies that discuss the advantages

and disadvantages of MCDA approaches.

But only a few that propose criteria for systematically

understanding the advantages and disadvantages of

MCDA approaches

It is doubtful if an identification of the “best” MCDA method

in general can be performed (De Montis et al, 2005)

It is impossible to characterize all the DMS; there might

exist as many DMS as there are decisions (Guitouni and

Martel,1998)

All methods have their assumptions and hypotheses, on

which is based all its theoretical and axiomatic

development - these are the frontiers beyond which the

methods cannot be used (Guitouni and Martel, 1998)

19

Guidelines to distinguish / select MCDA methods

1. Preference elicitation method

1. Mode: direct weighting or trade off?

2. Preference relation assumed: indifference, preference,

incomparability

2. Decision problem: ranking vs choice

3. Data handled: (i) ordinal, cardinal, (ii) deterministic or

non-deterministic

4. Theoretical assumptions: independence, comparability,

transitivity

20

Decision problem

Criteria 1. What is the decision

makers’ objective? Rank options or

measure their value

Criteria 2. Time and resources available

- Amount of data required by the method?

- Collection mode: survey, workshop

21

Criteria 3: Cognitive burden imposed on DM -

nature and amount of data required

Criteria 4: Problem solving process

4a. Break down problem into components

4b. Allow knowledge sharing

Criteria 5: Do the methods assumptions about the

nature of preferences correspond with DM’s

preference structure?

5a. Do DM accept that criteria are comparable?

5b. Do DM have linear or non-linear preferences?

22

Decision problem

Demands on participants

Decision makers

preferences

Theoretical requirement

Practical constraints

Criteria 6: Does the

method meet the

theoretical

requirements of the

DM’s objectives?

Criteria Value

measurement

Outranking

1. Decision – value measurement?

2. Time/ resource – low?

N/a 3. Cognitive effort – low?

4a. Break down problem?

4b. Allow knowledge sharing?

5a. Incomparable criteria

5b. Non-linear preferences N/a

6. Meets theoretical requirements?

Value measurement of outranking approaches?

23

Direct AHP Swing DCE

1. Decision – value measurement? n/a

2. Time/ resource – low?

3. Cognitive effort – low?

4a. Break down problem?

4b. Allow knowledge sharing?

5a. Incomparable criteria n/a

5b. Non-linear preferences ?

6. Meets theoretical requirements? ?

Which value measurement approach?

HTA Authorisation SDM

1. Decision – value measurement?

2. Time/ resource – low?

3. Cognitive effort – low?

4a. Break down problem? ? ? ?

4b. Allow knowledge sharing? ? ? ?

5a. Incomparable criteria

5b. Non-linear preferences ? ? ?

6. Meets theoretical requirements? n/a

24

25

Objective: associate a real number with each alternative in

order to produce a preference ordering consistent with DMs

value judgements

Often divided into two elements

49

Criterion 1

100

0

A B

Criterion 2

100

0

X Y

1. Partial

value

functions

2. Aggregation

using weights

B-A = 100

X-Y = 50

Requires 2 assumptions

50

Criterion 1

100

0

A B

Criterion 2

100

0

X Y

B-A = 100

X-Y = 50

1. Weights are scaling

constants, or trade offs

a=

70

b=70

b=55

a=40

Stakeholder is no worse off moving from intervention a to intervention b

26

51

Criterion 1

100

0

A B

Criterion 2

100

0

X Y

B-A = 100

X-Y = 50

2. Interval scale property – equal increments in value on a partial

value function should represent equal trade offs with other criterion

v1

v2

v4

v3 v5

If

v1-v2 = v2-v3

v1-v2 = v4-v5

Then

v2-v3 = v4-v5

52

1. Direct ration: How is important is outcome i?

2. AHP: how much more important is outcome I vs outcome j?

3. Not obvious that importance ratios expressed in this way correspond to the meaning of

the weigh parameter in the model

4. People express such importance ratios in a context-free way (regardless of the

magnitude of change on the criterion)

27

53

Swing weighing DCE

Weights explicitly determined or

implicit?

28

Importance or trade off?

Qualitative, quantitative, fuzzy?