1 operational research division / division de la recherche opérationnelle défense nationale...

1

Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale

National Defence

How to Use and Abuse MCDA and Consensus Support Methods

Dave Mason

Operational Research Division

National Defence Headquarters

Ottawa, Canada

2


National Defence

Outline

Definitions and applications Cardinal utility methods

» When they are applicable, desired attributes» How to abuse» Summary: How To Use Cardinal Utility Methods

Ordinal methods» Fundamental concepts (Arrow’s Theorem, Voter’s Paradox) » How to abuse common ordinal methods» Heading towards a solution to the Consensus Ranking Problem

– Solution concept: Maximization of rank correlation– Existing statistics: Kendall’s τb, Kemeny/Snell distance– Unified theories, and a new rank correlation statistic: τX

» Summary: How to Use Ordinal Methods

3


National Defence

MCDA andConsensus Support Methods

Multi-Criteria Decision Analysis Methods» Generally, any scheme that evaluates a list of

alternatives across a set of assessment criteria to determine a winner (or final ordering)

– Weighted sum methods are the most common form

Consensus Support Methods» Any methodology that attempts to resolve value

judgements of a group of individuals into a single collective set of judgements

4


National Defence

Applicable Decision Environments

Applicable types of decision environments:» Evaluation of contract bids» Ranking and selection of personnel» Prioritization of an organization’s work program» Inevitable stream of short-fuse ‘Tiger Team’ approaches» …

Generally, involves one of following types of decisions:» 1. Single Choice - selecting the winning alternative » 2. Prioritization - put the list of alternatives in a rank ordering» (3. Resource Allocation – assign resources to each alternative)

Methods are applied after the conduct of, not in lieu of a comprehensive cost-effectiveness analyses

5


National Defence

A Single Overall Measure of Utility?

Key question the analyst must ask: “Is the development of a single overall measure of utility reasonable in this case?”» A single measure may defy definition

» Is a potentially dangerous over-simplification

» But may be a necessary evil to enable an answer to be reached

YES. Then a cardinal utility method may be useful

NO. Then an ordinal methods approach may be a useful alternative

6


National Defence

CARDINAL UTILITYMETHODS

Generally the broad class of weighted sum methods are most commonly employed» Assign weights to key criteria

» Score the alternatives on each criteria

» Arithmetically calculate the weighted sum

Numerous approaches with enhanced mathematical complexity exist » Analytical Hierarchy Process (AHP)

» …

7


National Defence

(Clear glass doors)(Clear glass doors)

????“Black

Box”Method

“BlackBox”

Method

Qualitative and Quantitative InputsQualitative and Quantitative Inputs

Final winner or orderingFinal winner or ordering

Desirable Attributes: 1. Transparency of Method

8


National Defence

Desirable Attributes: 2. Reasonable Fidelity

Complex enough to adequately represent the key dimensions of overall utility, » “KISS” is fine, but don’t go too far

More Sophisticated Mathematics = A Better Solution?» Not necessarily

Oversold obscure methods with gratuitous mathematical complexity = Snake Oil?» You bet!

9


National Defence

How To Abuse WeightedSum Methods

Weight:

Sports CarSedan

Mini-VanSUV

3

45

108

Cost

Fun

Style

Relia

bilit

y

Versa

tility

Total2

61087

3

10327

2

10446

2

61087

86727585

($50K)($30K)($35K)($40K)

Add extra criteriathat boost your alternative

Shave the weightsof criteria that hurt

your alternative

Compress the scale of criteria that hurt your alternative

10


National Defence

How To Abuse WeightedSum Methods

Weight:

Sports CarSedan

Mini-VanSUV

3

45

108

Fun

Style

Relia

bilit

y

Versa

tility

Utility/K$2

61087

3

10327

2

10446

/50 =/30 =/35 =/40 =

1.481.731.711.78

74526071

11


National Defence

The Value Of Weighted Sum Methods

Question: In a weighted sum application, can I fiddle with the weights and scores in subtle ways to make just about any alternative the winner?» Answer: Yeah, pretty much …

Question: I guess that means weighted sum methods are not very useful?» Answer: On the contrary, they are transparent and flexible … just

make sure you use them properly

12


National Defence

Summary: How to UseCardinal Utility Methods

Simple weighted sum, multi-criteria utility methods are fine» MCDA methods are not a substitute for comprehensive cost-

effectiveness analysis» Conduct a (pairwise) sensibility check on criteria utility tradeoffs» Don’t treat cost as another criterion, but as an overall utility divisor» Do a complete sensitivity analysis on both weights and scores to

locate the decision cross-over points Add mathematical complexity if warranted, but there is no

positive value to using unnecessarily complex methods» For example, AHP calculates weights using a prescribed, vague

qualitative scale in a laborious pair-wise assessment process where the DM is permitted to be inconsistent

Horses for courses: Fit the method to your problem

13


National Defence

ORDINAL METHODS

If development of a single overall utility scale is not sensible, then ordinal methods may be a viable option» If there is more than one DM (or more than one criterion being

assessed) the problem is one of ‘social choice’, depending on the societal structure assumed

Nobel laureate Kenneth Arrow’s research from 1953 forms the logic foundation» “It … makes no sense to add the utility of one individual, a psychic

magnitude in his mind, with the utility of another individual.”» Coke or Pepsi? … Strength of preference, or simply preference?» “A utility function whose significance lies entirely in its ordinal

properties is superfluous. If we are concerned with ordinal properties it seems better to represent these directly.”

14


National Defence

Kenneth Arrow andSocial Welfare Functions

A ‘Social Welfare Function’ gives the ordering of alternative social states given the orderings by the individuals in the society

Arrow imposed several ‘apparently reasonable’ conditions:1. Universality – Individuals can express any possible ordering2. ‘Welfare’ Not ‘Illfare’ - social ordering must respond positively (or at

least non-negatively) to alterations in individual values3. Independence From Irrelevant Alternatives – The societal choice

amongst a set of alternatives should be unaffected if other alternatives are added or removed

4. Citizens’ Sovereignty – the social welfare function is not to be imposed (all outcomes are possible)

5. Non-Dictatorship – whenever the dictator prefers x over y, so does society

Arrow’s Theorem: You can’t have all five

15


National Defence

Arrow’s Third Condition & The Voter’s Paradox

The Voter’s Paradox:» Voter 1: a b c

» Voter 2: b c a

» Voter 3: c a b

In this example no winner is declarable, yet if you remove any alternative from consideration a clear winner emerges

Upshot: One cannot construct an ordinal solution using the argument: “If more individuals place a ahead of b, then the consensus or social choice will place a ahead of b”.

16


National Defence

How to Abuse Ordinal Methods

Borda Count (minimum sum of rank values) is commonly employed» Simple example: three rankers, two alternatives

– Ranker A: 1. Coke, 2. Pepsi– Ranker B: 1. Coke, 2. Pepsi– Ranker C: 1. Pepsi, 2. Coke

» Coke wins … any and every rational ranking scheme will agree Ranker C insists on including all contending colas (RC, Kik, …)

» Ranker C tactically and artificially ranks Coke low to benefit his choice:– Ranker A: 1. Coke, 2. Pepsi, T3. Kik Cola & RC Cola– Ranker B: 1. Coke, 2. Pepsi, T3. Kik Cola & RC Cola– Ranker C: 1. Pepsi, T2. Kik Cola & RC Cola, 4. Coke

The Borda Count winner now … Pepsi!! The Problem: Borda count fails any reasonable interpretation of the

‘independence from irrelevant alternatives’ condition The Lesson: Don’t conduct arithmetic operations on rank values

17


National Defence

A Proposed Solution Concept to the Consensus Ranking

Problem

“Given m rankings of n objects, what ranking best represents the consensus opinion?”

In its most general form:» Ties are permitted» Incomplete rankings are permitted» Relative importance of rankers is reflected (usually in form of numerical

weights) Solution Concept Employed: Select a measure of agreement

between pairs of rankings and select that ranking(s) which maximizes overall average agreement.

Possible measures:» Rank correlation coefficient» Distance measure operating on the space of all rankings of n objects

18


National Defence

One Measure of Agreement:Kendall’s Tau-b (1948)

1 if object i ranked ahead of object j

aij = -1 if i ranked behind j

0 if i ranked tied with j or if i=j

n

jiij

n

jiij

ij

n

i

n

jij

b

ba

ba

BA

1,

2

1,

2

1 1),(

Strengths: » If ties are not allowed, it is equivalent to the number of interchanges

of adjacent objects required to convert one ranking into the other.

Weaknesses: » Not well defined when ties are allowed

» The equivalent distance measure fails the triangle inequality test.

19


National Defence

Another Agreement Measure:Kemeny-Snell Distance

(1962)

Axiom 1.1. d(A,B) ≥ 0, with equality holding if and only if A and B are the same ranking.

Axiom 1.2. d(A,B) = d(B,A).

Axiom 1.3. d(A,B) + d(B,C) ≥ d(A,C), and equality holds if and only if B is ‘between’ A and C.

Axiom 2. If A' results from A by a permutation of the objects, and B' results from B by the same permutation, then d(A,B) = d(A',B').

Axiom 3. If A and B agree except for a set S of k objects, which is a segment of both, then d(A,B) may be computed as if these k objects were the only objects being ranked.

Axiom 4. The minimum positive distance is 1.

K&S defined a set of axioms that any reasonable distance measure operating on the set of all orderings of n objects should satisfy …

n

i

n

jijij baBAd

1 12

1 ||),(

They proved that there is a unique distance metric that satsifies these axioms, and that the following is an implemen-tation of that unique measure (using Kendall’s def’n for aij)

20


National Defence

Two Other Proposed Measures of Agreement

The Tau-X Rank Correlation Coefficient:1 if object i ranked ahead of OR TIED WITH object j

aij = -1 if i ranked behind j

0 if i=j )1(),( 1 1

nn

ba

BAij

n

i

n

jij

X

The Half-Flips Metric:» Define Elemental Transformations of any ordering:

1. The Merge: Any two adjacently ranked groups of tied objects of size p and q can be merged with pq simultaneous half-flips.2. The Split: Split any group of tied objects into two subgroups of adjacently ranked objects (of size r and s) with rs simultaneous half-flips.

» The Half-Flips Metric is defined as the minimum number of half-flips required to migrate from ranking A to B

21


National Defence

A Single Unified Theory

Lemma 1. The Half-Flips Measure satisifes the Kemeny-Snell axioms.

Lemma 2. The Kemeny-Snell metric formulation is equivalent to τX.

THEOREM. The Half-Flips Metric, the Kemeny-Snell Metric, and the τX rank correlation coefficient are equivalent representations of the unique measure that satisfies the Kemeny-Snell axioms.

22


National Defence

Max τX – A Simple Example

0 -1 -1 1 1 0 -1 1 1 1 0 1-1 -1 -1 0

0 1 1 1 1 0 -1 -3 1 3 0 1 1 3 1 0

0 ? ? ? ? 0 ? ? ? ? 0 ? ? ? ? 0

. MaxValue?

Ranking Matrix

5 Rankers (V, W, X, Y, Z ) and 4 Objects (a, b, c, d) Rankings (all equally weighted)

» V: c b a d

» W: a-c d b

» X: a-d b-c

» Y: d b a c

» Z: c-d a-b “Combined Input Matrix”:

» 1*V + 1*W + 1*X + 1*Y + 1*Z =

How do we find the solution ranking(s)?» If sign matrix represents a ranking then it is the unique solution (rare!)» Could search ranking space using brute force

23


National Defence

Our Simple Example

d,a-c-b

d,a-c,b

c-d,a-b

a-c-d,b

c,a-d-bd-c-b,a

d,c-b,a

d,c,a-b

c-d,b,a

d,c,b,a

c-d,a,b

d,c,a,b

c,d,a,b

c,a-d,b

c,b-d,a

c,d,a-b

a-d,c-b

d,a-b,ca,c-d,b

d,a,c-b

a-d,b,c

a-d,c,b

a,d,c-b

d,a,c,b

a,d,b,c

d-b,a-c

d,b,a-c

b,d,a-c

b,d-c,a

c-b,d,ab,c,d,a

d,b,c,a

d-b,c,a

b,d,c,a

b,d,a,c

d-b,a,c

d,b,a,c

c,d,b,a

c,b,d,a

d,a,b,ca,d,c,b

a-c,d,b

a,c,d,b

c,a,b,d

c,a-b,d

c,b,a,d

c,b,a-d

c,a,d,b

c,a,b-d

7

3

2

7

3

Input Rankings (5)

Solution Ranking (1) a-c-d, b

Minimum TotalDistance

=Minimum

Voter Unhappiness

24


National Defence

How Feasible IsThe Brute Force Approach?

Number of rankings of n objects with ties, Q(n)» Q(2) = 3 rankings» Q(3) = 13 rankings» Q(4) = 75 rankings» Q(5) = 541 rankings» …

Q(n) = no. of protons on earth for what value of n? Ans. O.A. Gross (1962) showed:

...442.12ln/12

)!()(

1

Cwhere

nCnQ

n

39

25


National Defence

A Cleverer Way?

Branch-and-bound algorithm devised» When implemented in FORTRAN on Pentium PC will solve for all

solutions– Up to 12 or so alternatives (any number of rankers): instantly

– Up to 15 or 20: seconds to minutes

– Up to 20 or 25: hours to days

– Over 30: basically hopeless

Heuristic algorithm developed for larger problems» Progressively applies B&B algorithm over a sliding window of 11

objects until it stabilizes

» Generates solutions correctly in virtually all cases tested

26


National Defence

DISCUSSION

Extension to Incomplete Rankings» CI Matrix doesn’t change if one adds (off diagonal) zeros – this naturally

represents the ‘no information’ situation between object pair» Accommodates various preferential voting schemes where objects can be

ranked against other blocks of objects but are left ‘unranked’ against each other (as opposed to ‘tied’)

Difference Between Ties and Indifference» A tie is a positive statement of agreement (‘strong’ ties), not a declaration of

indifference (‘weak’ ties)» Max τX method handles both

The Advantage of Dual Theories» Extension in one theory don’t necessarily have a parallel in the other » e.g. Incomplete rankings extension in τX domain has no parallel in Half-Flips

Metric domain

27


National Defence

Summary: How to UseOrdinal Methods

If development of single overall utility scale does not make sense, then you an ordinal technique may be a suitable choice

Avoid methods that involve arithmetic operations on the rank values themselves

Avoid methods that deal in strength of preference ‘Independence from irrelevant alternatives’ is the criterion most ordinal

methods run afoul of Arrow proved that one cannot construct an ordinal solution using the

argument: “If more individuals place a ahead of b, then the consensus or social choice will place a ahead of b”.

Determining consensus rankings by maximizing rank correlation using the τX statistic is a robust method» The problem-situation must be accepting of multiple solutions» It is computationally intensive to apply

28


National Defence

1 operational research division / division de la recherche opérationnelle défense nationale...

Documents