how to use and abuse mcda and consensus support methods
DESCRIPTION
How to Use and Abuse MCDA and Consensus Support Methods. Dave Mason Operational Research Division National Defence Headquarters Ottawa, Canada. Outline. Definitions and applications Cardinal utility methods When they are applicable, desired attributes How to abuse - PowerPoint PPT PresentationTRANSCRIPT
1
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
How to Use and Abuse MCDA and Consensus Support Methods
Dave Mason
Operational Research Division
National Defence Headquarters
Ottawa, Canada
2
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
Outline
Definitions and applications Cardinal utility methods
» When they are applicable, desired attributes» How to abuse» Summary: How To Use Cardinal Utility Methods
Ordinal methods» Fundamental concepts (Arrow’s Theorem, Voter’s Paradox) » How to abuse common ordinal methods» Heading towards a solution to the Consensus Ranking Problem
– Solution concept: Maximization of rank correlation– Existing statistics: Kendall’s τb, Kemeny/Snell distance– Unified theories, and a new rank correlation statistic: τX
» Summary: How to Use Ordinal Methods
3
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
MCDA andConsensus Support Methods
Multi-Criteria Decision Analysis Methods» Generally, any scheme that evaluates a list of
alternatives across a set of assessment criteria to determine a winner (or final ordering)
– Weighted sum methods are the most common form
Consensus Support Methods» Any methodology that attempts to resolve value
judgements of a group of individuals into a single collective set of judgements
4
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
Applicable Decision Environments
Applicable types of decision environments:» Evaluation of contract bids» Ranking and selection of personnel» Prioritization of an organization’s work program» Inevitable stream of short-fuse ‘Tiger Team’ approaches» …
Generally, involves one of following types of decisions:» 1. Single Choice - selecting the winning alternative » 2. Prioritization - put the list of alternatives in a rank ordering» (3. Resource Allocation – assign resources to each alternative)
Methods are applied after the conduct of, not in lieu of a comprehensive cost-effectiveness analyses
5
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
A Single Overall Measure of Utility?
Key question the analyst must ask: “Is the development of a single overall measure of utility reasonable in this case?”» A single measure may defy definition
» Is a potentially dangerous over-simplification
» But may be a necessary evil to enable an answer to be reached
YES. Then a cardinal utility method may be useful
NO. Then an ordinal methods approach may be a useful alternative
6
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
CARDINAL UTILITYMETHODS
Generally the broad class of weighted sum methods are most commonly employed» Assign weights to key criteria
» Score the alternatives on each criteria
» Arithmetically calculate the weighted sum
Numerous approaches with enhanced mathematical complexity exist » Analytical Hierarchy Process (AHP)
» …
7
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
(Clear glass doors)(Clear glass doors)
????“Black
Box”Method
“BlackBox”
Method
Qualitative and Quantitative InputsQualitative and Quantitative Inputs
Final winner or orderingFinal winner or ordering
Desirable Attributes: 1. Transparency of Method
8
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
Desirable Attributes: 2. Reasonable Fidelity
Complex enough to adequately represent the key dimensions of overall utility, » “KISS” is fine, but don’t go too far
More Sophisticated Mathematics = A Better Solution?» Not necessarily
Oversold obscure methods with gratuitous mathematical complexity = Snake Oil?» You bet!
9
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
How To Abuse WeightedSum Methods
Weight:
Sports CarSedan
Mini-VanSUV
3
45
108
Cost
Fun
Style
Relia
bilit
y
Versa
tility
Total2
61087
3
10327
2
10446
2
61087
86727585
($50K)($30K)($35K)($40K)
Add extra criteriathat boost your alternative
Shave the weightsof criteria that hurt
your alternative
Compress the scale of criteria that hurt your alternative
10
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
How To Abuse WeightedSum Methods
Weight:
Sports CarSedan
Mini-VanSUV
3
45
108
Fun
Style
Relia
bilit
y
Versa
tility
Utility/K$2
61087
3
10327
2
10446
/50 =/30 =/35 =/40 =
1.481.731.711.78
74526071
11
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
The Value Of Weighted Sum Methods
Question: In a weighted sum application, can I fiddle with the weights and scores in subtle ways to make just about any alternative the winner?» Answer: Yeah, pretty much …
Question: I guess that means weighted sum methods are not very useful?» Answer: On the contrary, they are transparent and flexible … just
make sure you use them properly
12
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
Summary: How to UseCardinal Utility Methods
Simple weighted sum, multi-criteria utility methods are fine» MCDA methods are not a substitute for comprehensive cost-
effectiveness analysis» Conduct a (pairwise) sensibility check on criteria utility tradeoffs» Don’t treat cost as another criterion, but as an overall utility divisor» Do a complete sensitivity analysis on both weights and scores to
locate the decision cross-over points Add mathematical complexity if warranted, but there is no
positive value to using unnecessarily complex methods» For example, AHP calculates weights using a prescribed, vague
qualitative scale in a laborious pair-wise assessment process where the DM is permitted to be inconsistent
Horses for courses: Fit the method to your problem
13
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
ORDINAL METHODS
If development of a single overall utility scale is not sensible, then ordinal methods may be a viable option» If there is more than one DM (or more than one criterion being
assessed) the problem is one of ‘social choice’, depending on the societal structure assumed
Nobel laureate Kenneth Arrow’s research from 1953 forms the logic foundation» “It … makes no sense to add the utility of one individual, a psychic
magnitude in his mind, with the utility of another individual.”» Coke or Pepsi? … Strength of preference, or simply preference?» “A utility function whose significance lies entirely in its ordinal
properties is superfluous. If we are concerned with ordinal properties it seems better to represent these directly.”
14
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
Kenneth Arrow andSocial Welfare Functions
A ‘Social Welfare Function’ gives the ordering of alternative social states given the orderings by the individuals in the society
Arrow imposed several ‘apparently reasonable’ conditions:1. Universality – Individuals can express any possible ordering2. ‘Welfare’ Not ‘Illfare’ - social ordering must respond positively (or at
least non-negatively) to alterations in individual values3. Independence From Irrelevant Alternatives – The societal choice
amongst a set of alternatives should be unaffected if other alternatives are added or removed
4. Citizens’ Sovereignty – the social welfare function is not to be imposed (all outcomes are possible)
5. Non-Dictatorship – whenever the dictator prefers x over y, so does society
Arrow’s Theorem: You can’t have all five
15
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
Arrow’s Third Condition & The Voter’s Paradox
The Voter’s Paradox:» Voter 1: a b c
» Voter 2: b c a
» Voter 3: c a b
In this example no winner is declarable, yet if you remove any alternative from consideration a clear winner emerges
Upshot: One cannot construct an ordinal solution using the argument: “If more individuals place a ahead of b, then the consensus or social choice will place a ahead of b”.
16
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
How to Abuse Ordinal Methods
Borda Count (minimum sum of rank values) is commonly employed» Simple example: three rankers, two alternatives
– Ranker A: 1. Coke, 2. Pepsi– Ranker B: 1. Coke, 2. Pepsi– Ranker C: 1. Pepsi, 2. Coke
» Coke wins … any and every rational ranking scheme will agree Ranker C insists on including all contending colas (RC, Kik, …)
» Ranker C tactically and artificially ranks Coke low to benefit his choice:– Ranker A: 1. Coke, 2. Pepsi, T3. Kik Cola & RC Cola– Ranker B: 1. Coke, 2. Pepsi, T3. Kik Cola & RC Cola– Ranker C: 1. Pepsi, T2. Kik Cola & RC Cola, 4. Coke
The Borda Count winner now … Pepsi!! The Problem: Borda count fails any reasonable interpretation of the
‘independence from irrelevant alternatives’ condition The Lesson: Don’t conduct arithmetic operations on rank values
17
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
A Proposed Solution Concept to the Consensus Ranking
Problem
“Given m rankings of n objects, what ranking best represents the consensus opinion?”
In its most general form:» Ties are permitted» Incomplete rankings are permitted» Relative importance of rankers is reflected (usually in form of numerical
weights) Solution Concept Employed: Select a measure of agreement
between pairs of rankings and select that ranking(s) which maximizes overall average agreement.
Possible measures:» Rank correlation coefficient» Distance measure operating on the space of all rankings of n objects
18
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
One Measure of Agreement:Kendall’s Tau-b (1948)
1 if object i ranked ahead of object j
aij = -1 if i ranked behind j
0 if i ranked tied with j or if i=j
n
jiij
n
jiij
ij
n
i
n
jij
b
ba
ba
BA
1,
2
1,
2
1 1),(
Strengths: » If ties are not allowed, it is equivalent to the number of interchanges
of adjacent objects required to convert one ranking into the other.
Weaknesses: » Not well defined when ties are allowed
» The equivalent distance measure fails the triangle inequality test.
19
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
Another Agreement Measure:Kemeny-Snell Distance
(1962)
Axiom 1.1. d(A,B) ≥ 0, with equality holding if and only if A and B are the same ranking.
Axiom 1.2. d(A,B) = d(B,A).
Axiom 1.3. d(A,B) + d(B,C) ≥ d(A,C), and equality holds if and only if B is ‘between’ A and C.
Axiom 2. If A' results from A by a permutation of the objects, and B' results from B by the same permutation, then d(A,B) = d(A',B').
Axiom 3. If A and B agree except for a set S of k objects, which is a segment of both, then d(A,B) may be computed as if these k objects were the only objects being ranked.
Axiom 4. The minimum positive distance is 1.
K&S defined a set of axioms that any reasonable distance measure operating on the set of all orderings of n objects should satisfy …
n
i
n
jijij baBAd
1 12
1 ||),(
They proved that there is a unique distance metric that satsifies these axioms, and that the following is an implemen-tation of that unique measure (using Kendall’s def’n for aij)
20
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
Two Other Proposed Measures of Agreement
The Tau-X Rank Correlation Coefficient:1 if object i ranked ahead of OR TIED WITH object j
aij = -1 if i ranked behind j
0 if i=j )1(),( 1 1
nn
ba
BAij
n
i
n
jij
X
The Half-Flips Metric:» Define Elemental Transformations of any ordering:
1. The Merge: Any two adjacently ranked groups of tied objects of size p and q can be merged with pq simultaneous half-flips.2. The Split: Split any group of tied objects into two subgroups of adjacently ranked objects (of size r and s) with rs simultaneous half-flips.
» The Half-Flips Metric is defined as the minimum number of half-flips required to migrate from ranking A to B
21
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
A Single Unified Theory
Lemma 1. The Half-Flips Measure satisifes the Kemeny-Snell axioms.
Lemma 2. The Kemeny-Snell metric formulation is equivalent to τX.
THEOREM. The Half-Flips Metric, the Kemeny-Snell Metric, and the τX rank correlation coefficient are equivalent representations of the unique measure that satisfies the Kemeny-Snell axioms.
22
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
Max τX – A Simple Example
0 -1 -1 1 1 0 -1 1 1 1 0 1-1 -1 -1 0
0 1 1 1 1 0 -1 -3 1 3 0 1 1 3 1 0
0 ? ? ? ? 0 ? ? ? ? 0 ? ? ? ? 0
. MaxValue?
Ranking Matrix
5 Rankers (V, W, X, Y, Z ) and 4 Objects (a, b, c, d) Rankings (all equally weighted)
» V: c b a d
» W: a-c d b
» X: a-d b-c
» Y: d b a c
» Z: c-d a-b “Combined Input Matrix”:
» 1*V + 1*W + 1*X + 1*Y + 1*Z =
How do we find the solution ranking(s)?» If sign matrix represents a ranking then it is the unique solution (rare!)» Could search ranking space using brute force
23
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
Our Simple Example
d,a-c-b
d,a-c,b
c-d,a-b
a-c-d,b
c,a-d-bd-c-b,a
d,c-b,a
d,c,a-b
c-d,b,a
d,c,b,a
c-d,a,b
d,c,a,b
c,d,a,b
c,a-d,b
c,b-d,a
c,d,a-b
a-d,c-b
d,a-b,ca,c-d,b
d,a,c-b
a-d,b,c
a-d,c,b
a,d,c-b
d,a,c,b
a,d,b,c
d-b,a-c
d,b,a-c
b,d,a-c
b,d-c,a
c-b,d,ab,c,d,a
d,b,c,a
d-b,c,a
b,d,c,a
b,d,a,c
d-b,a,c
d,b,a,c
c,d,b,a
c,b,d,a
d,a,b,ca,d,c,b
a-c,d,b
a,c,d,b
c,a,b,d
c,a-b,d
c,b,a,d
c,b,a-d
c,a,d,b
c,a,b-d
7
3
2
7
3
Input Rankings (5)
Solution Ranking (1) a-c-d, b
Minimum TotalDistance
=Minimum
Voter Unhappiness
24
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
How Feasible IsThe Brute Force Approach?
Number of rankings of n objects with ties, Q(n)» Q(2) = 3 rankings» Q(3) = 13 rankings» Q(4) = 75 rankings» Q(5) = 541 rankings» …
Q(n) = no. of protons on earth for what value of n? Ans. O.A. Gross (1962) showed:
...442.12ln/12
)!()(
1
Cwhere
nCnQ
n
39
25
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
A Cleverer Way?
Branch-and-bound algorithm devised» When implemented in FORTRAN on Pentium PC will solve for all
solutions– Up to 12 or so alternatives (any number of rankers): instantly
– Up to 15 or 20: seconds to minutes
– Up to 20 or 25: hours to days
– Over 30: basically hopeless
Heuristic algorithm developed for larger problems» Progressively applies B&B algorithm over a sliding window of 11
objects until it stabilizes
» Generates solutions correctly in virtually all cases tested
26
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
DISCUSSION
Extension to Incomplete Rankings» CI Matrix doesn’t change if one adds (off diagonal) zeros – this naturally
represents the ‘no information’ situation between object pair» Accommodates various preferential voting schemes where objects can be
ranked against other blocks of objects but are left ‘unranked’ against each other (as opposed to ‘tied’)
Difference Between Ties and Indifference» A tie is a positive statement of agreement (‘strong’ ties), not a declaration of
indifference (‘weak’ ties)» Max τX method handles both
The Advantage of Dual Theories» Extension in one theory don’t necessarily have a parallel in the other » e.g. Incomplete rankings extension in τX domain has no parallel in Half-Flips
Metric domain
27
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence
Summary: How to UseOrdinal Methods
If development of single overall utility scale does not make sense, then you an ordinal technique may be a suitable choice
Avoid methods that involve arithmetic operations on the rank values themselves
Avoid methods that deal in strength of preference ‘Independence from irrelevant alternatives’ is the criterion most ordinal
methods run afoul of Arrow proved that one cannot construct an ordinal solution using the
argument: “If more individuals place a ahead of b, then the consensus or social choice will place a ahead of b”.
Determining consensus rankings by maximizing rank correlation using the τX statistic is a robust method» The problem-situation must be accepting of multiple solutions» It is computationally intensive to apply
28
Operational Research Division / Division de la Recherche Opérationnelle Operational Research Division / Division de la Recherche Opérationnelle Défensenationale
National Defence