Why composite should be non compensatory

Giuseppe MundaUniversitat Autonoma de Barcelona

Dept.of Economics and Economic History

2

3

Step 1. Developing a theoretical

framework

Step 2. Selecting indicators

Step 3. Multivariate analysis

Step 4. Imputation of missing data

Step 5. Normalisation of data

Step 6. Weighting and aggregation

Step 7. Robustness and sensitivity

Step 8. Association with other variables

Step 9. Back to the details (indicators)

Step 10. Presentation and dissemination

Index construction- Steps[OECD-JRC Handbook]

4

Based on:

5

Structure of the presentation

Linear Aggregation Rule•Social Choice and Aggregation Rules•Non-Compensatory Aggregation Rules•Conclusions

6

Linear Aggregation Rule

7

Linear aggregation rules and preference independence

The variables X1, X2,..., Xn are mutually preferentially independent if every subset Y of these variables is preferentially independent of its complementary set of evaluators. The following theorem holds:

given the variables X1, X2, ..., Xn, an additive aggregation function exists if and only if these variables are mutually preferentially independent.

8

Preferential independence implies that the trade-off ratio between two variables is independent of the values of the n-2 other variables, i.e.

, 0x ySz

x, y Y ,z Z

Preferential independence is a very strong condition from both the operational andepistemologicalpoints of view.

9

The meaning of weights in linear aggregation rules

• “Greater weight should be given to components which are considered to be more significant in the context of the particular composite indicator”. (OECD, 2003, p. 10).

• Weights as symmetrical importance, that is "… if we have two non-equal numbers to construct a vector in R2, then it is preferable to place the greatest number in the position corresponding to the most important criterion." (Podinovskii, 1994, p. 241).

10

To use the compensatory approach in practice, we have to determine for each normalised variable, a mapping : RX ii which provides an interval scale of

measurement and to assess scaling constants in order to specify how the compensability should be accomplished, given the scales i between the different variables. Note that

the scaling constants appearing in the compensatory approach depend on the scales i ;

thus they do not characterise the intrinsic relative importance of the indicators (Roberts, 1979).

11

The aggregation of several criteria implies taking a position on the fundamental issue of compensability.

Compensability refers to the existence of trade-offs, i.e. the possibility of offsetting a disadvantage on some criteria by an advantage on another criterion.

E.g. in the construction of a composite indicator of environmental sustainability a compensatory logic (using equal weighting) would imply that one is willing to accept, let’s say, 10% more CO2 emissions in exchange of a 3% increase in GDP.

12

The implication is the existence of a theoretical inconsistency in the way weights are actually used and their real theoretical meaning.

For the weights to be interpreted as “importance coefficients ” (the greatest weight is placed on the most important “dimension”) non-compensatory aggregation procedures must be used to construct composite indicators.

Linear Aggregation is a correect rule iff

•Mutual preferential independence applies•Compensability is desirable•Weights are derived as trade-offs

14

Social Choice and Aggregation Rules

15

The Plurality Rule

Number of criteria 3 5 7 6 a a b c

16

Example (rearranged from Moulin, 1988, p. 228; 21 criteria and 4

alternatives )Objective: find best country

A first possibility: apply the plurality rule the country which is more often ranked in the first position is the winning one.

Country a is the best.

BUT Country a is also the one with the strongest opposition since 13 indicators put it into the last position!

# of indicators 3 5 7 6

1st position a a b c

2nd position b c d b

3rd position c b c d

4th position d d a a

This paradox was the starting step of Borda’s and Condorcet’s research at the end of the 18th century, but the plurality rule corresponds to the most common electoral system in the 21st century!!!

17

Borda approach

Rank a b c d Points

1st 8 7 6 0 3

2nd 0 9 5 7 2

3rd 0 5 10 6 1

4th 13 0 0 8 0

Frequency matrix (21 criteria 4 alternatives)

Jean-Charles, chevalier de Borda

18

Borda approach

Borda solution: bcad

Now: country b is the best, not a (as in the case of the plurality rule)

Rank a b c d Points

1st 8 7 6 0 3

2nd 0 9 5 7 2

3rd 0 5 10 6 1

4th 13 0 0 8 0

Frequency matrix (21 criteria 4 alternatives)

8 3 24

5 9 2 7 3 44

10 5 2 6 3 38

6 7 2 20

a

b

c

d

Borda score:

The plurality rule paradox has been solved.

19

Condorcet approach

0 8 8 8

13 0 10 21

13 11 0 14

13 0 7 0

a b c d

a

b

c

d

Outranking matrix (21 criteria 4 alternatives)

Jean-Antoine-Nicolas de Caritat, Marquis de Condorcet

20

Condorcet approach

• For each pair of countries a concordance index is computed by counting how many individual indicators are in favour of each country (e.g. b better than a 13 times).

• “constant sum property” in the outranking matrix (13+8=21)

0 8 8 8

13 0 10 21

13 11 0 14

13 0 7 0

a b c d

a

b

c

d

Outranking matrix (21 criteria 4 alternatives)

21

Condorcet approach

• Pairs with concordance index > 50% of the indicators are selected.

• Majority threshold = 11 (i.e. a number of individual indicators bigger than the 50% of the indicators considered)

• Pairs with a concordance index ≥ 11: bPa= 13, bPd=21(=always), cPa=13, cPb=11, cPd=14, dPa=13.

0 8 8 8

13 0 10 21

13 11 0 14

13 0 7 0

a b c d

a

b

c

d

Condorcet solution: c b d aYet, the derivation of a Condorcet ranking may sometimes be a long and complex computation process.

Outranking matrix (21 criteria 4 alternatives)

22

Which approach should one prefer?

Both Borda and Condorcet approaches solve the plurality rule paradox. However, the solutions offered are different.

Borda solution: b c a dCondorcet solution: c b d a

In the framework of composite indicators, can we choose between Borda and Condorcet on some theoretical and/or practical grounds?

23

An Original Condorcet’s Numerical Example

Number of criteria 23 17 2 10 8 a b b c c b c a a b c a c b a

24

Number of indicators

23 17 2 10 8

1st a b b c c

2nd b c a a b

3rd c a c b a

Rank a b c Points

1st 23 19 18 2

2nd 12 31 17 1

3rd 25 10 25 0

0 33 25

27 0 42

35 18 0

a b c

a

b

c

Example with 60 indicators

[Condorcet, 1785]

Frequency matrix

Outranking matrix

58, 69, 53a b c

concordance threshold =31 aPb, bPc and cPa (cycle)???

Borda approach Condorcet approach

25

From this example we might conclude that the Borda rule (or any scoring rule) is more effective since a country is always selected while the Condorcet one sometimes leads to an irreducible indecisiveness.

However Borda rules have other drawbacks, too

26

• Both social choice literature and multi-criteria decision theory agree that whenever the majority rule can be operationalized, it should be applied. However, the majority rule often produces undesirable intransitivities, thus “more limited ambitions are compulsory. The next highest ambition for an aggregation algorithm is to be Condorcet” (Arrow and Raynaud, 1986, p. 77).

27

Non-Compensatory Aggregation Rules

28

Condorcet approach

Basic problem: presence of cycles, i.e. aPb, bPc and cPa

The probability of obtaining a cycle increases with both N (indicators) and M (countries) [Fishburn (1973, p. 95)]

29

Condorcet approach

Condorcet himself was aware of this problem (he built examples to explain it) and he was even close to find a consistent rule able to rank any number of alternatives when cycles are present…

… but Maximilien de Robespierre

30

Condorcet approachBasic problem: presence of cycles, i.e. aPb, bPc and

cPa

Furter attempts made by Kemeny (1959) and by Young and Levenglick (1978) … led to:

Condorcet-Kemeny-Young-Levenglick (C-K-Y-L) ranking procedure

31

C-K-Y-L ranking procedure

Main methodological foundation: maximum likelihood concept.

The maximum likelihood principle selects as a final ranking the one with the maximum pair-wise support.

What does this mean and how does it work?

Indic. GDP Unemp. Rate

Solid wastes

Income dispar.

Crime rate

Country

A 25,000 0.15 0.4 9.2 40

B 45,000 0.10 0.7 13.2 52

C 20,000 0.08 0.35 5.3 80

weights .166 .166 0.333 .166 .166AB = 0.333+0.166+0.166=0.666

BA = 0.166+0.166=0.333

AC = 0.166+0.166=0.333

CA = 0.166+0.333+0.166=0.666

BC = 0.166+0.166=0.333

CB = 0.166+0.333+0.166=0.666

C-K-Y-L ranking procedure

Indic. GDP Unemp. Rate

Solid wastes

Income dispar.

Crime rate

Country

A 25,000 0.15 0.4 9.2 40

B 45,000 0.10 0.7 13.2 52

C 20,000 0.08 0.35 5.3 80

weights .166 .166 0.333 .166 .166

A

B

C

A B C

0 0.666 0.333

0.333 0 0.333

0.666 0.666 0

C-K-Y-L ranking procedure

AB = 0.333+0.166+0.166=0.666

BA = 0.166+0.166=0.333

AC = 0.166+0.166=0.333

CA = 0.166+0.333+0.166=0.666

BC = 0.166+0.166=0.333

CB = 0.166+0.333+0.166=0.666

Indic. GDP Unemp. Rate

Solid wastes

Income dispar.

Crime rate

Country

A 25,000 0.15 0.4 9.2 40

B 45,000 0.10 0.7 13.2 52

C 20,000 0.08 0.35 5.3 80

weights .166 .166 0.333 .166 .166

A

B

C

A B C

0 0.666 0.333

0.333 0 0.333

0.666 0.666 0

ABC = 0.666 + 0.333 + 0.333 = 1.333

BCA = 0.333 + 0.666 + 0.333 = 1.333

CAB = 0.666 + 0.666 + 0.666 = 2

ACB = 0.333 + 0.666 + 0.666 = 1.666

BAC = 0.333 + 0.333 + 0.333 = 1

CBA = 0.666 + 0.333 + 0.666 = 1.666

C-K-Y-L ranking procedure

35

The Computational problem

The only drawback of this aggregation method is the difficulty in computing it when the number of candidates grows.

With only 10 countries 10! = 3,628,800 permutations (instead of 3!=6 of the example)

To solve this problem of course there is a need to use numerical algorithms

C-K-Y-L ranking procedure

36

Conclusions

Compensabiliity is a fundamental concept in choosing an

aggregation rule

38

Aggregation ESI rank

with LIN rank with NCMC

Change in Rank

Azerbaijan 99 61 38

Spain 76 45 31

Nigeria 98 69 29 South Africa 93 68 25 Im

prov

emen

t

Burundi 130 107 23

Indonesia 75 114 39

Armenia 44 79 35

Ecuador 51 78 27

Turkey 91 115 24

Det

erio

ratio

n

Sri Lanka 79 101 22

Average change over 146 countries 8

Table 3. ESI rankings obtained by linear aggregation (LIN) and non-compensatory rule (NCMC): countries that largely improve or worsen their rank position

39

0

5

10

15

20

25

Lom

bard

y (I

T)

Mad

rid

To

sca

ny (

IT)

Nav

arra

Cat

alo

nia

Ven

eto

(IT

)

Rio

ja

Bal

ear

ic Is

land

s

Pai

s V

asco

Ara

gon

Val

enc

ia

Mur

cia

Sic

ily (

IT)

Can

tab

ria

Cas

tilla

y L

eon

And

alu

cia

Cas

tilla

la M

anch

a

Can

ary

Isla

nds

Krit

i (G

R)

Atti

ki (

GR

)

Ast

uria

s

Ext

rem

adu

ra

Ga

licia

St.

Ella

da (

GR

)

Th

essa

lia (

GR

)

Mu

lti-

Cri

teri

a R

ank

(30

sce

nar

ia)

Figure 1: Multi-criteria based ranking1

40

0

5

10

15

20

25

Na

varr

a

Ma

drid

Lo

mb

ard

y (I

T)

Pa

is V

asc

o

To

sca

ny

(IT

)

Ve

ne

to (

IT)

Rio

ja

Att

iki (

GR

)

Krit

i (G

R)

Ara

go

n

Ext

rem

ad

ura

Ca

talo

nia

Sic

ily (

IT)

Mu

rcia

Ba

lea

ric I

sla

nd

s

Va

len

cia

Th

ess

alia

(G

R)

Ca

na

ry I

sla

nd

s

Ca

still

a y

Le

on

Ca

nta

bria

An

da

luci

a

Ast

uria

s

St.

Ella

da

(G

R)

Ca

still

a la

Ma

nch

a

Ga

licia

Lin

ea

r R

an

k(3

0 s

ce

na

ria

)

Figure 2: Additive (linear) based ranking1

Using Borda too

The KEI example …

42

Methodological scenarios for the development of the KEI composite

Aggregation function Additive

(linear) Multiplicative (geometric averaging)

Non-compensatory multi-criteria analysis

All (total 29) Sub-dimensions included One-at-a-time excluded

All (total 7) Dimensions included One-at-a-time excluded

Preserved Pillar Structure

Not preserved z-scores Min-max Normalisation Raw data Factor analysis Equal weighting Weighting Data envelopment analysis

The frequency matrix (Borda type approach) of a country’s rank in each of the seven dimensions and the overall KEI was calculated across the ~2,000 scenarios~2,000 scenarios.

43

Ran

k 1

Ran

k 2

Ran

k 3

Ran

k 4

Ran

k 5

Ran

k 6

Ran

k 7

Ran

k 8

Ran

k 9

Ran

k 10

Ran

k 11

Ran

k 12

Ran

k 13

Ran

k 14

Ran

k 15

Ran

k 16

Ran

k 17

Ran

k 18

Ran

k 19

Ran

k 20

Ran

k 21

Ran

k 22

Ran

k 23

Ran

k 24

Ran

k 25

Ran

k 26

Ran

k 27

Ran

k 28

Ran

k 29

Sweden 54 46Denmark 55 30 14Luxembourg 36 4 14 25 4 7 7 4Finland 18 23 29 9 11 11USA 11 32 2 4 39 9 4Japan 4 7 18 32 36 4UK 2 5 16 38 39Netherlands 86 4 4 7Ireland 4 61 14 4 9 9Austria 18 50 18 7 7Belgium 11 4 11 57 16 2France 4 14 18 11 54EU15 4 57 39EU25 4 4 14 32 39 7Germany 7 79 4 7 4Slovenia 7 41 38 14Estonia 4 36 25 21 11 4Malta 7 13 9 21 23 27Cyprus 36 7 4 23 23 7Spain 4 4 32 25 29 7Czech. Rep. 4 7 30 39 5 7 7Latvia 20 36 11 21 7 5Italy 29 18 9 29 9 7Greece 4 4 4 29 18 21 7 14Lithuania 4 41 13 32 11Hungary 2 13 13 57 2 14Portugal 4 4 7 11 61 14Slovakia 4 7 18 71Poland 100Legend:Frequency lower 15%Frequency between 15 and 30%Frequency between 30 and 50%Frequency greater than 50%

Knowledge Economy Index

44

REFERENCES• Borda J.C. de (1784) – Mémoire sur les élections au scrutin, in

Histoire de l’ Académie Royale des Sciences, Paris.• Brand, D.A., Saisana, M., Rynn, L.A., Pennoni, F., Lowenfels, A.B.

(2007) Comparative Analysis of Alcohol Control Policies in 30 Countries, PLoS Medicine, 0759 April 2007, Volume 4, Issue 4, e151, p. 0752-0759, www.plosmedicine.org

• Condorcet, Marquis de (1785) – Essai sur l’application de l’analyse à la probabilité des décisions rendues à la probabilité des voix, De l’ Imprimerie Royale, Paris.

• Fishburn P.C. (1973) – The theory of social choice, Princeton University Press, Princeton.

• Fishburn P.C. (1984) – Discrete mathematics in voting and group choice, SIAM Journal of Algebraic and Discrete Methods, 5, pp. 263-275.

45

• Munda G. (1995), Multicriteria evaluation in a fuzzy environment, Physica-Verlag, Contributions to Economics Series, Heidelberg.

• Munda G. (2004) – “Social multi-criteria evaluation (SMCE)”: methodological foundations and operational consequences, European Journal of Operational Research, vol. 158/3, pp. 662-677.

• Munda G. (2005a) – Multi-Criteria Decision Analysis and Sustainable Development, in J. Figueira, S. Greco and M. Ehrgott (eds.) –Multiple-criteria decision analysis. State of the art surveys, Springer International Series in Operations Research and Management Science, New York, pp. 953-986.

• Munda G. (2005b) – “Measuring sustainability”: a multi-criterion framework, Environment, Development and Sustainability Vol 7, No. 1, pp. 117-134.

• Munda G. and Nardo M. (2005) - Constructing Consistent Composite Indicators: the Issue of Weights, EUR 21834 EN, Joint Research Centre, Ispra.

• Munda G. and Nardo M. (2007) - Non-compensatory/Non-Linear composite indicators for ranking countries: a defensible setting, Forthcoming in Applied Economics.

• Munda G. (2008) -  Social multi-criteria evaluation for a sustainable economy, Springer-Verlag, Heidelberg, New York.