CONCEPTUAL ISSUES CONCEPTUAL ISSUES IN CONSTRUCTING IN CONSTRUCTING

COMPOSITE INDICESCOMPOSITE INDICESNadia FarrugiaNadia Farrugia

Department of Economics, University of MaltaDepartment of Economics, University of Malta

Paper prepared for the Paper prepared for the INTERNATIONAL CONFERENCE ON SMALL STATES AND INTERNATIONAL CONFERENCE ON SMALL STATES AND

ECONOMIC RESILIENCE ECONOMIC RESILIENCE Organised byOrganised by

The Islands and Small States Institute The Islands and Small States Institute of the Foundation for International Studies at the University of of the Foundation for International Studies at the University of

MaltaMaltaand the and the

Commonwealth Secretariat, LondonCommonwealth Secretariat, LondonValletta, MaltaValletta, Malta

23 - 25 April 200723 - 25 April 2007

Presentation OutlinePresentation Outline IntroductionIntroduction Desirable Attributes for Developing Desirable Attributes for Developing

Statistics and Composite IndicesStatistics and Composite Indices Main Conceptual IssuesMain Conceptual Issues

Indicator SelectionIndicator Selection Dealing with Missing DataDealing with Missing Data NormalisationNormalisation Weighting and AggregationWeighting and Aggregation Testing and Reviewing the Results ObtainedTesting and Reviewing the Results Obtained

ConclusionConclusion

IntroductionIntroduction

DefinitionDefinition

A composite index, A composite index, is a weighted (linear) aggregation of a is a weighted (linear) aggregation of a

number of variablesnumber of variables wwjj is a weight, with 0≤ is a weight, with 0≤wwjj≤1 and ∑≤1 and ∑wwjj=1=1

XXcjcj is the variable of country is the variable of country cc in in dimension dimension jj

for any country for any country cc the number of policy the number of policy variables are equal to j=1,…,m.variables are equal to j=1,…,m.

1

m

c j cjj

I w X

UsesUses Describe complex phenomena in a single Describe complex phenomena in a single

indicatorindicator Cross-national comparisons of country Cross-national comparisons of country

performanceperformance Benchmarking exercisesBenchmarking exercises General trendsGeneral trends Policy priorities and performance targetsPolicy priorities and performance targets Several examples of renowned composite Several examples of renowned composite

indices, stock market indices, RPI, GDPindices, stock market indices, RPI, GDP

StrengthsStrengths

Summarises complex and multi-Summarises complex and multi-dimensional issuesdimensional issues

Helps set the direction for Helps set the direction for policymakers and to focus the policymakers and to focus the discussiondiscussion

Supports decision makingSupports decision making Helps disseminate informationHelps disseminate information Make stakeholders and the public more Make stakeholders and the public more

aware of certain problemsaware of certain problems Generates academic discussionGenerates academic discussion

WeaknessesWeaknesses

Subjectivity in computationSubjectivity in computation May send misleading policy May send misleading policy

messages and can easily be misusedmessages and can easily be misused May conceal divergences between May conceal divergences between

different components different components Increase difficulty in identifying Increase difficulty in identifying

proper remedial actionproper remedial action Measurement problemsMeasurement problems

Desirable Desirable AttributesAttributes

Quality FrameworksQuality Frameworks IMF – Data Quality Assurance FrameworkIMF – Data Quality Assurance Framework Eurostat FrameworkEurostat Framework OECD – Quality Framework and OECD – Quality Framework and

Guidelines for OECD StatisticsGuidelines for OECD Statistics Booysen – Dimensions for Classifying and Booysen – Dimensions for Classifying and

Evaluating Development IndicatorsEvaluating Development Indicators Briguglio – Desirable Characteristics for Briguglio – Desirable Characteristics for

Developing Vulnerability IndicesDeveloping Vulnerability Indices JRC-OECD – Handbook on Constructing JRC-OECD – Handbook on Constructing

Composite IndicatorsComposite Indicators

Desirable Attributes of Desirable Attributes of Composite IndicesComposite Indices

1.1. AccuracyAccuracy2.2. Simplicity and Ease of ComprehensionSimplicity and Ease of Comprehension3.3. Methodological SoundnessMethodological Soundness4.4. Suitability for International and Suitability for International and

Temporal ComparisonsTemporal Comparisons5.5. TransparencyTransparency6.6. AccessibilityAccessibility7.7. Timeliness and FrequencyTimeliness and Frequency8.8. FlexibilityFlexibility

Main Conceptual Main Conceptual IssuesIssues

Main Conceptual IssuesMain Conceptual Issues

Indicator SelectionIndicator Selection Dealing with Missing DataDealing with Missing Data NormalisationNormalisation Weighting and AggregationWeighting and Aggregation Testing and Reviewing the Results Testing and Reviewing the Results

ObtainedObtained

Indicator SelectionIndicator Selection

Define the conceptDefine the concept Select indicators which satisfy desirable Select indicators which satisfy desirable

attributesattributes Do not select variables which beg the Do not select variables which beg the

questionquestion Draft an initial indicator set and review the Draft an initial indicator set and review the

available dataavailable data Keep the number of variables as small as Keep the number of variables as small as

possible but not fewer than necessary possible but not fewer than necessary (PCA, FA)(PCA, FA)

Indicator Selection Indicator Selection (Cont.)(Cont.)

Check for correlation between the Check for correlation between the variables or sub-indices (rank variables or sub-indices (rank correlation test, Cronbach correlation test, Cronbach coefficient alpha, cluster and coefficient alpha, cluster and discriminant analysis)discriminant analysis)

Review the indicators selected and Review the indicators selected and seek external advice and opinionseek external advice and opinion

Dealing with Missing Dealing with Missing DataData

Exclude the country from the Exclude the country from the analysisanalysis

Imputation methods: Single or Imputation methods: Single or MultipleMultiple

Single Imputation Single Imputation MethodsMethods

Case deletionCase deletion Mean/median/mode estimationMean/median/mode estimation Hot deck imputationHot deck imputation Regression imputationRegression imputation

Multiple Imputation Multiple Imputation MethodsMethods

Regression MethodRegression Method Propensity Score MethodPropensity Score Method Markov Chain Monte Carlo Markov Chain Monte Carlo

AlgorithmAlgorithm

Quantifying Qualitative Quantifying Qualitative DataData

Using a mapping (Likert) scaleUsing a mapping (Likert) scale Optimal spread of the scaleOptimal spread of the scale Permits non-linearityPermits non-linearity Defect relates to subjectivityDefect relates to subjectivity

NormalisationNormalisation

RescalingRescaling Standardisation (or z-scores)Standardisation (or z-scores) Percentage differences over Percentage differences over

previous yearsprevious years RatiosRatios RankingsRankings Measuring the relative position vis-à-Measuring the relative position vis-à-

vis a specified pointvis a specified point

Weighting and Weighting and AggregationAggregation

Equal WeightingEqual Weighting Differential WeightingDifferential Weighting Country-Specific or Indicator-Country-Specific or Indicator-

Specific WeightsSpecific Weights Weights Over Time: Constant or Weights Over Time: Constant or

ChangingChanging

Differential WeightingDifferential Weighting

Weights Reflecting the Statistical Weights Reflecting the Statistical Quality of the DataQuality of the Data

Stochastic WeightsStochastic Weights Participatory MethodsParticipatory Methods Precautionary PrinciplePrecautionary Principle Regression MethodRegression Method Benefit-of-the-Doubt Weighting Benefit-of-the-Doubt Weighting

SystemSystem

AggregationAggregation

Linear or geometric aggregationLinear or geometric aggregation Aggregation methods and weighting Aggregation methods and weighting

systemssystems Non-compensatory multi-criteria Non-compensatory multi-criteria

aggregationaggregation

Testing and Reviewing the Testing and Reviewing the Results ObtainedResults Obtained

Uncertainty and Sensitivity AnalysisUncertainty and Sensitivity Analysis OutliersOutliers Expert OpinionExpert Opinion Analysing the Results ObtainedAnalysing the Results Obtained

ConclusionConclusion

ConclusionConclusion

Composite indices have their pros and cons.Composite indices have their pros and cons. Hard to imagine that the debate on the use of Hard to imagine that the debate on the use of

composite indices will be ever settled.composite indices will be ever settled. Composite indices should be identified for Composite indices should be identified for

what they are.what they are. However, their importance should not be However, their importance should not be

undermined. undermined. Provided they are built on sound Provided they are built on sound

methodological considerations they are very methodological considerations they are very useful to portray complex phenomena in a useful to portray complex phenomena in a simple manner.simple manner.

Thank you!Thank you!

farend@onvol.netfarend@onvol.net