lessons learnt from the use of prosperity indicators in policy making: towards community-generated...

Session 3a, 25 November 2015 eChallenges e-2015 Copyright 2015 Policy Compass

Lessons Learnt from the Use of Prosperity Indicators in Policy Making: Towards Community-Generated

Indicators

Ourania Markaki,Panagiotis Kokkinakos, Sotirios Koussouris, Costas Koutras,

John Psarras

National Technical University of AthensGreece


Prosperity Indicators and Policy Making

• With the advent of evidence-based policy making and the growing demand for government accountability, there is a heavy exploitation of indicators in the public policy arena.

• ‘40s: Evaluation of the US economy in terms of the ‘Monthly Economic Indicators’

• ‘60s: ‘social indicators’ movement, an explosion of indicators for social change• The idea of using social indicators passed gradually to the large international

organizations, e.g. the UN, OECD, etc.• early ‘90s: ‘community indicators’ movement’, motivated by the global

questions on environmental matters• Today: the Web 2.0 and the Open Data Movement open a new arena of

experimentation with social indicators.

An indicator is regularly conceived as a sort of ‘statistical measure’ that can adequately capture crucial aspects of a (social) phenomenon that should be

monitored, in particular when a specific policy measure is enforced to affect it


On the definition of Prosperity

• Prosperity is frequently used as a synonym to ‘welfare’, ‘well-being’ or ‘quality of life’ (=the product of the interplay among social, health, economic and environmental conditions affecting human and social development).

• Well-being (or prosperity) does not necessarily equal ‘economic growth’.

• Infinite growth, and thereby endlessly increasing production and consumption, is impossible on a finite planet.

• The GDP is one of the most successful indicators in economics(+) Captures succinctly the value of all goods in the economy(+) Known to the majority of citizens and the totality of people in economics and political science (-) Does not capture the correct prices for some goods (e.g. state provided healthcare)(-) Does not reflect the technological quality improvements(-) Fails to reflect a lot of things about everyday life

“…GDP measures everything, ...except that which makes life worthwhile” (R. Kennedy, 1968)


The beyond GDP Discussion

• Environmental and social factors have to be considered as well in measuring prosperity to ensure sustainable development.

• Attempts to construct prosperity / sustainable development indicators:– The environmental indicators of the Pressure-State-Response (framework (OECD,

1993).– Genuine Saving Indicator – A nation’s wealth is determined as the combination of

produced assets, natural capital and human resources (World Bank, 1995, 1997).– 134 indicators capturing social, economic, environmental and institutional

dimensions of sustainable development (UN Conference on Environment and Development, 1996).

– Human and ecosystem well-being are equally important for achieving sustainable development (Barometer of Sustainability, 2001).

– The Ecological Footprint measures humanity’s demand on biologically productive land and water required to produce the resources needed and absorb carbon dioxide emissions (Global Footprint Network, 2003).

– The Commission on the Measurement of Economic Performance and Social Progress provided legitimacy to the criticism raised about the GDP.


Trends in Indicator Design

• Constructing a single, composite indicator to capture a quality-of-life dimension.

• Using multiple, separate, indicators for social problems, e.g. crime rate, poverty level, air pollution, unemployment rate, etc.

• Preparing all-inclusive indicator reports, produced in a collaborative fashion and intended for wide distribution. Decision-makers and policy-analysts may consult these in an iterative fashion.

• The trend of re-inventing government: – dialogue in the design/ use of measures of performance and customer satisfaction with

government and their interpretation in a complex, changing context. – use of indicators to facilitate the work of many players to make better choices, solve

problems, and be better able to respond to context and change.

• The use of local indicators, parallel to international, national and regional ones seems to be gaining ground.


A Methodological Framework of Indicator Development [Coombes, Wong, 1994]

Step 1: Conceptual consolidationClarifying the concept to be represented by the analysis

Step 2: Analytical StructuringProviding an analytical framework within which indicators will be collated and analysed

Step 3: Identification of IndicatorsTranslation of key factors identified in step 2 into specific measurable indicators

Step 4: Synthesis of indicator valuesSynthesizing the identified indicators into composite index/indices or into analytical summary


Concerns in Indicator Design

• A well-defined and useful indicator should comprise (OECD, 2014):– Policy relevance: address issues that are of actual/potential public concern

relevant to policymaking. – Analytical soundness: being based on the best available science– Measurability: reflect reality on a timely and accurate basis, and be

measurable at a reasonable cost• Institutionalization: setup routine procedures and practices to ensure the

continuing existence of an indicator and to legitimize the method and concept of the measure.

• Avoiding bias: indicators should be produced by professional statistical agencies that have a strong awareness of policy issues, without having responsibility for them.

• Maintain the sensitive balance between (global) standardization and local democracy (identifying the relevant measure to the specific local circumstances).

• Address the fundamental question of integrating prosperity indicators in the policy lifecycle (causality dimension).


A Quick Recap

Citizens formulate their own judgement on the status and progress of prosperity in an environment, characterised by:

•Controversy around the definition of prosperity

•Diversity on the scope and scale of application

•Indicator design methodological issues

•Legitimacy concerns

•A multitude of indicator development initiatives and approaches of different interest or even conflicting nature


How can Policy Compass contribute?

Open new possibilities towards employing quantitative techniques to circumscribe social phenomena and evaluate the results of planned or enforced policy measures at local or regional level.

•Explore the limits of social computing with quantitative indicators for policy design and assessment, given the unprecedented access to the open data sources available.•Enhance the experimentation with various kinds of social indicators, ranging from well-known and widely established metrics, such as the GDP and its variants, to the composite indicators which can suitably apply to the regional or municipality level.•Experiment on the cross-fertilisation of today’s ICT capabilities, with the ideas and intuition of the social (and political/economic) motivation of describing societal welfare with well-defined, representative metrics.•Import the cause/effect component of policy analysis, directly into the indicator analysis process through the ‘injection’ of causal analysis tools in the Policy Compass platform and methodology.


Policy Compass Ingredients

I. Open Public Data

II. Prosperity Indicators

III. Fuzzy Cognitive Maps (FCMs)

IV. Argumentation Technology

V. Deliberation Platforms and Social Media

Key Platform Functionalities

• Prosperity indicators Construction

• Metrics Visualization

• Trend lines Annotation

• Causal Models Development

• Policy Models Simulation

• Sharing/Debating Metrics/Models

• Structured Surveys

How can Policy Compass contribute?


The Policy Compass Prosperity Indicator Framework


Search for relevant open

data sets on the World Wide Web

Import data sets to the platform

Search for indicators/data sets, already uploaded by other users

Select the indicators/data sets that best capture the concepts that reflect the users’ perception

of prosperity

Visualize indicators over time or for selected

geographical regions

Compare indicator

trend lines Search the platform database for recorded historical events

Search and import information on historical events Annotate trend lines with

events to indicate time points where graphs show abnormalities

Indicators Definition

Calculate the indicators’ values,

based on open data sets

Define the relative weights of the selected indicators/data sets and synthesize them into a composite indicator

Need to evaluate political actions

Need to evaluate political actions


Causal Models Construction

Seek for existing policy models, correlating policies with their effects

Edit or construct a new policy model by graphically creating concept variables and relating them through chains of cause and effect to formulate causal networks, and

incorporating fuzzy degrees of causality to model the strength of the causal

relationships identified.

Simulate the policy model to

confirm or reject a causal theory



Deliberation & Argumentation

Push developed metrics, policy

models and visualization

snapshots to the platform

deliberation space

Share developed metrics, policy models and visualization snapshots in Web 2.0 channels

Initiate a new discussion or

participate in an existing

deliberation topic

Produce the discussion

argumentation map

Obtain feedback for initiating another usage scenario …


Causal Models

Construction


Practical Usage Insights

• Where: Focused Project Workshops• Who: Researchers, students, e-Government experts, public sector

representatives• Demo Scenario:

– Development of a composite indicator to assess the quality of tertiary education• Public expenditure on Education• Ratio of academic staff involved against students• Ratio of tertiary education graduates against new entrants

– Effect of education-related policies (lifelong learning, vocational programs, etc.) on unemployment

• Participant Feedback:– Open data as a means of holding governments and politicians accountable– Credibility of data from unofficial sources– Great value in creating own prosperity indicators and causal policy models– Strong analytical skills required– Argumentation and deliberation as the main entry point for citizens


Conclusions and Lessons Learnt

• There exists a growing concern on the limits of GDP-like measures for measuring societal welfare. GDP is very successful, but it is impossible to derive deep results on many aspects of everyday life with simple aggregate indicators alone.

• It makes perfect sense to attempt constructing aggregate, headline or composite indicators, to measure important social phenomena at the regional /local level.

• There do not exist ‘objective’ or ‘neutral’ indicators of any kind. The community does not pursue objective indicators, it rather attempts to construct useful ones.

• A broad experimentation on the calculation and exploitation of indicators, given the multitude of data available, is definitely interesting scientifically and will provide useful "social" feedback.


Ourania Markaki, [email protected]

Thank you!

lessons learnt from the use of prosperity indicators in policy making: towards community-generated...

Science