performance measurement for social protection franziska gassmann bangkok, june 2012
TRANSCRIPT
Performance Measurement for Social Protection
Franziska GassmannBangkok, June 2012
Objectives
• Understand the importance of performance measurement as an essential tool for social protection governance. • What is performance measurement? • Where is it applied/can it be applied?
• Understand the basic principles underlying performance measurements in relation to social protection by discussing the type and characteristics of indicators and related issues of data collection and data quality.
• Link performance measurement to the policy making process and the dialogue with partner countries.
• Understand how to select indicators and set targets for EC programme management.
Social Protection
• Defined as public interventions to support poor and vulnerable households and individuals with the objectives to• Smooth consumption • Provide support during difficult times• Reduce poverty and inequality• Stimulate economic development
• The focus is on non-contributory programs financed from general government revenues, such as• Cash transfers• Subsidized goods and services• Direct food support• Public work programmes• Social services• Residential care provision
Outline
• Relevance of performance measurement for social protection
• Indicators for performance measurement• Different types of indicators• The characteristics of good indicators• Performance indicators for social protection policies• Measuring poverty
• Data requirements• Wrap up
Performance measurement…
• …monitors at a programme level the cost and quality of services and outcomes for clients in order to account for public expenditures
• …is a tool for controlling and managing resources • …requires the definition of outcome objectives for policies and
programmes• …provides regular and systematic information whether a
program, policy or development strategy is implemented as planned, is achieving its objectives and whether and how it can be improved
• …moves the focus from inputs/outputs to outcomes and results
Why focus on outcomes?(EC Guidelines 2009)
• Results matter• Encourage evidence-based policy making• Protect political space of beneficiary countries• Promote domestic accountability• Stimulate demand for high quality data
Performance measurement for social protection
• Non-contributory social protection policies are financed from general government revenues
• Spending on social protection is heavily scrutinized and subject to constant pressure
• It is vital for the government to prove that public spending on social protection achieves its objectives effectively and efficiently
The systematic and regular monitoring and evaluation of social protection policies is therefore crucial for any national government accountable to the public.
Regular performance measurement is especially important for social protection
• Social protection policies are developed at central government level, but service delivery takes place at local level
• Social programs rely on implementing agents• The poor often live in remote areas or socially
isolated surroundings• Market forces are less prevalent in social service
delivery
Guidelines for measuring progress towards intended outcomes in social protection
• Use information that is already available• Monitor the extent to which social protection systems are pro-poor• Agree on a common set of indicators• Invest in building national capacity(EC, 2003)
• Performance indicators measuring achievements of objectives should be coherent with national and sector policies
• Coherence with MDGs• A programme ensuring the quality of the indicators should be in place(EC, 2009)
Types of indicators
INPUTS
OUTPUTS
OUTCOMES
IMPACTS
INTERMEDIATE INDICATORS
FINAL INDICATORS
Resources committed to the program/policy
Quantity of goods and services generated
Results of the use of goods and services;
access, usage, satisfaction of users
Effects on welfare and living standards
A good indicator is…(World Bank, 2004)
• A direct and clear measure of progress (specific)• Varies across areas, groups and over time• Sensitive to changes in policies• Not easy to manipulate or blown off by unrelated developments• Relevant for policy making• Can be tracked in a cost-effective way (available)• Consistent with policy-making cycle• Easy to understand• Reliable• Consistent with data availability and data collection capacity
…and supported by intelligent benchmarks(Cichon et.al, 2004)
• Objective-based benchmarks• Time-based benchmarks• Comparative national benchmarks• Comparative international benchmarks
Two dimensions of social protection performance
• Horizontal efficiency:• Effectiveness of the programme in reaching the
target group• Under-coverage (exclusion) reduces horizontal
efficiency
• Vertical efficiency:• Efficiency of the programme in reaching the
target group• Inclusion errors reduce vertical efficiency
Targeting efficiency – evaluation matrix
Target household Non-target household
Household receives
transfer
Success Inclusion error
Type II error
Household does not
receive transfer
Exclusion error
Type I error
Success
Measuring exclusion and inclusion errors
• Exclusion error (under-coverage) =Number of poor w/o benefitNumber of all poor
• Inclusion error = Number of non-poor with benefitNumber of beneficiaries
• Leakage =Total amount of benefits received by non-poorTotal amount of benefits paid
Distributional analysis – example
Targeting performance of the Unified Monthly Benefit, percentages, 2005
Type of benefit Quintile
I Quintile
I I Quintile
I I I Quintile
IV Quintile
V Total Benefit incidence 25.9 23.7 17.3 5.4 0.8 14.6 Distribution of beneficiaries 35.6 32.3 23.7 7.4 1.1 100.0 Distribution of benefits 34.9 40.3 18.0 5.7 1.0 100.0 Adequacy (recipients only) 7.2 5.4 4.3 2.1 1.8 Note: Quintiles based on annual per capita consumption before all social transfers assuming a marginal propensity to increase consumption of 25%. * not significant at the 10% level. Source: Own calculations based on KIHBS 2005.
Measuring targeting effectiveness• Share of transfers going to the poor =
% of transfers received by poor% of poor in total population
• Example: • Poor are 40% of the total population• Benefits received by the poor = 20%• Share received by the poor = 20/40 = 0.5
• Interpretation:• Value = 1: targeting is neutral• Value < 1: targeting is regressive• Value > 1: targeting is progressive
Measuring adequacy
• Share of benefits in total household income (consumption) =
Benefit valueHousehold income
• Change in poverty rates before and after transfer• Absolute change = change in percentage points• Relative change = change as % of pre-transfer
poverty
Measuring Poverty (a very short intro to…)
• Why measure poverty?• Assess living standard of population, identify the poor• Identify needs/areas for policy support• Evaluate effectiveness of national poverty-reduction
and redistribution policies• Assess impact and performance of development aid
• What do we need (besides data…)?• Concept• Indicators• Measures• Analysis and interpretation
Conceptualizing poverty, cq well-being
• Welfarist approach:• Individual utility determines well-being• (Lack of) well-being expressed in the level of income,
consumption, assets, etc. • Monetary measures
• Non-welfarist point of view:• Other aspects determine well-being, such as
ownership of commodities, fulfillment of basic needs, access to basic services, being socially included….
• Lack of capabilities to function
Most common approaches
• Monetary poverty (income, consumption, expenditure, assets, …)• Capability approach (coined by A. Sen)• Social exclusion (think ‘EU’)• Vulnerability (origin in disaster-risk-reduction)
• Monetary poverty: one-dimensional (although, implicitly multi-dimensional)
• Other approaches: multi-dimensional
• Monetary poverty: Still most common approach used for policy evaluation and poverty assessments
Poor ≠ Poor
• Each approach has different perspective on what constitutes a good life
• Each requires methodological assumptions
• Each has different policy implications• MA: ensuring higher income solves problem
• CA: wider range of action needed (social provision of goods, improved allocation within family, efficient use of goods to achieve outcomes)
• SE: relative aspect important; just raising the level does not do the trick; redistribute policies
In order to measure poverty, we need…
• Welfare indicator• What indicator is best suited to determine the living
standard of a household or individual in a given society?
• What indicator allows ranking households from poorest to richest?
• Different welfare concepts imply different indicators, and my identify different groups of poor
• Define a cut-off threshold, cq. poverty line• Absolute versus relative poverty• What constitutes a minimum living standard?
Proportion of health deprived children by income quintile, Vietnam 20060
.02
.04
.06
.08
.1pr
op
heal
th p
oor
1st quintile 2nd quintile 3rd quintile 4th quintile 5th quintile
Relative vs. absolute
• Relative poverty lineSet as a share of average or median income or expendituresMoving concept; there will ‘always’ remain a class of poor peopleClosely linked with income distributionExamples: EU social indicator (60% of median income)
• Absolute poverty lineBased on minimum consumption basket (food plus?)Survey-based or derived from other standards (e.g. national
legislation, WHO)Examples: 1$ per day line, national poverty lines
Example: Poverty in Country X
0
10
20
30
40
50
60
70
80
90
100
1998 1999 2000 2001 2002 2003 2004
%absolute
extreme
relative
How to determine the minimum living standard?
• In monetary terms:• Objectively defined minimum, e.g. based on a basket of goods
and services – absolute level• As a percentage of the average living standard – relative level
• In non-monetary terms: • Minimum years of schooling• Access to primary health care• Possession of x household appliances• Having one hot meal per day• …
Examples of poverty lines
• International absolute poverty lines: • 1.25 USD PPP per capita per day• Useful for international comparison, but not necessarily for
national policy making• International relative poverty lines:
• 60% of median income per adult equivalent• World Bank methodology based on actual consumption for national
poverty lines:• Extreme poverty: costs to obtain a minimum food basket• Absolute poverty: Minimum food basket plus allowance for non-
food goods and services• Subjective poverty lines
• Derived from minimum income question• Frequently higher than ‘objective’ poverty lines
• Guaranteed minimum income (budget based)
y
n
zHC
PG
Poverty measures
q
The 3 most common poverty indicatorsn=total population; q=poor; yi=welfare level; z=poverty line
• Poverty incidence (headcount) P0 = q/n = number of poor
total population
• Poverty gap = average gap as a percentage of the poverty lineP1 = 1/n ∑ q [(z – yi) / z];non-poor: gap is zero!
• Poverty severity = weighted average gapP2= 1/n ∑q [(z – yi) / z]2 INDEX!
Example: Kyrgyz Republic 2005Source: WB (2010)
Poverty headcount (%) Poverty gap (%) Poverty severity by region Issyk-kul 51.5 14.3 5.1 Jalalabat 55.9 16.3 6.1 Naryn 51.2 14.4 5.4 Batken 59.1 17.3 6.1 Osh 55.9 10.4 3.2 Talas 44.4 11.7 4.2 Chui 22.0 4.5 1.3 Bishkek 10.8 1.9 0.4
Poverty indicators by region (absolute poverty line)
Total country: 43.1% 10.5% 3.6
Scorecard Kyrgyz Republic (WB, 2009)
Program Spending / share of
GDP (% , 2007)
Coverage share of
poorest Q (% , 2005)
Targeting accuracy /
share of benefits captured
by Q1 (% , 2005)
Benefit generosity / share of benefit in consumpti
on (% , 2005)
Poverty impact / relative
reduction of extreme
poverty gap (% , 2005)
Poverty impact / relative
reduction of extreme
poverty rate (% ,
2005)
Cost efficiency/ c
ost per 1 KGS
reduction of poverty and
extreme poverty gap
UMB 0.53 28.2 38.0 7.2 11.6 3.1. 1.6 / 4.7 Performance Moderate Moderate Strong Weak Strong Weak Strong MSB 0.21 12.7 33.3 7.6 6.3 6.4 3.0 / 5.5 Performance Low Weak Strong Weak Moderate Moderate Strong Categorical benefits and subsidies
0.6
16.3*
12.6*
1.1*
0.8*
0.5*
6.3 / 27.3
Performance High Weak Weak Weak Weak Weak Weak Pensions 5.1 55.5 28.5 25.0 47.8 32.6 3.3 / 10.2 Performance Moderate High Moderate Moderate High High Moderate
Data requirements – Types of data
• Administrative data• Provide information on inputs and outputs• Collected by line ministries or special agencies• Less suitable for analysis of policy outcomes
• Household survey data• Representative sample of population• Essential for poverty analysis• Essential for assessing policy outcomes
• Population census• Basic information on all inhabitants (limited indicators)• Very costly, at most every 10 years• Needed for poverty maps• Usually not suitable for assessing policy outcomes
Data requirements – household surveys
• Essential for analysis of policy outcomes• Availability improved considerably over the past decade• Many variations, but not all equally suitable
• HBS, LSMS, MICS, DHS, etc.
• Have their own shortcomings• Reliability of information (e.g. income from transfers)• Questionnaires not always in line with latest policies• Target groups cannot always be identified (e.g. disabled, war
veterans)
• How frequent should they be available?• No general rule• At least every 2-3 years, if financially affordable
Data requirements – qualitative data
• Provides information not provided in surveys• Subjective dimensions of well-being• Intra-household inequality• Barriers to access (benefits, services)• Cultural, political or social factors that determine
policy outcomes and impacts
• In-depth interviews, focus group discussions• To inform survey design or to validate findings
from survey
Wrap up
• Performance measurement ensures the accountability of the policy maker and the transparency of the decisions taken
• Availability and measurement of performance indicators is essential for social protection policy making• Ensure regular and systematic collection of reliable information• Analyse the collected information from various perspectives
• Data requirements need to be carefully assessed as resources are often scarce• Better have a limited set of core indicators which are regularly
collected and analysed• Performance measurement is part of the policy process and the interpretation
of indicators needs to be placed in the political and economic context