introduction to monitoring and evaluation systems poverty reduction and equity...
TRANSCRIPT
2
Why M&E Systems?
1. Improve the effectiveness of public policy
2. Improve the allocation of public expenditure for higher welfare
3. Increase accountability4. Strengthen design, implementation,
and results of programs and policies
3
Outline of the Presentation
1. What is M&E?
2. How M&E influences Policy?
3. Country M&E Systems
4
1.- What is M&E
Monitoring
Provides regular information on performance
A continuing function 1) Indication of extent of progress towards goals
- poverty rates - coverage of basic services
2) systematic data collection3) usually conducted in-house
5
What to Monitor?
1. Identify few indicators that can be measured and are of interest to policy makers
2. Prioritize input, output, outcome and impact indicators for monitoring
3. Develop a data collection system to sustain monitoring
6
Inputs Outputs Outcomes Impact
How to Monitor?
• Use different types of information: surveys and censuses, administrative data, and participatory exercises
• Draw on logical framework and arrangements for results monitoring which identifies:-Data sources for each indicator-Frequency of measurements-The organization responsible for collecting information.
• Different types of Monitoring: -Annual progress report-Database for continuous monitoring-Desk reviews
7
Evaluation
Evaluation helps to analyze deviations from targets and goals ( e.g Why learning achievement increased?
Why poverty decreased?) Usually conducted by independent
evaluators A systematic and objective
measurement of the results achieved by a project/program/policy
8
What to Evaluate?
Evaluation can be time and resource intensive Plan ahead evaluation and identify questions
Strategy, operation, and learning
Reserve impact evaluation in cases such as Strategic significance of a policy or program Contributing to close knowledge gap Innovative nature of the policy or program
9
High
Low
Low High
Participantjudgment andexpert opinion
Complexity and sometimes cost
Strength of causal inference
Before-and-after
Quasi-experimental
with statisticalcontrols
Experimentalwith randomassignment
Trade Offs in Evaluation
Summing up…
Monitoring assesses progress in implementation of ongoing programs
Evaluation provides a snapshot against some benchmarks
Monitoring looks at progress relative to targets and assumes there is causality
Evaluation seeks to establish and prove causality
11
12
2. How M&E influences Policy?
13
M&E and the Policy Cycle
M&E needs to accompany the policy cycle so decisions can be based on evidence
Define what you want set goals and targets
Analyze what might work to reach targets design programs and evaluate ex-ante
Implement set performance indicators, a monitoring system , and implement evaluations
14
But we are far from there…
4 years ago, more than half of programs in Mexico had no logical frameworks No explicit objectives, no definition of target population
Countries seldom set targets using scientific methods: Reduce violent crimes from x to y in 5 yrs… is that a
meager performance or a great performance? Countries rarely define the outcome (say, nutrition),
and then decide in which sector the correct intervention is
Countries seldom analyze ex ante who the winners and losers of a policy are
Definition of Objective
s and Targets
Inputs Activities
Outputs
Policy and
Program
Design
Outcomes Impacts
Implementation Results
The Results Chain
Amount of services provided
Health literacy
Consumption, Life expectancy, poverty
Definition of Objective
s and Targets
Inputs Activities
Outputs
Policy and
Program
Design
Outcomes Impacts
Implementation Results
Pla
nn
ing
S
tag
e • Benchmarking• Logic Framework• Poverty Analysis• Ex-Ante Distributional and Poverty analysis
The Results Chain
•What are the extent and causes of poverty in a given country? •How is a proposed policy expected to affect the poor?•How to set strategic targets for a given policy/program?
Definition of Objective
s and Targets
Inputs Activities
Outputs
Policy and
Program
Design
Outcomes Impacts
Implementation Results
Monitoring Stage
• Logical Framework• Performance Indicators• Dashboards
The Results Chain (cont)
•What is a given policy/program trying to achieve? •What are the causal links through which the project is supposed to achieve its goals? •Is the project achieving its objectives? How to measure outputs and outcomes?
Definition of Objective
s and Targets
Inputs Activities
Outputs
Policy and
Program
Design
Outcomes Impacts
Implementation Results
Evaluation Stage
•Design Evaluation•Process Evaluation•Consistency Evaluation•Impact Evaluation•Economic Analysis/Modelling
The Results Chain•Is the execution of the project activities conducive to the expected results? •Are the chosen indicators good indicators? •What is the impact of the intervention? •Are the changes in well-being indeed attributable to the intervention?
19
Several methods to support evidence policy
Performance indicatorsThe logical framework (logframe) approachTheory-based evaluationFormal surveysRapid appraisal methodsParticipatory methodsPublic expenditure tracking surveysImpact evaluationCost-benefit and cost-effectiveness analysis
Example: Consistency and Results
Consistency and Results covers: Design, Strategic Planning, Target Population
and Coverage, Operation, Perception of Beneficiaries and Final Results.
Instrument applied by external institutions under terms of reference prepared by a Central Unit
The evaluation includes a Summary Strengths, Challenges and Recommendations
for every program
DESIGN Does de program identifies clearly the problem it’s trying to
solve? Is it clear that the products or services produced by the
program logically contribute to reduce the problem the program is tacking?
STRATEGIC PLANNING Does the program have updated strategic plans for the short,
middle and long-term? In the S-plan, does the final results are clearly established?
TARGET POPULATION AND COVERAGE Does the program have a method to identify and quantify the
potential and target population? Does the program have a coverage strategy for the short,
middle and long-term?
Consistency and Results: Questions
OPERATION Are there standard and adequate procedures for
the selection of projects, beneficiaries? Are there documents showing that the procedures
are done according to the rules of operation. PERCEPTION OF BENEFICIARIES
Does the program have instruments to measure the satisfaction of the beneficiaries with the program?
FINAL RESULTS Does the program systematically collect objective
information about its Goal (Fin y Propósito)? Does the program have rigorous impact
evaluations?
Consistency and Results Questions (cont)
Findings CONEVAL-Mexico
Topic Findings
DESIGN
67%
have a Purpose and Goal that relate to the problem’s solution.
50%
of the programs defined necessary and enough outputs to achieve the Purpose.
STRATEGIC PLANNING
21%
of the plans clearly establish the results they want to accomplish.
23%
of the programs have strategic plans in the short, middle and long-term.
TARGET POPULATION AND
COVERAGE
24%
of the programs quantify the target and potential population.
23%
of the programs have a coverage strategy for the short, middle and long-term.
Findings CONEVAL-Mexico
Topic Findings
OPERATION
72%
have standardized and adequate procedures for the project/ beneficiaries selection.
84%
of the programs have efficacy indicators in their operation.
BENEFICIARIES PERCEPTION
50%
of the programs have instruments to measure the satisfaction level of the target population.
FINAL RESULTS
34%
of the programs collect objective information about their final results.
26%
have external evaluations that allow to measure their impact.
25
3. Country M&E Systems
Country M&E Systems: Keys for success
Systems which permit the generation of information (basic data, indicators, evaluations, etc), reliable, credible and high quality
A high level of utilization of the information generated by the M&E system
Sustainability for the future
Improve the supply
Incentives from the demand side
The demand side: specific uses of an M&E system (or..why it’s needed?)
To support government planning at the national, sub national, local and sectoral level in the establishment of targets/goals and permanently ask the question why are targets not being met? focus on results
To support management public programs through monitor and evaluate performance and effectiveness
To informing the redesign and design of programs and interventions
To inform budget allocation decisions
To facilitate accountability
Countries choose one or all of these roles for M&E systems
Contradictoy?
Increasing effective demand and use of M&E
Pressure from civil society for budget accountability and vigilance of state´s performance
Need to maintain macro - equilibrium and simultaneous pressures to extend coverage of public services efficiency
Produce information (data , indicators, evaluations) which can effectively be used. Worry about the trust on the quality of information
Incentives for sectoral and budget officials, congress, etc. so that they use the information
Leaders who persuade and motivate the government to use the information generated a champion (person or institution) of results-orientation
All this is usually more important than legislation
Building blocks on the supply side
1. Improve Statistics:
household surveys, sectoral surveys and censuses
Better administrative records and sectoral statistics
Integration of systems to generate input and outcome indicators
Regional and local level data
to feed the production of M&E
Sectoral/program monitoring systems
Government –wide monitoring systems (Systems of presidential goals, PRS monitoring).
Performance evaluation systems for public services
Ex-ante and Ex-post Evaluations (Process evaluations , Rapid Assessments, Impact E.)
Ensuring sustainability
Permanently redefine the system. M&E tools have to be effectively used to improve performance and effectiveness of the state at all levels: Are we spending right? Are programs being well implemented? Are programs attaining its targets? Are we spending on the right things? Are programs being effective
Establishing strategic alliances: M&E a permanent function of different entities that should work together Finance ministries Planning agencies, Supreme Audit Institutions National Statistical Offices Evaluation offices ( embedded in an high leverage institutions) Sector ministries and its statistical offices Local governments
32
Thanks you