The political side of social program

evaluation

Ministry of Social Development MEXICO

Gonzalo Hernández Licona

Objective Analyse the institutional challenges facing Mexico,

specifically SEDESOL, in constructing a Monitoring and Evaluation (M&E) system.

How can we institutionalize an M&E system? The political constraints and challenges

How can we go beyond the impact evaluation of Oportunidades?: More programs evaluated in a regular basis Constructing a Results-based management system Insert M&E within the Social Policy process

Outline Analytical framework

What type of M&E system we’re aiming for?

The need of creating and designing institutions in order to build an M&E system

What have we done and what’s next?

Conclusions

Social Policy Identification of social

problems and objectives

Analysis

Program design Program operation and

resources Budget

Monitoring and Evaluation

Evaluation: why and how Helps re-designing and

improving programs

Supports efficient use of public funds

Adds objective and technical elements to the social policy debate

Promotes Transparency and social accountability

Emphasis on results We need to measure

indicators …but consider qualitative evaluations

(Long run) impact evaluation and (frequent) monitoring of every day operation

External (and good) evaluators

Objective and useful evaluations: search for program’s stakeholders cooperation and participation

Decision taking Firm Social Program ProcessI. Identifying benefits

II. Measuring Impact

III. Information

IV. Who should evaluate?

V. Monitoring

VI. Who demands evaluations?

VII. What do we do with the results?

Well-beingpoverty health infrastructure income nutrition perceptioneducation social cap. satisfaction

I-C. IRR What would have happened without.. .? Counter-factual

In order to know the

whole process

It doesn’t work usually looking for results

Profits

Owners; share-holders

Efficient use of information

Results: what for?

Not clear Public resources

In general preciseThe firm pays for it

Lots of information Who pays for it?

The firm itselfExternal auditors

The programExternal evaluators

I. Identifying benefits

Precise Rules of Operation Better rules now Still problems due to the fear for auditors

We still have social programs that have only political objectives: Opciones productivas, PET, Acuerdos para el Campo: Vivienda

rural, adultos mayores del campo

II. Measuring Impact: How? Guidelines for the annual Evaluation of

Programs issued in 2002 by the Ministry of Finance and the Ministry of Audit (Contraloría) The guideline indicates the need to measure impact in

every program every year Very ambitious... …but helpful in the short run to accelerate the creation of a

culture of evaluation Pressure from donors

Progresa CIMO Probecat..

II. Measuring Impact: Promoting long-run impact evaluations

External support: technical and financial (WB, IDB, Conacyt, international academics)

At least 9 impact evaluations in SEDESOL Progresa-Oportunidades 1997-2004 Liconsa fortified milk Microsimulation: Oportunidades, Liconsa, Diconsa Food program Micro-regions Strategy Habitat Housing program Tu Casa Coahuila State’s Piso Firme Jóvenes con Oportunidades

II. Measuring Impact: The politics of the evaluation design Progresa: Centralized program with relatively

little participation from beneficiaries or local authorities: it favoured randomization.

Decentralized programs sometimes should seek for other methodologies

An experimental design requires political support We tried to include in the SDL the possibility to

have randomization, when feasible.

III. Information From the institutional point of view it is not clear who

should pay for the information The program? Sedesol? Hacienda? Donors?

Sometimes programs hide information from evaluators

Next step: La Contraloría

IV. Who should evaluate? Presupuesto de Egresos de la Federación

External Evaluators National evaluators

International evaluators not allowed sometimes Creation in 2002 of the Under Secretariat of Planning,

Prospective and Evaluation Social Development Law

External evaluators The SDL explicitly bans consulting firms from evaluating

social programs

IV. Who should evaluate?: The Human capital of External evaluators Not enough (good) evaluators

Impact evaluation Monitoring

A good evaluator should be able to evaluate a program with the existing ingredients: experiments are rare.

Impact vs Monitoring Seminars (impact evaluation, monitoring, qualitative

evaluation, power calculations, etc.) We’re promoting partnerships between national and

international evaluators

IV. Who should evaluate?: The bidding process The bidding process favours the cheapest proposal

Public universities don’t have to go through the

bidding process Monopolies

Excess transparency Changing the external institution every year Annual contracts

Incentives to present good results in order to evaluate again in the future

V. Monitoring There is no obligation to do this Indicators demanded by Hacienda, Función

Pública, Presidencia, Congress, with little management purpose: Highly inefficient

International support: WB, IDB Create a true Results-based management system for every

program New Dirección General de Evaluación y Monitoreo

de Programas Sociales, in order to promote internally the construction of a monitoring system

VI. Who demands evaluations?

1. International donors IDB support: the need for evaluation Receptive authorities: Levy, Gómez de León

Internal battle (técnicos vs rudos)

2. The opposition in Congress Since 2000, Congress demands annual external evaluations for

every public program. Presupuesto de Egresos de la Federación (PEF) In 1999 the PRI was not a majority in Congress The opposition feared the use of social programs for the 2000

election There were good and reasonable academics in Government

VI. Who demands evaluations?

The Social Development Law (2003-2004),

institutionalizes the evaluation process National Council for the Evaluation of Social Policy Evaluation of programs not only in Sedesol Poverty measurement We’re including in the Social Development Law (SDL)

the obligation to evaluate every new federal program Law of Transparency and Public Access to Information

Democracy

VII. What do we do with the results? In the past almost nothing

The results were useful to Progresa in order to survive, but it was not useful for the everyday management

A “small” institutional change: Our internal indicator % of programs evaluated every year % of external recommendations attended by programs The weaknesses and recommendations (summary) are officially

sent to every program manager The programs have to give an official answer on what actions

will they implement The office of internal affairs (contraloría interna), demands to

see proofs of the actions taken every year

VII. What do we do with the results? Evaluations are taken more into consideration

Better reports by evaluators

Programs make changes

Evaluations became this year a tool for the budget

process within Sedesol We still need to link more closely (and formally,

evaluations with the budget process)

VIII. The politics of the evaluation process: very important If we aim for an objective but yet useful M&E, we

need to take into account stakeholders We need the participation of stakeholders

involved with evaluations The DGEM published internal rules for the

evaluation process: In the process operators should participate in TORs,

analysis and reviewing sessions of developed work Continuous dialogue with external agents DGEM runs the party

Decision taking Marks Social Program ProcessI. Identifying benefits

II. Measuring Impact

III. Information

IV. Who should evaluate?

V. Monitoring

VI. Who demands evaluations?

VII. What do we do with the results?

Well-beingpoverty health infrastructure income nutrition perceptioneducation social cap. satisfaction

What would have happened without.. .? Counter-factual

It doesn’t work usually looking for results

Results: what for?

Not clear Public resources

Lots of information Who pays for it?

The programExternal evaluators

?

?

?

?

Conclusions The Evaluation must be part of the Social Policy

process Building a Monitoring and Evaluation System is a

political task, that requires technical elements It is important to institutionalize the process and

take into consideration the program’s stakeholders for the evaluation process

In Mexico, it is crucial to build-up on the evaluators, policy-makers and congress official’s technical abilities

Too many programs, not many evaluations

Between 1990 and 2002 Mexico spent almost 550 billion dollars in social policy

There were very few evaluations: Probecat, Liconsa, Diconsa, Progresa

The market itself doesn’t solve the need for evaluation in social programs