Lessons from Programme Evaluation in Romania

First Annual Conference on EvaluationBucharest

18 February 2008

Objectives of Session 2

increase the awareness of participants of the existing programme evaluation in the Romanian public sector through the Interim Evaluation of Phare programmes; and

identify lessons from interim evaluation for the Romanian Administration in the further use of evaluation in Public Administration.

Approach to Session 23 Components.For each component:

Short presentations (max 10 minutes)Question and answer/ discussion (max 15

minutes)Conclusion (5 minutes)

Session Conclusions (max 15 minutes)

The three componentsComponent 1Short presentation on Phare interim evaluation

(origins, position in the Phare Management System, link between IE and monitoring, the actors)

Component 2Short presentation on key lessons learned from

IE reports (Sector, Country Summary and Thematic Reports)

Component 3Short presentation on lessons learned for the

future use of IE in Romanian Public Administration, (structures, management and quality issues in IE)

Supporting Material in the Conference Binder

Session AbstractSimplified Guide to Interim EvaluationIntroduction to the IE Website

www.evaluarephare.ro

Component 1: Interim Evaluation is Programme Evaluation in RomaniaLate 1990s: Results monitoring of Phare –

(Monitoring and Evaluation combined)2001: IE introduced when Phare became the

major pre-accession instrument (Separate Interim Evaluation):IE is a management toolIE has very close links to the monitoring systemThere is a well established group of actors

needed to make IE successful

Current Interim Evaluation Project 20 sector programme evaluation reports – 10

each for 2007 and 2008.3 Country Summary Reports for the Joint

Monitoring Committee meetings.3 Three Thematic evaluation reports on:

Procurement practices and proceduresReview of Results IndicatorsPhare lessons learned for structural funds

1 Applied Methodologies Report1 Quality Assurance Review report

Effects of the Evaluation processProvides independent assessment of sectors

and programmes according to relevance, efficiency, effectiveness, impact and sustainability

Provides overall assessments for the Joint Monitoring Committee meetings

Alerts the authorities through the “Early Warning” system

Supports the monitoring system through recommendations to improveProgramme performanceSystems for monitoring and control

The key elements of the IE system Key evaluation capacity elementsA sound methodologyIndependent evaluation capacityA Central Co-ordinating UnitA separate Quality Assurance GroupWider elementsA formal programme monitoring structureAdoption of the programme cycle management

approachSupport from High Civil Servants and the Political

Branch

Discussion Questions for Component 1

Is the Phare Interim Evaluation System a good model to be used for Programme Evaluation in Romania?

How should the system be (a) established (b) resourced?

Component 2: Some Key Lessons from IE Reports

Programme objectives need to be aligned to National and Sectoral strategies that are kept up to date – there is not enough attention paid to updating the strategies.

Switching to a National Procurement System is a complex task that needs careful planning, a lot of training and ad hoc support – the current procurement system should be simplified.

Success comes from investing in people – the Romanian administration needs to strengthen its training and development structures and resources.

Monitoring systems should focus more on results than on activities or inputs – Further results based reform is needed.

A modern strategic management system (e.g. for Structural Funds) requires that responsibility for budgets and results should be decentralised – there is too much control at the centre of the administration.

Discussion Questions for Component 2

Do you agree with the lessons from IE on the previous slide?

Who are the decision makers that are responsible for each lesson?

How can the programme evaluation process influence the decision makers?

Component 3: Future use of Programme Evaluation in the Romanian Administration

Lessons from the Interim Evaluation Scheme are:A properly defined and resourced structure for

monitoring and evaluation is needed for the Programme Evaluation system to work.

Communication of evaluation conclusions and recommendations to decision makers is the most important but difficult aspect of the evaluation process

Discussion Questions for Component 3

What is needed for the Programme Evaluation process to be more effective?

Is the Ministry of Finance a good place for the Central co-ordination of Programme Evaluation activity?

Should all Ministries and Agencies have a programme evaluation capacity?

Should all programmes be subjected to programme evaluation?If Yes – at what frequency?If No – how should be programmes to be evaluated be

selected?

A final questionA Programme Evaluation system is expensive

to create – So, how should the Value of Programme Evaluation be assessed?