rom reviews saskia van crugten nsa-la thematic programme - management of the project cycle and...
TRANSCRIPT
ROM reviews
Saskia Van CrugtenNSA-LA Thematic programme - Management of the Project Cycle and Monitoring of Project ImplementationBrussels, 17-18 September 2013
2
Origin of ROM reviewsMay 1999: the Council of the EU invited the EC to establish a performance-based monitoring system with the objective to strengthen Monitoring, Evaluation and Transparency.
EuropeAid established the Results-Oriented Monitoring system (ROM).
Initially, ROM bridged the gap between EC Task Managers (Brussels) and the project (in the field)…
…today however, most EC Project Manager work in Delegations.
2
3
Main features of ROM system OECD DAC evaluation criteria
PCM / Logical Framework Approach
For on-going and closed projects/programmes (on-going ROM, ex-post ROM, regional ROM)
Standard methodology for all projects Selection of projects for ROM Mission procedure Questions guiding the assessment Report formats
Grading system
External experts 3
ROM reviews: a
“snapshot” of the
project’s quality and
performance
4
Selection of actions for ROM Interactive process between ROM contractors, EU Del, ROM Task
Manager, Headquarters
All projects with budgets exceeding 1 million € and a sample of smaller projects
All projects that have been implemented for at least six months and at least six months remaining before closure
4
ROM should be coordinated with evaluation activities (e.g. not within six months of a mid term review).
Currently, roughly 1/3 of DEVCO projects/programmes are “ROMed” every year.
6 months 6 monthsROM period
StartImplementation End
5
Timing of ROM missions
The ROM exercise is to be timely and quick. 5 days for field visit of an ongoing project. 1 day preparation 2 days reporting.
Ex-post ROM can require more days (up to 10) especially to identify interlocutors.
ROM reports available 15 days after the field visit.
5
6
ROM documents
Project Synopsis (PS): a concise overview on project background, intervention logic and administrative data.
Background Conclusion Sheet (BCS): guiding questions to assess a project and give grades for each of the criteria
ROM Report (MR): findings and grades for ROM criteria, conclusions and recommendations.
Response Sheet (RS): comment on report quality, reply to recommendations and feedback from other stakeholders (to be filled in by Task Manager).
6
Cut paper work…
Keep the
essential…
7
ROM assessment criteria - BCSRelevanceRelevance•1.1 For target group•1.2 For partner government•1.3 For E.U.Quality of designQuality of design•1.1 Intervention logic •1.2 Stakeholder support•1.3 Crosscutting issues in designEfficiencyEfficiency•2.1 Availability of means / inputs•2.2 Implementation of activities•2.3 Output achievement •2.4 Partner contribution Effectiveness Effectiveness •3.1 Output achievement•3.2 Outcome achievement
Impact ProspectsImpact Prospects•4.1 Direct impact•4.2 Indirect impactPotential SustainabilityPotential Sustainability•5.1 Financial/ economic viability •5.2 Target group ownership •5.3 Policy support•5.4 Institutional capacity Horizontal IssuesHorizontal Issues•6.1 Quality Systems, Monitoring and Evaluation•6.2 Technical Cooperation•6.3 EC visibilityCross-cutting issuesCross-cutting issues•7.1 Gender•7.2 Environment•7.3 Good Governance•7.4 Human Rights
7
BCS = Excel file
with 1 sheet
criteria + sub-
criteria
88
1.
2.
Using the BCS
9
ROM grading system
9
Don’t focus too much on
grades at the expense of
substance…
Grades strongly reduce the
complexity!
10
ROM performance categories
Made to describe the performance of a project as a whole;
not quantitative averages of the grades for the performance criteria, but qualitative categorisations.
Category Subcategory Definition
I Very good performance At least three "a", no "c", no "d"
II Good performance
II a No "c", no "d"
II b Maximum of one "c", no "d"
II c Maximum of two "c", no "d"
III Performing with problems At least three "c", no "d"
IV Not performing, major difficulties At least one "d"
11
Why ROM reviews ?
For project managers and EC task managers: ROM informs stakeholders on project performance and helps to focus on results.
For higher level of EC management: ROM provides statistics on overall portfolio performance and highlights problems and success stories.
For knowledge production: ROM can be a source of lessons learnt via meta-analysis/evaluations).
For project beneficiaries: opportunity to discuss project progress and outputs.
11
12
What a ROM review is not ?
An Audit
A mechanism replacing internal monitoring by EC staff
An in-depth analysis of a project/ programme
An assessment of EC Task Managers performance
13
ROM and Quality assurance in the project cycle
13
ProgrammProgramminging
IdentificatiIdentificationon
ImplementatioImplementationn
FollowFollow-Up-Up
PPCM
----Closure-Closure-
--oQSG
2
oQSG 1
Internal Monitoring
Ex-
Post
ROM
FormulatioFormulationn
Final Project Evaluati
on
Ongoi
ng
ROM
DEL managedHQ managed Lessons
Learnt
Accountability
Project Mgmt
Ongoing
ROM
Mid term Project Evaluation
Ex-Post Project Evaluation
14
ROM reviews for supporting project management
Using external experts ROM offers fresh eyes to scrutinize a project:
Is the project, with all due flexibility, pursuing its planned course and will it attain its objectives?
What can or should be done to improve project performance?
Are there issues which could be subject to a more in-depth analysis (evaluation)?
14
Rom reviews have their limits, but
the tool exists and can help. Make
the best of it!