1

IFAD’S RESULTS MEASUREMENT FRAMEWORK

Kevin Cleaver, Associate Vice President, ProgrammesGary Howe, Director, Strategic Planning April 2010

2

Results areas and indicators that IFAD uses in managing its operations, resource mobilization, human resources, risks and efficiency

Measurement and reporting of results

Baseline results

Targets for 8th Replenishment

Content of presentation

3

IFAD uses its loans and grants for

Integrate livestock to Integrate livestock to match rising demandmatch rising demand

Develop private Agro-Develop private Agro-processing & mktgprocessing & mktg

Include cash crops: Include cash crops: exports are growingexports are growing

Improve basic foods Improve basic foods and staplesand staples

4

RMF 2007-09: A snapshot of achievements compared to the 2005 Independent Evaluation of IFAD

RESULTS% of projects designed in 1995-2001 period rated satisfactory by IEE (2005)

Latest results (RIDE, for Dec 2009 Board)(% of projects satisfactory)

Medium-term (2011) target established by IFAD’s Board (% of projects rated satisfactory)

Project effectiveness at completion61% ARRI – 82%

PCR – 87%80%

Satisfactory impact on poverty at completion37% ARRI – 91%

PCR – 80%70%

Satisfactory impact in learning, scaling up and/or knowledge management at completion

25-55% ARRI – 100%PCR – 71%

65%

Sustainable at completion 40% ARRI – 73%PCR – 75%

80%

Satisfactory country strategy impact on food security, income empowerment

n.a. (ARRI) 69% 70%

Satisfactory overall at entry (effectiveness, poverty impact, sustainability, innovation)

- 100% 90%

% of country programs satisfactory for adherence to aide effectiveness agenda

- 96% 90%

Average time from project approval to effectiveness (months)

18 13.7 n.a.

Average time from project approval to first disbursement (months)

n.a. 20 n.a.

5

Level 1: Macro outcomes

Level 2: IFAD Country programme and project outcomes

Level 3: IFAD Country programs and project outputs

Level 4: IFAD Country programme and project design and implementation support

Level 5: IFAD’s Institutional management & efficiency

IFAD’s second Results Measurement Framework:the Results Chain

Outc

om

e m

ore

rele

vant

to

IFA

D O

bje

ctiv

es

Attrib

utio

n to

IFAD

Stre

ngth

ens

6

RMF in the context of IFAD: Operating Model

7

Level 1: Macro and sectoral Outcomes Baseline figures and global targets that IFAD will monitor

IndicatorBaseline

yearBaseline

value2012

target

1.1. MDG 1: Population living on less than a $1.25 a day (%) 1/

2005 26 21

1.2. MDG 1: Prevalence of under-nourishment in population (%) 1/ 2002 - 2004 17 10

1.3. MDG 1: Children under 5, underweight % 1/ 2005 27 17

1.4. Crop production index (1999-2001 = 100) 2/ 2006 112.4 Tracked

1.5. Agricultural value added (annual % growth) 2/ 2004 4.1 Tracked

Source:1/. United Nations, The Millennium Development Goals Report, 20082/. World Bank, World Development Indicators, 2009

8

Level 2: IFAD Country programmes & project outcomes

Project performance measured at Project

start-up and at completion by: Relevance – consistency of the project objectives with the

priorities of poor rural people Effectiveness – how well projects perform in delivering against

their objectives; and Efficiency – how economically resources are converted into

results.

Sustainability, the net benefits sustained beyond the implementation period

[1] High-level Forum on Aid Effectiveness, final report of Roundtable 4. Accra, Ghana, 2008.[2] OE identifies them as ‘impact’, involving anticipated effects as well.

9

Replication and scaling up, to add value to successful innovations and

Gender equality and women’s empowerment, integration of women’s as well as men’s concerns, so that women and men benefit equally.

Poverty impact - the changes that have been perceived to have occurred in the lives of poor rural people.

Food security, availability, access to food, and stability of access

Physical and financial assets, for income and crisis coping Empowerment, as an end in itself, as well as a means of

reducing poverty

Level 2: IFAD Country programmes & project outcomes contd …

10

Level 1: IFAD Country programmes & project outcomes: 2009 Baseline Values and 2012 Targets

Chart 1: Country Programme and Project Outcomes:Targets for 2012 and Achievements in 2009

0%

20%

40%

60%

80%

100%

% o

f pro

ject

s m

od

era

tely

sa

tisfa

cto

ry

or

be

tter

PCR 2008-09 - Baseline RMF target for 2012, PCR ARRI 2009

PCR 2008-09 - Baseline 94% 84% 65% 83% 75% 71% 76%

RMF target for 2012, PCR 90% 90% 75% 90% 75% 75% 80%

ARRI 2009 91% 82% 55% 91% 73% 100%

Relevance Effectiveness EfficiencyRural poverty

impactSustainability

Innovation, learning and replication

Gender equity

11

Level 2: Data Sources and Reporting

Data obtained from the project completion reports and project evaluations (summarised in ARRI)

Evaluation instruments use a six-point scale.

6= highly satisfactory, 5 = satisfactory

4= moderately satisfactory, 3= moderately unsatisfactory

2= unsatisfactory, 1 = highly unsatisfactory

Broadly, ratings of 4 and above are taken as acceptable and 3 or below as unsatisfactory

RMF targets are set mostly as percentage of projects rated 4 or above at start-up determined by QA findings and at completion

12

At the third level, RMF reports on outputs Project outputs are reported on an absolute basis

such as hectares of land brought under irrigation, number of active savers and borrowers.

Key outputs are aggregated and reported against IFAD’s six strategic areas using the RIMS system reporting on a sex-disaggregated basis, where

applicable

Level 3: Project Outputs

13

Level 3: Project outputs against: Baseline Values and Targets

3.1 People receiving services from IFAD-supported projects 2007 29 million 60 million

Male : Female ratio 2007 57:43 50:50

3.2 Area under constructed/rehabilitated irrigation schemes (ha) 2008 470 thousand

 

3.3 Common Property Resources (CPR) land under improved management practices (ha)

2008 3.86 million

 

3.4 People trained in crop production practices/technologies Male : Female ratio

2008 1.72 million50:50

3.5 People trained in livestock production practices/technologies Male : Female ratio

2008 1.07 million35:65

Indicator Baseline year

Baseline value

Target2012

14

Level 3: Project outputs against: Baseline Values and Targets contd …

IndicatorBaseline

year Baseline Value Target

3.6    Active borrowers (Number) 2008 4.35 million  

Male : Female ratio   52:48  

3.7    Savers (Number) 2008 5.44 million  

Male : Female ratio   51:49  

3.8    Roads constructed/rehabilitated (‘000’ km) 2008 15 thousand  

3.9    Marketing groups formed/strengthened (Number) 2008 25 thousand  

3.10  People trained in business and entrepreneurship (Number) 2008 162 thousand  

Male : Female ratio   53:47  

3.11  Enterprises accessing facilitated non-financial services (No) 2008 19 thousand  

3.12 People trained in community management topics (No) 2008 672 thousand  

Male : Female ratio   38:62  

3.13  Village/community action plans prepared (in '000) 2008 24 thousand  

15

Level 3: Data Sources and Reporting

RIMS is IFAD’s trademark definition of key impact indicators, which it suggests be included in project M&E systems

RIMS operates at three levels of the results chain: outputs, outcomes and impact, based on the data from the Project M & E system

By providing common definition of indicators, RIMS permits aggregation of outputs across all IFAD projects, as presented in previous tables

16

Results and Impact Management System (RIMS)

About 40 IFAD projects have established baselines for impact assessment using special RIMS surveys

RIMS was developed prior to the development of the self and independent evaluation systems: indicators are therefore not fully aligned

Management will review RIMS in order to bring about full coherence among results system

RIMS helps define indicators to be included in M&E systems; it does not substitute M&E system.

Results and Impact Management System (RIMS)

17

At the fourth level IFAD measures the performance of country programmes and projects at entry and during implementation

The main focus is on:

Quality at entry of country programmes for income, food security, empowerment of women and men

Adherence of aid effectiveness agenda in terms of country ownership, alignment and harmonisation

Quality at entry of projects for effectiveness, poverty impact, sustainability, & innovation, learning & scaling up

Better implementation support, measured by time elapsed for Board approval to 1st disbursement, % of problem projects and pro-activity, time-overrun, and delay in processing withdrawal applications etc.

Level 4: Country Programme and project management

18

Level 4: IFAD Country Programme: Baseline and Targets

Indicator SourceBaseline

yearBaseline value (%)

2012 target

(%)

% of country programmes rated satisfactory or better for:

4.1 Contribution to increasing the incomes, improving the food security, and empowering poor rural women & men

QA 2008/09 >90 90

4.2 Adherence to aid effectiveness agenda Client survey

2009 96 90

% of projects rated satisfactory or better at entry for:

4.3 Effectiveness QA 2008/09 93 90

4.4 Rural poverty impact among the target group, such as (a) physical and financial assets, (b) food security, (c) empowerment, and (d) gender equality

QA 2008/09 91 90

4.5 Sustainability of benefits QA 200809 81 90

4.6 Innovation, learning &/or scaling up QA 2008/09 86 90

19

Level 4: Project Implementation Support

Better Implementation Support

4.7  % of on-going projects actually receiving international co-financing

PPMS

 

2008 61 65

4.8 Average time from project approval to first disbursement (months)

PPMS 2008 20 14

4.9  % of problem projects in which major corrective actions are taken (Pro-activity index)

Div PPR 2008 63 75

4.10 Percentage of projects for which IFAD performance rated 4 or better

ARRI 2008 64 75

4.11 % of problem projects in the ongoing portfolio PPMS 2008-09 19 15

4.12 Percentage of time overrun for average project PPMS 2008-09 22 20 

4.13 Time taken for processing withdrawal applications (directly supervised)

WATS 2008 35 -10% over 2009

IndicatorBaseline

yearBaseline

value2012 targetSource

20

Level 4: Data Sources and Reporting

How undertaken:

The QA at entry rating is done independently by the consultants

The client survey, which reports on the perception of in-country stakeholders (‘clients’) comprising of representatives of governments, donor agencies and civil societies, undertaken annually, ensures anonymity of the respondents

Non-rated absolute numbers, such as average time elapsed, co-financing levels are generated through PPMS and WATS.

For day to day management monitoring in real time, Operations Dashboard has been developed. Brings data from different systems in a user-friendly interface.

21

Level 4: Data Sources and Reporting (cont’d)

22

Level 4 deals with what IFAD does operationally to achieve field level impact through its country programmes – in projects and at the macro-level of the MDGs

Level 5 addresses how well IFAD manages its resources to support these operations – specifically, how well:

It focuses its workforce on the operational area It engages the commitment of its workforce It creates a diverse staff drawing on experience and capabilities It achieves efficiency in expenditures It manages risks

Level 5: Institutional management and efficiency

23

Level 5: Baseline and targets

5.1 % achieved of replenishment pledges Peoplesoft Financials

2008 93 100

5.2 Staff engagement index (% staff positively engaged in IFAD objectives)

5.3 % of workforce in programmes (operations)

5.4 % of workforce from Lists B and C Member States

5.5 % of women in P5 posts and above

5.6 Average time to fill professional vacancies (days)

5.7 Cost per payslip (US$)

Global staff survey

Peoplesoft HR

Peoplesoft HR

Peoplesoft HR

HR

Peoplesoft Financials

2008

2008

2008

2008

2007

2008

70%

56%

33%

30%

141

US$90

75%

65%

Tracked

35%

100

Tracked

5.8 % of high-priority internal audit recommendations that are overdue

5.9 Budgeted expenses per US$1 of loan and grant commitments

ROL

POW&B

2008

2008

76%

16.30%

20%

13.5%

IndicatorBaseline

year

Baseline value(%)

2012 target (%)Source

24

Managing to achieve results and impact

The RMF targets are not just a basis for reporting, they are the points of orientation for the management of IFAD

Since 2007, IFAD has operated a comprehensive corporate planning and performance management system (CPPMS) that concentrates on the elements of performance directly managed by IFAD.

The CPPMS is IFAD’s system for managing to achieve the results. It establishes objectives for all managers It focuses on what managers can actually manage and be directly

accountable for

Focuses on Levels 4 and 5 – on what IFAD directly manages, on the activities that lead to development and institutional efficiency results

Project development and implementation support The institutional platform for project-related work

25

The results management matrix - 1

At the corporate level, management to achieve results revolves around targets and performance indicators for four activity clusters:

Cluster 1. Country programme development and implementation Cluster 2. High-level policy dialogue, resource mobilization and strategic

communication Cluster 3. Corporate management, reform and administration Cluster 4. Support to Members’ governance activities

Each of these groups has sub-clusters with specific targets (what we call Corporate Management Results – CMRs), e.g. Cluster 1 has:

CMR 1 Better country programmes; CMR2 Better project design; and CMR3 Better project implementation support

26

The results management matrix - 2

Each CMR includes an objective the achievement of which will contribute to better performance against the RMF targets

Each CMR has a basis for measuring performance - a number of Key Performance Indicators, e.g.,

CMR 1: Adherence to aid effectiveness agenda

CMR 2: % of projects rated satisfactory or better at entry for effectiveness

CMR 3: % of problem projects in the ongoing portfolio

27

The results management process

Each year every department prepares a plan organizing all activities around the CMRs they directly contribute to (according to function) – according to corporate level priorities

Every division prepares their own plan indicating their planned contribution to achievement of the corporate and departmental CMRs

Every individual prepares a work plan organizing their contribution to achievement of divisional plans

Performance is reviewed on a quarterly basis on corporate, departmental and divisional basis in performance conversations covering every department and every division, focused on:

progress towards results risks that need to be dealt with in achieving results resource management issues

The objective is identification of management actions (including financial and human resource deployment) that need to be taken in real time to keep performance on track

28

The performance conversation process

- By 2nd week after close of qtr

- Division Directors & staff

- Review performance and risks

- By 3rd week after close of qtr

- Department Heads & Division Directors

- Review performance and risks (department risk registers)

- By 4th week after close of qtr

- Operations Management Committee & Executive Management Committee

- Results, resources and risks report (corporate risk register)

29

Anchorage in day-to-day management systems - 1

This system is not designed to produce reports to the Executive Board. The reports on progress (RIDE) is produced from the information that IFAD needs and generates to manage itself for results on a day-to-day basis.

The key to the effective functioning of managing for results is accurate information on outputs and inputs. The Management Information System is being constantly upgraded:

The CPPMS brings together all plans, KPIs and risk reports within the Peoplesoft system

Budget and financial information is drawn from the Peoplesoft financial and budget systems

Loan and grant information is drawn from the Loan and Grant System (to be replaced in 2011)

30

Anchorage in day-to-day management systems - 2

Project performance and supervision data are drawn from the PPMS, supplemented by:

Project Status Reports On-line

PMD Management Dashboard (reports)

Human resource management data are drawn from the Peoplesoft HR database, supplemented for reporting and analysis by a new HR dashboard

Although all of these systems contribute to tracking performance against RMF, none are specifically for that purpose. They are all dedicated to normal processes of outcome, output and input tracking.

31

Bringing efficiency under scrutiny

The results-based management system has been focused on development impact, and the monitoring system in this area is very robust – contributing to a major up-turn in IFAD’s project performance

There are still issues in management for development results – particularly data quality

The key issues for further elaboration lie in the efficiency area

IFAD has been raising its overall efficiency – and has actually cut administrative budgets in real terms

It needs the efficiency drive to be based on detailed and accurate understanding of work load and volumes – to set more realistic targets and performance indicators

This work is under way, and we expect more comprehensive indicators to be in place by the end of the 2nd Quarter 2010