Monitoring and EvaluationPlan for Africa RISING

Naomi Sakana (IFPRI)Africa RISING East and Southern Africa Research Review and Planning

Meeting, Arusha, Tanzania, 1-5 October 2012

Outline

AR Objectives, activities, and expected resultsMonitoringEvaluationM&E ObjectivesPrinciplesIndicatorsMethodsRoles and Responsibilities

Africa RISING purpose• Objective

Transform agricultural systems through sustainable intensification

Sustainable intensification: producing more (agricultural) output from same area of land, while:reducing negative environmental impactsincreasing contributions to natural capital

and flow of environmental services

Africa RISING activities• Implementation activities

Situation Analysis and Program-wide Synthesis (diagnosis & design of

technology combinations)Integrated Systems Improvement (on-farm & on-station trials of new

combinations of existing technologies)Scaling and Delivery (institutional and/or value chain improvements)

• Systems and SitesCereal-based farming systems – Guinea-Savannah zone of West

AfricaCrop-livestock systems - Ethiopian highlandsMaize-legume-livestock & Rice-vegetable systems - Eastern &

Southern Africa

Expected results• Expected key, direct results of implementation

• Sustainably increased whole farm productivity• Reduced negative environmental impacts• Increased contributions to natural capital & flow of

environmental services• Value chain conduct and performance improved? /

Improved market efficiency? / Improved extension effectiveness?

• Many other results possible: labor use, WUE, poverty, nutrition, hunger, etc.

Monitoring“process of systematic collection and analysis of

data on specific”• Describes the “what” of implementation• Useful for:

Management - to see if project is on trackReporting – to inform client & other stakeholders of

progress• Indicators

USAID FTFOthers (custom) useful to AR for monitoring or

evaluation or both

Evaluation“periodic assessment of worth or significance of an

activity, policy or programme”• Helps understand “how” & “why” of implementation• Determines & attributes impact• Qualitative and/or quantitative• Dimensions

Sites (different levels) Development domains Household types Technologies & combinations Implementation processes

M&E Objectives

• Support effective project management (provides the project managers with timely information on the status of activities

and the ongoing results they are achieving)• Help all stakeholders to learn about the project’s

successes and failures (provide opportunities to learn what

works and what does not)• Provide the data for timely reporting to USAID

M&E Commitments• FtF Compliance: M&E standards, best practices, and core indicators established for

the entire FtF initiative.

• Open-access platform: deliver and maintain an open-access, M&E data management and analysis platform to serve the needs of SI implementation partners and other stakeholders.

• Monitoring & projection: generate ex ante evaluations (e.g. project targets) for a range of farming system and livelihood outcome indicators on an annual basis to provide enhanced research management and outcome mapping needs.

• Multi-scale reporting: provide the capability to support multi-scale monitoring and evaluation

SSA-wide: cross-system reporting to serve the needs of SI wide roll-up of indicators across the three investment geographies/system “project sites” (Guinea Savanna, Ethiopian Highlands, Eastern and Southern Africa)

M&E Commitments• Multi-scale reporting: provide the capability to support multi-scale monitoring

and evaluation

Site-wide report: for each of the three project sites

Country report: Breakout of site-wide reports to serve the needs of national stakeholders (e.g., USAID country missions, national institutions)

Custom/Sub-system reports: Some reporting needs will need to be met by customized aggregation of sub-system indicators (e.g. to generate reports by CRP or by farming system)

• Scaling-up and out potential outcomes and impacts: To inform planning and longer-term projections of potential innovation impact at scales beyond the actual action research sites, forward looking analysis will explore the productivity and sustainability consequences of a range of adoption scenarios and geographic/system spillover pathways across broader landscapes and regions.

Guinea – Savannah

East and Southern Africa Maize Mixed

Ethiopian Highlands

Africa Rising M&E Components, Activities, and Outputs

Program/Project Site Identification Outputs

FtF Indicators / reports by- Research sites- Country / National level- Project sites- Program / SSA

Perfomance Variables(modeling & validation)-∆Whole farm productivity-Technology performance ∆ Yield ∆ Labor prod.- by gender ∆ NUE, WUE- ∆Revenues, Costs, Profits

M&E Outputs-FtF Indicators-Outcome mapping (incl. nutrition & market effect)-Cost/Benefit analyses-Experimental /RCT evaluation -Adoption studies?

SI Innovation Catalogue- Inventory (cross-site)- Characterization- Open access

Project Planning & ManagementImproved insights into innovations , delivery platforms, and site selection

Learning

Data/Analysis Platform

Contextual Data(national/regional)- Statistics- HH survey & census- Spatial data

Derived Indicators- HH Typologies- Intensification Index- Sustainability Index- Nutrition index?

Ranking domains by key AR attributes

A ________C ________B ________

Project Site Stratification(Development Domains)

B

A

A C

Project/Activity/Partner Inventory- Project DB (& maps)

Action Research Site selection criteria-Site access-Existing activity/platforms-Research design-Intervention type-M&E approach-…….M&E approach

Identify action research sites in

priority domains that satisfy selection

criteria

Site Data - Climate, soils, market access, etc- Community/HH survey data - Experimental data- Model input data

Whole-farm models

Innovation Inventory- Standard metadata- User interfaces

Research Site ActivitiesBaseline survey Set up trialsMonitoring Mid-line survey (?)End-line survey

++

+++

+

Communities/Farms/Plots

FTF Goal: Sustainably Reduce Global Poverty and Hunger

FTF First Level Objective 1: Inclusive Agricultural Sector Growth

Sub-IR 1.3: Improved Agriculture Policy Environment

Output Indicator: #8 No. Policies/

Regulations/Admin Procedures in each stage of development as a result of USG assistance

IR 1: Improved Agricultural Productivity Outcome Indicators: #10 Gross margin per hectare (whole farm and by crop) #11 Number of hectares under improved technologies or management practices

IR 3: Increased Investments in Agriculture and Nutrition-Related Activities

Output Indicator: #9 Number of public-

private partnerships formed Outcome Indicator: #16 Value of new private

sector investment in the agriculture sector or food chain

Sub-IR 1.1: Enhanced Human and Institutional Capacity Development for Increased Sustainable Agriculture Sector Productivity Outcome Indicators: #5 Farmers who applied new

technologies or management practices…

#6 Private enterprises/organizations that applied new management practices…

#12 Stakeholders implementing risk-reducing practices/actions to improve resilience to climate change

Output Indicators: #1 Individuals who received long-

term training #2 Individuals who received short-

term training #3 Private enterprises/ organizations

receiving assistance #4 Producer/community based

reorganizations receiving assistance

Sub-IR 1.2: Enhanced Technology Development, Dissemination, Management, and Innovation Output Indicators: #7 No. of new technologies

or management practices: 1) Under research, 2) Under field testing, or 3) Made available for transfer

#13 No. of rural households benefiting directly from USG interventions

IR 2: Expanding Markets and Trade

Outcome Indicator: #14 Value of incremental sales Custom Outcome Indicator: #15 Farmer satisfaction with quantity, quality and timeliness of extension and input supply services (Sub-IR 2.3: Improved Market Efficiency)

USAID-provided IRs and indicators Additional suggested IRs and indicators

IR 5: Increased Resilience of Vulnerable Communities and Households

Output Indicator: #17 No. of

vulnerable households benefiting directly from USG interventions

Africa RISING Results Framework and Indicators (based on FtF)

Improved connectivity to and utilization of markets and input suppliers

Wider dissemination of integrated SI innovations leading to similar impacts beyond the AR Action Research Sites or SI, leading to similar impacts beyond the AR action research sites

RO 1&2: Sustainable increase of whole-farm productivity by integrated innovations for targeted households at research sites

RO 1&2: Increased nutrition and reduced poverty, especially for women and children

RO 1: Situation Analysis & Program Synthesis RO 2: Integrated Systems Improvement

Diagnosis - Site selection & Characterisation Mega-site stratification by development domains Prioritizing mega-site strata geographic units Project Action/Research Site Selection Action/Research Site Characterisation Problem diagnosis Construction of farm household typologies Identifying constraints and opportunities (disabling

environment, options, entry points) Conducting value chain assessment Literature review Baseline survey Development of common key indicators

(biophysical, socio-economic & institutional) Technology inventory and characterisation Ex-ante analysis of potential options integrated systems priority setting & planning Cooperation and collaboration with partners

Developing approaches for farm level interventions Identifying key components of integrated systems Identification of intensification trajectories Sequencing interventions to suit stage of

intensification of household types / systems Developing participatory tech. selection methods Identifying models and decision support tools to

guide ex-ante technology identification Ex-ante sustainability & resilience evaluation Identify high impact sweet spots/ best bets/fits

Participatory evaluation and adaptation of appropriate combinations of technologies and interventions Combining improved legumes with improved

livestock feeding practices Managing soil fertility and experimenting with

novel approaches to increase productivity Developing incentives for better soil management Developing site specific recommendations Combining identified technologies

(e.g., Agroforestry / MPT, alternatives to draught power to save feed, CA with a livestock lens, fertilizer trees, fiderbia/ Acacia/ leguminous trees, improved management of seasonal feed resources, kitchen gardens/ continued poultry, legume rotations (effective rhizobia, biological N fix), micro dosing, more effective contribution to livestock to nutrient management, supplemental irrigation, rainwater harvesting,etc.)

Knowledge and Capacity building Testing novel extension models Establishing a linked system of models Training on market oriented production Addressing new research challenges and

opportunities emerging from the activities

RO 1, 2&3: Improved understanding of landscape level ecosystem stability from the aggregate impact of farming practices at the household level

RO 1, 2 &3: Improved community-based & on-farm NRM

RO 3: Scaling and Delivery of Integrated InnovationsRO 3: Scaling and Delivery of Integrated Innovations

Increased R4D community ability to design and implement farm-scale SI action research, outreach and support approaches, and related D&KM systems

Increased R4D community ability to design and implement farm-scale SI action research, outreach and support approaches, and related D&KM systems

RO 1, 2 &3: Increased ability of R4D community to design and implement farm-scale action SI research, outreach and support & related D&KM systems

RO 1, 2 &3: Increased ability of R4D community to design and implement farm-scale action SI research, outreach and support & related D&KM systems

Figure 3: AR Results Framework based on Research Design/Plan Document.

(compare with Figure 2)

Scaling up/out successful technologies and interventions Assessing the scalability of integrated innovations Identification and development of scaling

approaches for targeted integrated innovations indicators Testing approaches for scaling up and scaling out

SI innovations in action sites with project area Developing a costed program for scaling by

development investors Evaluating aggregated impact of household level

interventions at landscape level Evaluation and validation of scaling approaches for

integrated systems

Analytical Approaches & Tools • Delineation and characterization of target

farming systems • Change estimation/projection models for

selected indicators• Outcome mapping• Attribution assessment, including site

selection and stratification, potential comparison group(s), and impact assessment design

Impact evaluation methodsMethod Pros Cons Likely to

be used

Outcome Mapping Easier to implement and interpret. Forces articulation of impact pathways

Primarily qualitative. Subjective assessment approach

Yes

DD (difference in difference)/RCTs

Provide quantitative evidence

Not as rigorous if RCTs are not carried out

Yes

PSM/IPW (propensity score matching methods)

Provide quantitative evidence, although second-best option

Not as rigorous if RCTs are not carried out

Yes

RDD (regression discontinuity design)

Provide rigorous evidence

Big sample needed, sharp cut-off based on continuous eligibility criterion (which AR is currently not supporting)

No

Roles and Responsibilities• Monitoring implementing

partners (IITA and ILRI and Collaborators),Sister CG centers, NARS, FOs, NGOs,Private sector

• Evaluation (IFPRI and partners)

Information Systems Team(IFPRI)

M&E Coordination Team(IFPRI)

Collaborators(SpatialDev, ABT, MSU, etc.)

West Africa Project M&E

Team(IITA)

Ethiopian Highland Project M&E Team

(ILRI, IFPRI)

ESA Project M&E Team

(IITA, MSU)

AR M&E Learning Task Force• 3 month brief (1st Qtr 2013)• M&E Task Force (AR, CSISA, FEEDBACK, CRSP, Local

Institutions M&E specialists)• Provide CSISA-AR cross-learning and FtF FEEDBACK

best-practice guidelines• Visit all sites, meet local teams and refine M&E

strategy that; (1) appropriate for interventions being tested, (2) strikes appropriate balance between rigour and cost/feasibility, (3) lays out strategy for remainder of 5 years.

Thank you!