GRANT PROPOSAL WRITING WORKSHOP

Monitoring, Evaluation and Learning

(MEL) for Community and Public

Engagement (CPE)

Peter Kiriri

United States International University – Africa (USIU-A)

pnkiriri@usiu.ac.ke

+254 722 988 777

8th July 2020

Agenda

Introduction

Defining MEL and its Scope

Importance of MEL in CPE

Components of MEL for CPE

Key Proposal Consideration Factors

Different sources were used in developing this presentation. The presenter acknowledges all the sources. A complete listing of sources is available

CPE and MEL

Key issues in CPE

Funders require researchers identify and be

involved in engagement initiatives. Therefore:

What is the quality and impact of engagement

activities?

How do we know whether engagement is

achieving its aims?

Are we having the ‘impact’ that we are being

asked to demonstrate?

How do we build an evidence base to inform and

advocate for engagement practice?

Proposal Expectations in MEL for CPE

Show how the proposal will enhance

Scientific Citizenship at applicants

research institutions

So, how do we engage with the public?

What community and public engagement

plans and strategies can we identify?

How then shall we monitor and evaluate the

success of the initiatives to ensure enhanced

Scientific Citizenship?

Purpose of MEL for CPE

How do we know when we are doing good

engagement and for the right reasons?

How do we learn from what we do and

feed that learning back into our practice?

How can we be clear about what we want

our engagement activities to achieve?

Do those we are engaging with have the

same expectations?

Scope of MEL

Monitoring

Learning

Evaluation

What did we Learn?

What did we Learn?

Key Questions - MEL for Effective CPE

Did engagement influence scientific practice and decision-making?

Did engagement influence other participants and stakeholders to alter their attitude, behave differently, or interact in new ways?

How many people were engaged? What type of persons were engaged?

What was the quality of the engagement process?

MEL Will Help Address…..

Impact

What was the impact of the project on the public, professionals, practice and

policy?

How did it affect people’s knowledge, behaviour, attitudes, emotions, awareness

and skills?

Reach

Who was the project for and who did it reach? Consider both primary and

secondary groups reached and don’t forget reach within research/science as one

of your participant groups.

Quality

How good was the project?

Did the audience relate to it?

Was the content rigorous?

What were the production values and artistic expression?

Value for money

What did the project achieve in comparison to the amount of funds spent?

How does this compare to other projects?

Was this good value for money?

12

CURRENTSITUATION

A Reason to Engage

DESIRED GOAL

Enhanced SC

Community Understanding and Mapping

BaselineEvaluation

Mid Term Evaluation

End Term Evaluation

MEL for CPE

CSFs

CSFs

BARRIERS

13

CURRENTSITUATION

A Reason to Engage

DESIRED GOAL

Enhanced SC

Community Understanding and Mapping

BaselineEvaluation

Mid Term Evaluation

End Term Evaluation

MEL for CPE

CSFs

CSFs

BARRIERS

Community Resistance

Lack of Ownership

Continued poor health outcomes

The Framework for Measuring Engagement:

Head, Heart & Hands

Thinking Understanding,

Knowledge, Awareness

HEAD

Motivation, Emotion, Liking,

Feelings

HEART

Behavior, Action

HANDSWhat do I do

differently?

What’s in it

for me?

Why should I

Support You?

Engagement takes place in three arenas

14

MEL is Anchored on the Theory of Change

Theory of Change (TOC) is a method of

clarifying the underlying assumptions

about how change happens in a project

and the expected sequence of

intermediate changes needed to work

towards a longer-term goal.

It places the activities of your project in the

wider frame of other actors and influence

in the context in which you work.

TOC Components

TOC

MEL for CPE

Inputs

Activities

Outputs

Short-term Outcomes

Intermediate Outcomes

Long-term outcomes

Goal / Impacts

TOC Components

DELTAS Africa TOC for SC Impact (for the overall DELTAS Africa TOC)

Outcome (specific to SC) Enhanced scientific citizenship at research

institutions

Outputs Researchers and programmes are engaging with

the public

Activities Formulation of Community and Public

Engagement plans and strategies

Inputs Funding and supporting infrastructure

Activities – To lead to desired results

What CPE Activities can we engage in? Consultation of communities directly affected by a

particular piece of research.

Establish community advisory forums to promote understanding and dialogue around research priorities and agenda.

Gather community and public input to inform research process

Build capacity of communities affected by research to identify and communicate their needs and priorities around the research focus.

Strengthen the capacity of researchers to communicate research to the media, policy makers and the public.

Output Sample activitiesMembership in

networks,

collaborations, or

partnerships

Travel costs (visa, accommodation, ground travel, conferences etc)

Membership fees, subscription, fellowship costs

Subscription for periodicals, fellowship costs

Engaging with the

public. Formulation of

CPE plans and

strategies

Designing publicity material

Social media engagement

Radio and TV engagement

Soliciting feedback from the public

Personnel costs, time resources

Transport

Meeting venues and permits

Costs of hard to reach groups

Meals and refreshments

Website and technological gadgets

Disseminating

research findings

through media.

Formulation of

communication and

media engagement

plans and strategies

Extracting summarized and easy to read reports – microblogs, news

portals

Creating podcasts

Buying space in print media

Dissemination by brief video clips, SMS, handouts,

Extracting blogs from scientific papers

Social media advertising

Media consultancy

Engaging with and

generating interest

from policy makers.

Develop engagement

plans and proposals

for policy makers

Developing policy briefs

News media (radio, television, newspaper)

Newsletters

Media consultancy

One on one meetings

Workshops and seminars

Research reports

Professional journals

CEP Outputs – Immediate Results

On completion of CEP Activities Number of community and public engagements

Number of participating community leaders/members

Signed memorandum of understanding on engagement

Operationalization of Community Advisory Boards

Increased community and public awareness of existing research

Increased public debate and media on research issues informed by community level experiences and scientific evidence.

Outcomes

Changes in ways of doing things

Improved relationship between the research institutions and the community and publics

Enhanced utilization of research findings by communities and publics

Enhanced scientific practice and decision-making in the research institutions

Changed community and public attitudes towards the research process and findings

Improved quality of the engagement process?

Impact (1)

The longer-term sustainable change attributable to a project or intervention that remains after the project has finished.

Direct causal influence of a project that can be measured

Achieved through activities and immediate outputs of a project (often called ‘outcomes’).

These then continue to have an impact after the duration of the project – see the Results Chain

Impact (2)

Long term sustainable change

Community improved health status as a result

of adoption of research findings

Improved quality of life of the community due

to acceptance of research findings

Community and public ownership of the

engagement initiatives and adopting the same

as best practices

M&E and TOC

Based on TOC, Key Questions Arise

How will we know we are being effective?

What kind of changes should we expect to

see, and for whom, as a result of our

engagement activities?

How will we track and measure such

changes?

Key Issues to Consider in MEL for CEP

What are the desired results in

engagement?

What are the key indicators?

How do you measure output, outcome and

impact using the indicators?

Which MEL frameworks and tools can we

use?

What are the challenges of MEL?

Indicators

The Power of Measuring

If you do not measure results, you cannot tellsuccess from failure.

• If you cannot see success, you cannot reward it.

• If you cannot reward success, you are probablyrewarding failure.

• If you cannot see success, you cannot learn from it.

• If you cannot recognize failure, you cannot correctit.

• If you can demonstrate results, you can win donorsupport.

Source:

Adapted from Osborne & Gaebler 1992

Indicators

Markers that help to measure change by showing progress towards meeting objectives

Means of measuring what actually happens against what has been planned in terms of quantity, quality and timeliness, for every level of result.

A quantitative or qualitative variable that allows the verification of changes produced by a planned intervention

Observable, measurable, and agreed upon

Used to judge the success of an initiative, project or program.

Type and Level of Each Indicator

Input indicators: that will track the means allocated for implementation of the activities either financial, personnel (technical assistance volunteers) facilities, equipment & supplies

Process indicators: that will track the activities in which the inputs are utilised for instance in training, in establishment of a logistic system, in planning of the service delivery

Output indicators: track the direct and immediate results of input and processes at project level

Outcome indicators: refer to the intermediate results at the target population level that are closely linked to the project

Impact indicators: long-term results of interventions on the health of individuals, households, communities

Examples of Indicators

Infant mortality rate,

under 5 mortality rate

Funding amounts ($), trainers

(person-months), ORT supplies (numbers),

etc

Number of health personnel trained in

ORT, number of media ads produced, etc

Number & % of mothers knowing

about & having access to ORT services

Number & % of child diarrhea

cases treated with ORTPurpose

Outcome indicators

Goal

Impact indicators

Outputs

Output indicators

Activities

Process indicators

Inputs

Input indicators

Objectives

resultsResults monitoring

Strategies

meansImplementation monitoring

Type a

nd L

evel of E

ach Indic

ato

r

35

MEL Frameworks

MEL frameworks are the overall conceptual framing or theory about the most appropriate ways to track and measure change.

Frameworks are ways of organizing and keep you focused on what you want your outcomes to be and what information you will collect to keep you on track.

Frameworks help to focus thinking and clarify definitions.

MEL Frameworks

Theory of Change (ToC)

Logical Frameworks (Logframe) Approach

(LFA)

Outcome Mapping (OM)

Participatory Monitoring and Evaluation

(PME)

MEL Tools

Interviews

Surveys

Rapid Appraisal

In-depth Case Studies

Participatory Video

Focus Group Discussions

Cost–benefit Analysis

Action Learning Sets.

Proposal Requirement – MEL for CPE

Describe how your consortia shall evaluate programme outcomes and impact, including timeframes, proposed success metrics and roles.

How is your proposed evaluation approach linked to the DELTAS Africa II theory of change and indicator framework?

Describe the programme's key milestones, outlining any key dependencies, constraints, and assumptions

Consideration for the Proposal

1. Identify CPE desired results/GOAL - MEL Framework to adopt eg TOC

. Identify required inputs2

3. Identify CPE activities

4 Determine results at different levels of TOC – Output, Outcome, Impact

5. Determine how you will measure the results – Indicators

6. Determine the measurement techniques and tools to adopt

7. Determine how results will be communicated to key stakeholders

Why MEL Fails

The Wrong Goal

The researcher and the stakeholders are united in a direction which does not create value / impacts

Lack of Direction

No clear framework for integration MEL in the Scientific Research project

Disconnected LeadershipResearchers have a clear understanding of impacts but haven’t communicated to the community and public

Competing Goals

Different groups within the research context disagree on how to undertake MEL

How does MEL Work?

Two words

Focus

Alignment

44

Lack of

Results

CPE monitored and evaluated Impacts

MEL for CPE is about…….

CPE not monitored and evaluated

Dr. Peter Kiriri

United States International University – Africa (USIU-A)

pnkiriri@usiu.ac.ke

+254 722 988 777 45