monitoring, evaluation, reporting, and learning (merl) for ... · undp united nations development...

Monitoring,

Evaluation,

Reporting, and

Learning (MERL) for

Peacebuilding

Programs

Pact is a promise of a better tomorrow for all those who are poor and marginalized. Working in partnership to develop local solutions that enable people to own their own future, Pact helps people and communities build their own capacity to generate income, improve access to quality health services, and gain lasting benefit from the sustainable use of the natural resources around them. At work in more than 30 countries, Pact is building local promise with an integrated, adaptive approach that is shaping the future of international development. Visit us at www.pactworld.org.

September 2016

Disclaimer:

Portions of this module have been made possible by the generous support of the American people through the U.S. Agency for International Development (USAID). The entirety of the contents of this module are the responsibility of Pact and do not necessarily reflect the views of USAID or the United States Government.

Recommended citation:

Pact. 2016. Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs. Washington, D.C., U.S.: Pact.

Contact:

Pact’s Results and Measurement Team Pact 1828 L Street NW, Suite 300 Washington, D.C. 20036 202-466-5666 [email protected]

http://www.pactworld.org/

Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs

Contents

Abbreviations and Acronyms ............................................................................................ i

Acknowledgements .......................................................................................................... ii

Introduction ..................................................................................................................... 1

Chapter 1: Overview of MERL Principles ......................................................................... 2 What is MERL? ................................................................................................................................. 2 What is peacebuilding? .................................................................................................................... 3 Why is MERL important for peacebuilding? .................................................................................. 3 Definitions ........................................................................................................................................ 4 Who is on a MERL team? ................................................................................................................. 8 Key Reasons for Developing Quality MERL Systems ..................................................................... 8 MERL in the Project Cycle ............................................................................................................. 10 MERL in Summary .......................................................................................................................... 11

Chapter 2: Peacebuilding Approaches: Defining What Peace Looks Like ....................... 13 What are peacebuilding approaches? ............................................................................................ 13 Approach 1: Incentives for Peace ................................................................................................... 14 Approach 2: Addressing Root Causes of Injustice ........................................................................ 14 Approach 3: Addressing Individual Attitudes and Relationships and Building Trust ............... 14

Chapter 3: Situation Analysis .......................................................................................... 16 What is situation analysis?............................................................................................................. 16 Types of Situation Analysis ............................................................................................................ 16 Disseminating Situation Analysis Findings .................................................................................. 19

Chapter 4: Theories of Change and Developing Project Goals........................................22 What are Theories of Change? ....................................................................................................... 22 Theories of Change in the Context of Peacebuilding Programs ................................................... 23 Develop Your Peacebuilding Project’s Theory of Change ............................................................ 25 Develop Project Goals and Outcomes from Your Theory of Change ........................................... 26 Tying Situation Analysis and Theory of Change to MERL System Design ................................. 28

Chapter 5: Conceptual Frameworks and Assumptions ...................................................32 What are conceptual frameworks? ................................................................................................ 32 Advantages and Challenges of Using Conceptual Frameworks ................................................... 33 The Results Framework ................................................................................................................. 34 Logical Frameworks ....................................................................................................................... 39 Basic Logframe Outline (Hierarchy of Objectives) ....................................................................... 39 The Role of Assumptions ............................................................................................................... 40

Chapter 6: Indicators ..................................................................................................... 43 Overview ......................................................................................................................................... 43 How to Develop Indicators ............................................................................................................ 45 Qualities of a Good Indicator ......................................................................................................... 47 Indicator-Related Terms ................................................................................................................ 48 Indicator Definitions ...................................................................................................................... 49 Collecting Indicator Data ............................................................................................................... 49

Chapter 7: Monitoring .................................................................................................... 52 What is monitoring? ....................................................................................................................... 52 Conflict and Context Monitoring ................................................................................................... 53 Implementation Monitoring .......................................................................................................... 54 Assumption Monitoring ................................................................................................................. 56 Monitoring Plans ............................................................................................................................ 57


Chapter 8: Evaluation .....................................................................................................59 What is evaluation and why is it important? ................................................................................ 59 Defining Evaluation ........................................................................................................................ 59 Monitoring versus Evaluation ....................................................................................................... 61 Evaluation versus Research ........................................................................................................... 61 Why evaluate? ................................................................................................................................. 62 Types of Evaluation ........................................................................................................................ 62 Internal and External Evaluation .................................................................................................. 63 Involving Stakeholders in Evaluation ........................................................................................... 64 Barriers to Evaluation .................................................................................................................... 64 Planning for an Evaluation ............................................................................................................ 64 Writing an Evaluation Purpose Statement ................................................................................... 67 Evaluation Questions ..................................................................................................................... 68 What is evaluation design? ............................................................................................................ 69 Overview of Evaluation Approaches .............................................................................................. 69 Creating Evaluation Worksheets ................................................................................................... 70 The Evaluation Terms of Reference ............................................................................................... 71

Chapter 9: Learning ........................................................................................................ 74 Overview ......................................................................................................................................... 74 Using an Adaptive Management Approach .................................................................................. 74 Developing a Learning Agenda ...................................................................................................... 75 Ensuring Learning Processes are Participatory ............................................................................ 76

Chapter 10: Reporting .................................................................................................... 80 Overview ......................................................................................................................................... 80 Introduction to Reporting .............................................................................................................. 80 Identifying Appropriate Communication Tools ........................................................................... 81 Reporting Schedule ........................................................................................................................ 81 Generic Outline for Quarterly and Annual Progress Reports ...................................................... 83 Generic Outline for Final Performance Report ............................................................................ 85 General Writing Tips ...................................................................................................................... 87 Success Stories and Photography .................................................................................................. 87

Chapter 11: Data Quality and Ethics ............................................................................... 89 The Importance of Ensuring Quality Data .................................................................................... 89 Commonly used Criteria for Assessing Data Quality ................................................................... 89 Data Management .......................................................................................................................... 90 Getting Started with Developing Systems to Address Data Quality Issues ................................. 91 Improving Data Quality along the Data-Management Chain ...................................................... 92 Data Ethics ...................................................................................................................................... 94

Chapter 12: Mobile Technology for Data Collection ........................................................95 Overview ......................................................................................................................................... 95 What is information communication technology for development? ........................................... 95 Advantages of Mobile Data Technology ........................................................................................ 96 SMS versus GPRS ........................................................................................................................... 97 Developing Your Mobile Technology Strategy and Why it is Important .................................... 98 When You Should Continue Collecting Data on Paper .............................................................. 104 Develop a Budget .......................................................................................................................... 105

Appendix 1: Success Stories Guide ................................................................................ 107

Appendix 2: MERL Plan Template ................................................................................ 109

Appendix 3: References ................................................................................................ 125


page i

Abbreviations and Acronyms

APEA applied political economy analysis

CBO community-based organization

CDC U.S. Centers for Disease Control and Prevention

COP chief of party

DCOP deputy chief of party

DFID U.K. Department for International Development

DIY do it yourself

DQA data quality assessment

GIS geographic information system

GPRS general packet radio service

GPS global positioning system

ICT4D information communication technology for development

Log frame logical framework

M&E monitoring and evaluation

MERL monitoring, evaluation, reporting, and learning

MPICE Measuring Progress in Conflict Environments

NGO nongovernmental organization

OECD Organization for Economic Cooperation and Development

PEA political economy analysis

PIRS Performance Indicator Reference Sheet

SMS short message service

SOP standard operating procedure

TH trauma healing

TOC Theory of Change

TOR Terms of Reference

U.K. United Kingdom

U.S. United States

UNDP United Nations Development Programme

US$ United States dollar(s)

USAID U.S. Agency for International Development


page ii

Acknowledgements

The initial draft of this module was compiled by Lynn McCoy, Hannah Kamau, and Margaret Elise during their time with Pact.

Special thanks are given to Pact’s PEACE III project MERL team—Jacqueline Ndirangu, Lauren Serpe, Josiah Imbayi Mukoya, Michael Kahindi, and Christopher Kinyua—for customizing this module for use in Pact Kenya’s PEACE III program and for broader publication to share with other peacebuilding programs.

Many other Pact staff contributed to the development of the peacebuilding MERL module. Gratitude goes to Nanette Barkey (technical oversight), Leighton Clark (technical contributions), Maggie Dougherty (graphic design), Rachel Elrom (editing and layout), and Mason Ingram (technical contributions).


Introduction | page 1

Introduction

Robust monitoring, evaluation, reporting, and learning (MERL) are critical components of successful programming. The MERL components enable program stakeholders to monitor progress and evaluate the achievement of expected results. Reporting processes and timelines should be clearly defined and tailored to meet the needs of key audiences and stakeholders, and provision should be made for the program to continually reflect and learn from experiences gained during implementation.

Measuring the success of peacebuilding programs poses specific challenges that are unique to this program area. This module was developed to guide PEACE III local program partners—peacebuilding practitioners—through the development and implementation of effective and practical MERL systems for their projects. This five-year cross-border peacebuilding program is implemented by Pact in partnership with Mercy Corps and a range of local partners with activities in Ethiopia, Kenya, Somalia, South Sudan, and Uganda. PEACE III aims to strengthen cross-border conflict management in the Horn of Africa and is pursuing two related objectives: 1) to strengthen local cross-border conflict management and 2) to improve the responsiveness of regional and national institutions to cross-border conflict.

This manual was created to support and provide examples to peacebuilding practitioners and is an addition to Pact’s existing MERL Modules.1 Other useful examples of MERL training manuals for peacebuilding programs exist. This module does not seek to replicate those manuals, but rather draws on them and integrates their expertise here. Parts of this manual were also drawn from Pact’s MERL Modules but customized with practices and examples relevant for peacebuilding programs. It is the authors’ hope that this module can serve as an introduction to MERL for peacebuilding practitioners and can point them to other relevant resources in the field. Each chapter begins with an outline of the learning objectives, includes learning activities throughout the chapters, and ends with a summary of key points and learning.

1 Pact has developed four core training modules to guide MERL staff and to provide a framework for Pact’s partners on M&E basics. These are: Module 1: Building Basic Monitoring and Evaluation Systems; Module 2: Field Guide to Data Quality Management; Module 3: Field Guide for Evaluation; Module 4: Mobile Technology Handbook. All modules can be found in Pact’s Resource Library: http://www.pactworld.org/library.

http://www.pactworld.org/library


Chapter 1: Overview of MERL Principles | page 2

Chapter 1: Overview of MERL Principles

Chapter Objectives In this chapter you will learn:

The four key interrelated processes found in a monitoring, evaluation, reporting, and learning (MERL) system

Key reasons for having a high quality MERL system

How MERL is embedded in the project cycle

Learning Activities

Defining peacebuilding for your project

What do you want to achieve with your peacebuilding project’s MERL system?

Checking and improving the project cycle

What is MERL?

MERL is a contemporary of the commonly used abbreviation, M&E. The “M” and “E” are the

same: monitoring and evaluation. The “R” and the “L” stand for reporting2 and learning. Pact uses this more-comprehensive terminology to emphasize that the four components are inherently linked. Without reporting on and learning from results, monitoring and evaluating programs is pointless. M&E as a standalone activity is like discovering that the brakes on your car do not work, but not telling anyone or getting them fixed. Learning about something is only useful if you apply what you learn.

Figure 1.1: The four separate but interrelated processes in a MERL system

MERL is:

A very important component of effective peacebuilding

A way for you to learn about what in your program works and what does not work

Even more important to conflict-based development initiatives than in stable environments: As the conflict and environment changes, so must peacebuilding activities and foci. MERL systems provide key insights into what has changed and what needs to be changed.

Ideally conceived at the same time as your peacebuilding approaches are developed; however, even if you are already in the midst of program activities, you can still develop useful MERL tools to help guide, improve, and assess your program

2 Of note, the “R” in MERL also can represent “Research;” however, for the purposes of this manual, the “R” will represent “Reporting.”



MERL is not:

A system separate from your programs and projects; it is closely interwoven with your project planning and implementation

Just for funder reporting

Meant to be a burden to program staff: Even when certain MERL activities are required by funders and parent organizations, your team should always work with them to develop processes that are mutually beneficial.

The job of an M&E specialist alone: MERL is a team effort and involves stakeholders from within your organization and the communities where you strive to build peace.

What is peacebuilding?

Have you ever tried to define the term peacebuilding? Defining peacebuilding can be

challenging, mirroring the complexity and range of activities needed to bring about peace. Despite the term’s complexities, most people working on this issue have a similar understanding of the change they want to bring about. Ending fighting, reducing potential for future conflict, improving relations between peoples and communities, and reducing injustice are common goals of peacebuilding programs.

Peacebuilding approaches can address the effects of violence or the root causes or drivers of conflict. Activities can range from direct interventions to stop fighting to initiatives aimed at improving education, health, and prosperity. In fact, peacebuilding can take place where there has not been violence: helping individuals understand how they can better live with their neighbors, creating just societies, and collaborating to make communities prosper are all part of peacebuilding.

Why is MERL important for peacebuilding?

Organizations and professionals working in the peacebuilding sector often state that they need quality information about their activities and the changes they bring about to improve their work and to meet the needs of communities. Yet, many organizations often make critical decisions based simply on personal and staff judgment, anecdotal information, or haphazardly collected field data.

Areas beseiged by conflict are subject to frequent changes, in the conflict itself and in the overall environment. Peacebuilding programs likewise need to be adaptable to meet the changing needs of people and institutions. But in order to do so, peacebuilding programs must have access to timely and reliable information, not just about the effects of their direct interventions, but about the evolving situation around them.

Developing and implementing a high quality MERL system allows program teams to access information to make good decisions before, during, and after program implementation. Through MERL, a program team can see if they are achieving their targets, whether their work is having its desired (or other) effects, and whether it is necessary to make corrections to the initiative. Also, programs can learn if cumulative efforts of multiple activities are bringing about the peacebuilding changes sought.

It is just as important to learn about what is not working as it is to learn about what is working. Monitoring helps to identify what is not working early so that program managers can make adjustments throughout the life of the program. Periodic formative evaluations have a similar purpose and can help programs understand why they see the results they see.



Learning Activity: Defining peacebuilding for your project

Understanding how you define peacebuilding is important not only to project design but to the project’s M&E system, as we shall learn in upcoming chapters. Consider your answers to the following questions.

1. In my project, peacebuilding is defined as: _______________________________

______________________________________________________________

______________________________________________________________

2. What MERL processes do we use currently in our peacebuilding programs? ________

______________________________________________________________

______________________________________________________________

3. Do our MERL processes align with our understanding of peacebuilding? ___________

______________________________________________________________

______________________________________________________________

Definitions

A MERL system is composed of four separate but interrelated processes: monitoring, evaluation, reporting, and learning.

Monitoring is the systematic process of collecting and analyzing information to

track the program’s performance in achieving goals and to observe changes in the conflict or environment.

Monitoring data can be used to measure how productively inputs (money, time, equipment, personnel, etc.) are used in the creation of outputs (products, outcomes, results) so a program team can compare what they planned to accomplish in a given period to what actually occurred in that time period.

Monitoring also can be used to keep track of the context and, specifically, the conflict in which program activities are occurring, as well as assumptions on which program activities are based. This type of monitoring is particularly important for program effectiveness because changes in conflict and context often necessitate changes in program activities in order to meet desired outcomes.

How program teams use monitoring data

Program staff use monitoring data to manage program implementation, budgets, timelines, and human resources. Monitoring data also is used as a basis for reporting to funding agencies in quarterly, semi-annual, and/or annual reports. Monitoring happens periodically throughout the life of the project and is ideally set up so key information about program efficiency is routinely delivered to program staff to help them actively manage the program.



Monitoring data helps project teams answer key questions, such as the following.

What activities have been completed (compared to those planned)?

At what cost and in what timeframe are activities being accomplished and how does this compare to what we planned for a given period?

What short term results have occurred, for example, how many people have been mobilized, trained, or reached in a given time period?

Have key benchmark activities been completed as planned?

Does data tell us anything about differences between activities and/or program sites?

What has changed around us? Are there reasons to change our upcoming plans?

Evaluation is the systematic collection and analysis of information to

determine the effectiveness of an intervention or of an organization’s achievement of peacebuilding goals.

Evaluation focuses on intervention outcomes and impacts3 (intermediate and long-term effects) and lets program teams know if change is occurring in a community or conflict situation. Evaluation also provides regular feedback that helps organizations assess their relevance, scope, and sustainability. In its simplest terms, evaluation is the collection and analysis of information to assess the impact of the organization’s work.

Like monitoring processes, evaluations can be conducted periodically throughout the project or program. And, while monitoring activities focus on what happened, evaluations look more in depth at how well things happened and whether they resulted in desired outcomes or are making progress toward achieving them. Evaluations often can be broken down into three phases:

Baseline: collecting values at the start of a program or intervention to be compared against at later points

Midterm: checking statutes against baseline figures and planning for changes to the approach, if necessary

Endline: measuring changes from beginning of the project to the end

Figure 1.2: Evaluation phases

3 It is important to note that in this context, impact assessment implies looking at long-term, broader-scale changes in the community that result from activities and processes among smaller populations within that community. This distinction is important because the term “impact assessment” is sometimes used to describe an evaluation whose goal is attributing change to a particular intervention. Such studies are highly controlled, costly, and difficult to implement, even in stable environments.



How program teams use evaluation data

Evaluation happens periodically, and information provided is used by the program staff to help them understand the short-, intermediate- and long-term results of their programs.

Evaluation data can help program teams answer key questions, such as the following.

Are activities effective in achieving desired outcomes (short- and long-term)? Are we heading in the right directions?

Are our approaches as efficient at achieving outcomes as they could be (or is there a better way)?

What are the secondary- and highest-level effects of our interventions? Are our efforts resulting in changes that are leading to stability and peacebuilding?

Will changes persist after our direct influence has left? Will our interventions’ positive changes and effects be able to adapt over time, as needed, to fit the local context?

What unintended outcomes have occurred?

Program teams can use evaluation data to analyze what is working well to maintain, expand, or replicate it and to review what has not worked well to make program course corrections, avoid repeating mistakes, and improve the quality of their initiatives.

Reporting is the systematic and timely provision of useful information at

periodic intervals.

Reporting provides regular feedback that helps organizations inform themselves and others (stakeholders, partners, funders, etc.) on the progress, problems, successes, and lessons of program implementation. M&E report writers have the responsibility of conveying relevant information to various audiences in effective ways. The challenge for MERL teams is to turn raw data into useful knowledge, then communicate that information to the different program audiences (community members, program staff, partners, etc.) in ways that will most effectively meet their needs and enable them to use it as needed.

MERL teams should think of reporting as broader than simply providing written updates to a funder. Rather, the role of reporting in a MERL system is to ensure that data and analysis is being distributed to those who can use the information to better manage the program, provide oversight, or reflect on the events and outcomes. Effective reporting at regular, strategic intervals is critical for informing program staff, funders, partners, program participants, and other stakeholders of the progress of peacebuilding interventions. Reports can take many forms, can be formal (reports, spreadsheets) or informal (phone calls, quick briefs), and should communicate successes and failures, positive and negative results, and any and all lessons learned.

When creating reports and a reporting schedule, MERL team members should consider:

Who needs it

What specifically they need and why

When and how often they need it

Developing a matrix that details this information is immensely helpful to appropriately plan for reporting. An example of this matrix is presented in Table 1.1.



Table 1.1: Example of MERL audience information needs assessment

MERL

Audience

WHAT data they need from our organization

WHY they need the data

WHEN they need the data from our organization and what reporting format is best

How this audience can participate in the MERL program

Internal Audience

Board members

Updates on:

Activities, projects, and programs

Management issues

Financial issues

Resources (inputs)

Lessons learned on program impacts

To monitor the efficiency and effectiveness of the organization and its people, products, and impacts

To provide guidance

To problem solve

To help us fundraise

Monthly memo and financial statement to CEO

Quarterly meeting presentation (January, April, July, and October)

Annual report and financial statement (December)

Review project design and objectives

Review performance data specifically to evaluate the efficiency and effectiveness of the project

Use the data to guide their decision-making

External Audience

Community members

Context/conflict analysis results

Updates on peacebuilding efforts and community participation in ongoing efforts

Outcomes resulting from interventions, such as peace dialogues and peace dividends

To stay updated on the status of peace in their community

To understand their contribution to peacebuilding efforts

To understand the longer term changes in the peace situation and implications for them

Quarterly presentations at community briefing meetings (January, April, July, and October)

Participate in a situation analysis

Review project design and objectives

Participate in project assessments/ evaluations, e.g., as respondents in focus group discussions and key informant interviews

Share their stories of change

Learning involves internalizing information and analyzing it such that it can be

of use in decision-making. Program management and other stakeholders should use the data generated from their M&E processes to review progress, learn from what is working or not working well, and make decisions on the required changes for successful programming.

Learning is a deeper process than simply reading reports; it involves interpreting M&E results and taking action based on what is learned. Learning is the systematic review of achievements against desired results and is used to change course or modify approaches if necessary. Learning benefits the current project and future projects.

Project technical specialists and MERL team members must participate in planning M&E activities because they all are responsible for learning from the results, including successes and failures, and be willing to accept unexpected results and take action to resolve problems that arise. It is also important to involve other MERL audiences in planning and/or review of MERL plans: program staff, funders, community partners, and others.

During the planning phase, it is important to think about who will be responsible for decision-making regarding changes to be made as a result of learning. Developing a learning



schedule also can be helpful to ensure project staff hold regular learning reviews within your organization and externally with relevant stakeholders. Also think about your program design: If we wanted to change something in the future, could we? How easy or difficult would this change be? Is there a point at which there’s “no turning back” (implying you should learn about the program’s effectiveness before then)?

How to determine if your program is prepared to learn from M&E activities

Are we generally willing to examine our work and challenge what we know and do?

Are we testing our assumptions? Are our assumptions well founded to begin with?

When faced with negative feedback, do we react in a constructive way?

Is our knowledge being shared? Are regularly scheduled learning reviews being held?

Is our knowledge accessible to all key stakeholders? To the wider development and peacebuilding communities?

Is our learning relevant to our programming and ultimate goals or is it limited in focus to funder requirements?

Who is on a MERL team?

Pact encourages all partners to have at least one full-time MERL officer to lead and coordinate MERL activities. But, that does not mean that the MERL officer is the only staff member involved in MERL activities; in fact, it is just the opposite. All program staff play a role in determining what should be measured, assisting with quality data collection efforts, analyzing findings, and, most importantly, using the data to improve program implementation and peacebuilding practices. Community members and partners also play critical roles, helping shape the focus of MERL and offering advice on effective local MERL practices. MERL efforts are team efforts.

Key Reasons for Developing Quality MERL Systems

MERL systems provide peacebuilding programs with various benefits.

MERL provides managers with a tool to gain timely information on the progress of project activities, allowing them to compare what was planned to what is actually occurring.

MERL provides organizations with a tool to evaluate how effectively the organization promotes change.

MERL fosters public and political cooperation and supports particular information needs for target audiences (such as funders, community members, staff, board members).

MERL enables organizational learning and encourages adaptive management.

Other benefits of MERL include ensuring that our projects, services, and activities:4

Meet community needs

Reach those they are intended to reach

Make adequate, timely progress toward our objectives

Make efficient use of resources (capital, human, and other)

Yield results that are in line with our efforts

Are sustainable5

Are appropriate for the changing environment (political, social, conflict context, etc.)

4 Social Impact 2009.

5 For more resources on sustainability, see http://microlinks.kdid.org/library/search/sustainability.

http://microlinks.kdid.org/library/search/sustainability



Effective MERL also benefits institutions in developing future programs in the same or another context. Learnings from a particular project can be used to develop future initiatives during planning, design, and implementation and even at the proposal writing stage. Funders may use past MERL results to evaluate the suitability of your organization or your particular approach to a proposed activity.

Commonly, MERL is viewed as complex, time intensive, and costly. Some organizations also consider MERL to be simply a requirement of funders and thus see MERL as an external rather than internal necessity. Therefore, an effective MERL system is organized and focused, affordable, time efficient, and useful to program staff, funders, and other key program stakeholders.

Learning Activity: What do you want to achieve with your peacebuilding

program’s MERL system?

Throughout this manual, you will learn the steps involved in designing and implementing your MERL system, including choosing your goals and the focus of your assessments. A good way to start is to brainstorm a list of specific information or types of information that would be helpful to you in carrying out a successful program. Do not worry yet about how or when you would get the information.

You can do this activity with a group from your project team.

1. Ask several staff members to brainstorm their information desires on their own. You can ask them the following questions to help shape their brainstorming.

a. What would you most like to know about the community(ies) in which you work?

b. What would you like to learn about or from others who have worked in this community or who have tried similar approaches to peacebuilding?

c. What uncertainties do you have (about the community, the context or conflict, the [proposed] program design)? What would you need to know to be certain?

d. What are the biggest misconceptions you think others have? What information would help to correct those misconceptions?

2. If possible, bring in voices from the community and from your funder and/or parent organizations. At minimum, assign someone to create a list of desired information from the perspective of funders and/or community members.

3. Organize a meeting in which each person shares their lists with a goal of determining the most common informational needs and prioritizing the most important needs.

4. Be sure to consider priorities for the organization and project as a whole and for different groups within your team.

5. Assign one person to record both the entire list of brainstormed ideas and the prioritized list.

6. Refer back to these lists during program and MERL design.



MERL in the Project Cycle

It is important to fully understand the project cycle of peacebuilding projects. The project cycle is the process through which projects are conceived, developed, tested, and adjusted based on what happens. It emphasizes the iterative nature of projects and how we must always be willing to adjust and adapt based on successes and failures. This is especially important in peacebuilding initiatives. There are many variations on the project cycle, but all include four main parts, as shown in Figure 1.3. Below is a quick review of the project cycle and some key focus points for MERL work.

1) Project identification

At this stage, as someone who works on peacebuilding, you probably are looking at the conflict in an area and hoping to find a way to improve the existing situation. You develop your visions of peace and peacebuilding. You analyze the conflict, context, and most pressing needs.6 You start to identify the goals and objectives you will ultimately try to achieve and which will be the focus of your MERL system.

Once you understand what is happening and what is needed, you determine what your organization is best suited to do about it. This will likely be done in collaboration with other local organizations, government offices, or other partners. Together you will decide what your intervention will be.

2) Project design

In the design phase you take your idea of what to do and determine more specifically how to do it. Project design is about coming up with a good Theory of Change (see Chapter 4) and developing specific strategies (and activities) for bringing about the desired changes. Project design includes identifying resource needs (funds, time, personnel, equipment, etc.) to complete the program successfully and establishing the project’s basic framework and MERL system.

3) Project implementation

The majority of project activities will take place in this stage. A baseline study is usually conducted just before a project begins to establish your starting point and for later comparison. Monitoring activities take place regularly throughout the life of the project to track progress and resource allocation.

4) Project evaluation

Periodic evaluations can be used to assess ongoing progress toward peacebuilding goals. What you learn from M&E activities leads you back to program design and implementation to make adjustments or initiate new activities as indicated. A comprehensive project evaluation is an important activity to carry out once a project has concluded. It is a way for you to learn what ultimately worked and what did not work, to inform others who remain (if your activity has ended), and to guide future initiatives.

6 Conflict, context, and needs assessments are discussed in more detail in Chapter 3.

Figure 1.3: Project cycle



MERL in Summary

Figure 1.4 summarizes the MERL processes that will be covered throughout this module.

Figure 1.4: MERL in summary

Summary: Key points from this chapter

Definition of peacebuilding: Ending fighting, reducing potential for future conflict, improving relations between peoples and communities, and reducing injustice are common goals of peacebuilding programs.

Definition of MERL (monitoring, evaluation, reporting, and learning) and its importance for peacebuilding projects

MERL is a team effort! Program and MERL staff must work together for successful implementation.

When preparing reports, it is important to consider the “who, what, when, how, and why” of the end user to package the reports in a useful way for the end user.

It is important to understand the project cycle of identification, design, implementation, and evaluation when designing peacebuilding projects and their MERL systems.



Learning Activity: Checking and improving the project cycle

In this exercise, you will use your past experience following a project cycle to determine how to improve the process. Refer to the detailed project cycle in Figure 1.5 as you complete this activity.

Think of a project you have worked on or you have seen implemented. See if you can list major activities or processes that happened during:

Project identification

Project design

Project implementation

Project evaluation

(If you could not differentiate between the different phases of the cycle, that tells you something, as well: there is a need to review what processes you are following in identifying, designing, implementing, and evaluating your program to determine which aspects of the project cycle need to be strengthened!)

What did you think went well in this project? Is there anything you would have changed?

Figure 1.5: Detailed project cycle


Chapter 2: Peacebuilding Approaches: Defining What Peace Looks Like | page 13

Chapter 2: Peacebuilding Approaches: Defining What Peace Looks Like


Why defining your peacebuilding approach is critical to program design and monitoring, evaluation, reporting, and learning (MERL)

About three specific approaches to peacebuilding that your project might employ

Pact’s general approach to peacebuilding

Learning Activity

Identify and write your peacebuilding approach

What are peacebuilding approaches?

Peacebuilding approaches describe how your project will go about realizing its vision for peace, i.e. “what peace will look like.”

As noted in the previous chapter, peacebuilding approaches can address the effects of violence and/or the root causes or drivers of conflict. Activities can range from direct interventions to stop fighting to initiatives aimed at increasing education, health, and prosperity. In fact, peacebuilding also can take place where there has not been violence, helping individuals understand how they can better live with their neighbors, creating just societies, and collaborating to make communities prosper are all part of peacebuilding.

Why is this a starting point for talking about MERL? Because we can’t measure our progress toward achieving ultimate goals without setting them. Defining your approach to peacebuilding effectively declares what peace should “look like,” thus giving you goals to monitor progress toward throughout the life of your project.

A MERL system is a tool that organizations and managers use to see if they are achieving change. Understanding how programs define peacebuilding is essential to program staff because it tells us about the critical changes anticipated that will need to be monitored and evaluated during the life of program to see if those changes have been brought about.

Research on effective and ineffective peacebuilding initiatives has shown that lacking a clear vision of what peacebuilding looks like is a significant barrier to success. Peacebuilding approaches that lack a unifying ultimate goal (i.e., “what peace looks like”) are often less effective than those that have a specific aim in mind.

Below are three examples of approaches to peacebuilding. Your intervention may fit one of these approaches or something different. Consider both individual programs (“What is the ultimate peacebuilding goal that this effort hopes to achieve?”) and the collective efforts of your organization in a particular region (“Are our programs united by a singular view of how they will contribute to peacebuilding here?”).

When asked what they most want to learn about their programs efforts, program staff often say that they want to know whether they “work.” To answer this question, we must first know what the project would look like if it were, in fact, working. This becomes the basis for assessment. The more detail the staff can identify about what “working” looks like (specific goals and objectives) the more easily they will be able to determine if it is doing so.



Approach 1: Incentives for Peace

Some peacebuilding projects define their work as being focused on incentives: either encouraging change by providing incentives for peace or by eliminating incentives for violence. Incentives either induce action or motivate one’s efforts or behavior.

Projects that want to increase the incentives for peace might reward countries or communities with aid if an agreement is reached, increase trade between countries when a conflict ends, or develop initiatives to open borders when a level of security is reached.

Projects that want to eliminate incentives for violence might increase legitimate control over natural resources that fuel conflict, try to make it more difficult for parties to acquire power by force or ease, create zones of peace, control borders to make it more difficult to acquire goods that are needed for war, or seek to make weapons less available.

If, for example, a program takes an eliminating incentives-based approach to peacebuilding, we might anticipate seeing MERL programs that focus heavily on measuring changes that will result from making it more difficult to acquire weapons and goods needed for war or offering amnesty to individuals who voluntarily surrender their weapons. As such, we might measure the number of people disarmed, extent of weapons availability, or average cost of a gun at an open market.

Approach 2: Addressing Root Causes of Injustice

Peacebuilding programs that define peacebuilding as addressing the root causes of injustice in a community include those that:

Aim to empower people to use non-violent means to reduce injustice, such as advocacy projects, human rights education projects, and projects to reform government institutions

Include long-term campaigns for social and structural changes, such as truth and reconciliation initiatives, and changes in social institutions, laws, regulations, and economic systems, among others

Aim to address the underlying causes of injustice, oppression, exploitation, threats to identity and security, and people’s sense of victimization, including projects for economic development, decentralization, self-help, education, and health

If a program defines peacebuilding as addressing root causes, then MERL systems would focus heavily on capturing social and structural changes in society and people’s lives. Some example changes we might study include the number of people reporting an improvement in the level of security, perceived or legislated level of independence of the judiciary, the number of arms passing through border points, or the number of development projects in resource-poor or marginalized communities.

Approach 3: Addressing Individual Attitudes and

Relationships and Building Trust

Peacebuilding programs that define peacebuilding as addressing individual attitudes and relationships and building trust include those that:

Encourage dialogue through activities such as roundtables, peace meetings, mediation efforts, and other events where multiple parties gather to discuss the conflict or environment in general



Encourage coexistence and include joint problem solving projects, multi-ethnic community and economic development initiatives, training and community events where parties at conflict attend, and promoting collaboration between parties in conflict

Train individuals and groups on ways to deal with differences, including conflict resolution and negotiation training events, community mediation training and set-up assistance, justice initiatives, and non-violence training events

If a program defines peacebuilding as addressing individual attitudes and relationships and building trust, then MERL programs would focus on assessing changes in perceptions, attitudes, and values at the individual, community, and institutional levels. Changes might include perceptions of whether co-existence has improved between communities/parties, the number of people reporting an improvement in the level of security, the number of joint initiatives between conflicting parties, and perceptions of tolerance among conflicting communities.


Defining your peacebuilding approach is important for developing a good MERL system because the former is based on how the project describes what they think “peace looks like.” This forms the basis for developing the project goal and, thereafter, how the project will monitor and measure achievement of this goal.

Research on effective peacebuilding has shown that lack of a clear vision of what peacebuilding looks like is a significant barrier to the project’s success!

Three types of peacebuilding approaches are: 1. Incentives for peace 2. Addressing root causes of injustice 3. Addressing individual attitudes and relationships

Pact’s approach to peacebuilding uses a combination of the “addressing root causes of injustice” and addressing individual attitudes and relationships” approaches.

Learning Activity: Identify and write out your peacebuilding approach

Refer to the three peacebuilding approaches discussed in this chapter. Do any of them apply to your project? If yes, identify which one and describe how your project uses this approach. If no, what approach do you use in your project? Describe it.

Name of project: __________________________________________________

Peacebuilding approach: _____________________________________________

Description: _____________________________________________________

______________________________________________________________

______________________________________________________________

______________________________________________________________

______________________________________________________________

______________________________________________________________


Chapter 3: Situation Analysis | page 16

Chapter 3: Situation Analysis

Chapter Objectives In this chapter you will:

Learn the basics of conducting situation analyses by reviewing three types: conflict analysis, context analysis, and needs assessment

Learn about situation assessment’s relevance to program design and routine monitoring

Learning Activity

Carrying out situation analyses

What is situation analysis?

In the project identification stage of the project cycle, your aim is to choose the best way to improve the situation in the communities, region, or country where you work. In most cases you do not yet know exactly what project you will implement. In other cases, you have an idea of the project you want to implement, but want to make sure that the idea you have is appropriate for the situation and will have the desired effects.

It is critical to understand the specific conflict, context, and needs of the communities in question to avoid misguided program activities or poor indicators for your project. If program activities are misguided, then the indicators developed to measure the results of these activities will not be well aligned to the project objectives and cannot be used to demonstrate achievement of results. Because of the importance of these preliminary situation analyses, we review them here, with special attention paid to how they relate to MERL processes.

By the time you are done carrying out a situation analysis, you will have a clearer picture of the situation on the ground in the area you hope to affect, as well as the larger national or regional scope of the situation. And, you should be keenly aware of what others are doing to address the conflict.

Using this information, the project team can develop strategies (and related activities) to bring about the changes sought. Some basic questions for reflection are:

What would make the challenges to peace diminish?

What can we do (with our partners and stakeholders) to increase successful peacebuilding?

Types of Situation Analysis

Situation analyses are key to finding answers to these questions. In this chapter, we will look at three examples of situation analysis that programs can carry out:

Conflict analysis

Context analysis

Needs assessment

Table 3.1 summarizes the types of situation analyses, when they are conducted, and how to carry out a situation analysis. Detailed descriptions of each type of analysis follows the table.

It’s never too late to do situation analysis. Even if you are in the middle of program activities and have never taken a formal, close look at the conflict around you, take some time to update yourself and your organization on the current state of things.



Table 3.1: Types of situation analysis

Situation analysis type What is it?

When is it conducted?

How is it conducted?

Conflict analysis Examines the root causes, drivers, nature, and primary actors of a conflict

Ongoing through the life of the project

Asking key questions during:

Key informant interviews

Group assessments

Context analysis7 Analyzes the political, economic, and social systems and actors in a system, even if they do not have a role in the existing conflict8

Ongoing through the life of the project

A review of the existing systems and key actors


Group assessments

Needs assessment Identifies gaps in systems or services

Proposal stage or months 1–4 of the project

Desk research


Group assessments

Conflict analysis examines the root causes, drivers, nature, and primary

actors of a conflict with the goal of gaining an in-depth understanding of the conflict’s dynamics.

Some of the key questions often incorporated into conflict analysis are:

Who are the key actors/stakeholders in the conflict?

What are the main causes of the conflict? What is dividing certain groups and uniting others? How did it start? Are there different driving factors now?

What are the economic, political, and socio-cultural contexts of the conflict at the local, national, regional, and international levels?

What are the current conflict trends? What is happening?

What are the current windows of opportunity to address the conflict?

What efforts are currently in place to build peace/end the conflict?

What has been successful in reducing the conflict? What has failed?

Understanding what caused the conflict to start and what is causing it to continue is important if we are to find ways to mitigate it and build peace. However, because of the dynamic nature of conflict, it is important to regularly re-assess the conflict to see if the drivers have changed or if new barriers to peace have emerged. Ongoing conflict assessment focused on the factors most likely to change should be part of the MERL process.

7 One type of context analysis is political economy analysis (PEA). The Organization for Economic Co-operation and Development (OECD) defines PEA as being concerned with the interaction of political and economic processes in a society: the distribution of power and wealth between different groups and individuals and the processes that create, sustain, and transform these relationships over time. PEA is used to understand the explicit legal, policy, and economic frameworks and the implicit and unwritten norms, values, and interests that help determine how individual and group actors behave. See DFID. July 2009. Political Economy Analysis How to Note. Available at https://www.odi.org/sites/odi.org.uk/files/odi-assets/events-documents/3797.pdf.

8 Pact’s Applied Political Economy Analysis (APEA) aims to provide project teams with practical help in identifying and mapping the political, economic, and social incentives that influence key stakeholders’ actions and decisions. This project- or problem-based methodology is designed to directly support project decision-making. A full description of the approach is available at at http://www.pactworld.org/library/applied-political-economy-analysis-tool-analyzing-local-systems.

https://www.odi.org/sites/odi.org.uk/files/odi-assets/events-documents/3797.pdf

http://www.pactworld.org/library/applied-political-economy-analysis-tool-analyzing-local-systems

http://www.pactworld.org/library/applied-political-economy-analysis-tool-analyzing-local-systems



Conflict analysis and the project cycle

Project cycle management is closely related to conflict analysis. While analysis of project-level data deals with a wide range of project issues, conflict analysis deals with broader conflict-specific issues that the project aims to address. While the two are complimentary, they are distinctly important to designing a comprehensive project-specific MERL system. While conflict analysis helps to focus on the changes the project wants to achieve in the conflict setting—a transformation from conflict to peace—analysis of the project’s design and implementation helps the organization establish sound operational systems and processes to support achievement of anticipated changes. At the same time, the MERL system is used to assess the efficiency and effectiveness of actions needed to bring about the changes sought.

Context analysis examines existing political, economic, and social structures

at the local, regional, and national levels (or across a larger area if the conflict crosses state borders).

Conflict analyses look at these same systems and structures, but with a focus on those directly affecting the conflict. Context analysis looks more broadly at these systems, even if they do not have a direct role in propelling the conflict. Such factors are important because they impact the way people live their daily lives and are subject to being brought into or affected by the conflict. Assuming your intervention will affect people’s daily lives, it is important to understand all of the factors that impact their habits, activities, interactions, and livelihoods.

Some of the key questions often incorporated into a context analysis are:

What policies, systems, structures, and forces exist inside and outside the immediate local context (village, district, province, nation) of the conflict?

How can the policies, systems, structures, and forces outside the immediate local context be addressed?

What kind of cooperation or linkages between the local, national, regional, and international levels need to be made?

Who are the key actors that can be used to leverage this cooperation (who could be the project’s “champions”)? Who may want to see the project fail (who may be the project’s “spoilers”)?

Where does our organization fit within the context? On what systems do we rely? With whom are we directly aligned?

Like conflict analysis, context analysis needs to be regularly updated to monitor changes. Context analysis often includes looking at systems not directly linked to the conflict. As such, it is important for peacebuilding programs to continuously monitor the emergence of new conflict drivers and how these affect their program context.

Needs assessment involves finding out what services or systems are missing

or require improvement in a given context.

Needs can be assessed on a large scale (national level) or more locally, depending on the scope of your intervention, keeping in mind the impacts local changes have on the larger scale. Such assessments help to determine what has to happen to move from the current toward the desired state of things. The project team’s goal is to develop strategies (and related activities) to fill the gaps and meet needs.



Some of the key questions often incorporated into a needs assessment are:

What are the gaps/needs for the community or target group?

What action is being taken to address them? What more is needed?

How is the community currently trying to meet its needs?

How can the challenges be addressed?

What resources are currently available within the community to address needs and gaps?

What barriers have prevented these needs from being met before?

Ongoing monitoring can help program staff track when needs have been met, changed, or reprioritized. As with other analyses, this requires close collaboration with multiple community members, leaders, experts, and others with direct local knowledge.

Institutional capabilities

Alongside your situation analysis, it is important to constantly consider your place: your own institution’s capabilities, strengths, and potential role in peacebuilding. Consider your organization’s history in the community and region, partnerships, staff capacity, interests, expertise, and any other factors you feel would strengthen or weaken your institution’s ability to carry out successful peacebuilding efforts.

Disseminating Situation Analysis Findings

It is important to disseminate situation analysis findings because they can inform what other stakeholders are doing and identify opportunities for collaboration. Dissemination of findings can be done via different media, including:

Dissemination workshop

Circulating summary reports to targeted stakeholders

Posting on the website

Community events

Periodic newsletter

The process used will depend on resources available, target audience, and purpose of the dissemination. For instance, a dissemination workshop would be ideal to collect input from stakeholders or validate findings.


Situation analysis helps peace-builders understand on-the-ground realities of the conflict environment.

Situation analysis can be done in different ways, including conflict analysis, context analysis, and needs assessment.

It is important to disseminate situation analysis findings, usually to targeted stakeholders.



Learning Activity: Carrying out situation analyses

Situation analyses can be time consuming and complex. A good first step is to look for existing analytical reports, data collection instruments, surveys, or other sources of information to help guide your conflict analysis, context analysis, or needs assessment.

When looking to these other sources of information (either within your local organization or from other development agencies), take into consideration the following.

What factors affect the conflict your project is designed to address?

Who may be the “champions” or “spoilers” of peace in your project areas?

What is your vision for peace?

What changes are needed in individuals, communities, institutions, organizations, cultures, and/or societies to realize your vision of peace? Refer to Table 3.2.

In your groups, respond to the above questions as they relate to your project. Make note of where there may be existing data to answer these questions or where you may need to undertake your own situation analyses.

Table 3.2: Examples of specific changes9

Type of change Examples of specific changes

Relationship From adversaries to partners in problem solving

From suspicion to solidarity

From different ethnicities to a common nationality

Former neighbors reconciled

Status From soldier to veteran

From rebel leader to parliamentarian

From criminal to entrepreneur

Behavior From violent behavior to assertiveness

From disrespecting women to respecting women

From ignoring youth to taking their interests into consideration

Circumstance From politically marginalized to able to vote

From displaced victim to community member

Functioning Increased transparency

From authoritarian to consultative policy development

Increased cost efficiency

Attitude Greater tolerance of different perspectives

From fear of others to trust in others

From apathy and fatalism to hope and self-determination

From a narrow focus on the neighborhood to a broad focus on inter-communal interests

Knowledge Understand interdependence of groups

Understand how globalization affects local livelihoods

Understand rights and how justice systems should work

Knows how political resources are allocated

Skills From power-based to interest-based negotiations

Moving discussions from mutually exclusive interests to framing issues in mutually acceptable language

Able to introduce items onto the agenda in local governance

9 This table was taken from Church & Rogers, 2006, pp. 18–19, which was adapted from M.Q. Patton. 1997. “Outcome Examples.” Utilization-Focused Evaluation (3rd ed.). Thousand Oaks, C.A., U.S.: Sage, Inc.



Type of change Examples of specific changes

Maintenance Continue to celebrate cultural heritage

Maintain existing social cohesion

Continue to practice traditional dispute resolution processes

Prevention Peaceful transfer of power

Increase awareness of military accountability to civilian ministries

Prevent exodus of trained and educated professionals

Process From shuttle diplomacy to face-to-face negotiation

From hate-mongering to balanced reporting

From divisive methods to methods that bring people together

From concentrations of authority over others to equitable engagement with others


Chapter 4: Theories of Change and Developing Project Goals | page 22

Chapter 4: Theories of Change and Developing Project Goals


Learn what Theories of Change (TOCs) are and how to develop them

Review several TOCs that can be used in evaluating peacebuilding programs

Review several conceptual models

Learn the importance of testing assumptions

Understand the relevance of conceptual models to program design and monitoring, evaluation, reporting, and learning (MERL)

Learning Activity

Describe your project’s TOC

What are Theories of Change?

TOCs are a set of beliefs about how change happens.10 They form an important part of a project because they are used to describe how the micro-level aims (activities/outputs) will have a macro-level result (outcomes/impact). In essence, TOCs pay attention to how the immediate aims of the project can be linked to how society can achieve sustainable peace and how progress along both these paths can be measured.

TOCs help guide thinking during the project design and evaluation stages. Peacebuilding practitioners select project goals, methods, approaches, and activities based on underlying theories of how peace can be achieved in a specific context. Effective projects usually clarify their TOCs early in the life of the project and continually test them against on-the-ground realities.

TOCs can be used for several reasons within a project:

To reveal and understand project assumptions: Generally, two assumptions are inherent in every TOC: 1) how change works and 2) the strategic advantage of the chosen theory over other theories for the context. By linking the immediate or internal project goals, the TOC develops an idea and tests the assumptions of how it will contribute to change in society.

To ensure alignment with all levels of the program design when setting goals and objectives: TOCs explain the basis of change behind particular choices/decisions, thereby tightening the program logic and enabling identification of gaps/unmet needs.

10 Church & Rogers, 2006, p. 118.

Definition of “theory”

An assumption about how something works, or a prediction of what will happen as a result of an action (Lederach et al., 2007, p. 4)



Theories of Change in the Context of Peacebuilding

Programs

Table 4.1 summarizes 10 TOCs that can be applied to peacebuilding programs.

Table 4.1: Peacebuilding Theories of Change11

Theory Basis of theory Methods/intervention activities

Individual change

Peace comes through transformative change of a critical mass of individuals and their consciousness, attitudes, behaviors, and skills.

Invest in individual change through training

Personal transformation/consciousness-raising workshops or processes

Dialogues and encounter groups

Trauma healing (TH)

Healthy relationships and connections

Peace emerges from a process of breaking down isolation, polarization, division, prejudice, and stereotypes among and between groups.

Processes of intergroup dialogue, networking, relationship-building, joint efforts, and practical programs on substantive problems

Withdrawal of the war resources

Wars require vast amounts of resources (e.g., weapons, supplies, transport) and human capital. If the project can interrupt the supply of people and goods to the war making system, it will collapse and peace will break out.

Anti-war campaigns to cut off funds/national budgets

Conscientious objection and/or resistance to military service

International arms control

Arms embargoes and boycotts

Reduction of violence

Peace will emerge as we reduce the level of violence perpetrated by combatants or their representatives.

Cease-fires

Creations of zones of peace

Withdrawal from direct engagement

Peacekeeping forces

Observation missions

Promotion of non-violent methods for achieving political, social, and economic ends

Root causes/justice

Peace can be achieved by addressing the underlying issues of injustice, oppression, exploitation, threats to identity and security, and people’s sense of injury/victimization.

Long-term campaigns for social change, truth, and reconciliation changes in social institutions, laws, regulations, and economic systems

Institutional development

Peace is secured by establishing stable/reliable social institutions that guarantee democracy, equity, justice, and fair allocation of resources.

New constitutional and governance arrangements/entities

Development of human rights, rule of law, and anti-corruption

Establishment of democratic/equitable economic structures

Economic development

Democratization

11 See Church & Rogers, 2006, pp. 14–15. The text on those pages was used to create Table 4.1.



Theory Basis of theory Methods/intervention activities

Political elites Peace comes when it is in the interest of political and other leaders to take the necessary steps. Peacebuilding efforts must change the political calculus of key leaders and groups.

Raise the costs and reduce the benefits for political elites to continue war and increase the incentives for peace

Withdraw international support/funding for warring parties

Grassroots mobilization

“When the people lead, the leaders will follow.” If we mobilize enough opposition to war, political leaders will have to pay attention.

Mobilize grassroots groups either to oppose war or to advocate for positive action

Use of the media

Non-violent direct action campaigns

Education

Mobilization effort

Organizing advocacy groups

Dramatic events to raise consciousness

Economics People make personal decisions, and decision-makers make policy decisions based on a system of rewards/incentives and punishments/sanctions that are essentially economic in nature. If we change the economics associated with war-making, we can bring about peace.

Use of government or financial institutions to change supply and demand dynamics

Control incentive and reward systems

Boycotts

Public attitudes

War and violence are partly motivated by prejudice, misperceptions, and intolerance of difference. We can promote peace by using the media to change public attitudes and build greater tolerance in society.

TV and radio programs that promote tolerance

Modelling tolerant behavior

Symbolic acts of solidarity/unity

Dialogues among groups in conflict, with subsequent publicity

There are several reasons why monitoring, evaluating, and reporting on peacebuilding projects is more difficult than other types of development. Some of these challenges include the different definitions of peacebuilding, the different theories of what causes conflict and brings about peace, the complexity of conflict, and the need to understand the conflict before a peacebuilding project can start. In addition, the general lack of security and resultant lack of trust of people interviewing them about issues in their community is also a challenge in these types of programs.

When monitoring and evaluating a peacebuilding project, the difficulties named above lead to a greater problem: frequent disagreement about the ultimate ideal outcome(s). Not all people agree on what peace looks like, thus it may be difficult to come to consensus on your ultimate goals. If your ultimate goals are not clear, it is impossible to assess whether you have reached them.

A second difficulty in evaluating a peacebuilding project is the question of attribution. How do projects, for instance, attribute a desired TOC to their interventions? Were there other factors such as environmental issues or a change in governance system that can be attributed to a desired TOC alongside the project’s interventions?



Despite these challenges there are steps to take that can help you develop your TOC. Conducting a situation analyses for instance can help to identify what other factors in the environment are likely to affect your TOC and how the project will account for them.

Develop Your Peacebuilding Project’s Theory of Change

After completing a thorough situation analysis, you have a clear view of the factors affecting the conflict. Among other things, you should understand the following three critical elements. 1. What is making the conflict worse? 2. What conditions or factors could reduce the conflict? 3. What are the ongoing peacebuilding activities/efforts in the target areas, and how can

your organization best contribute to these ongoing efforts?

In this context, TOC describes processes that you believe will lead to peace.12,13

For example, suppose that the situation analyses identified tensions between ethnic groups in a particular community as being a driving force of conflict. Because these groups cannot get along (manifesting in a number of problems with service delivery/provision and violence), your peacebuilding strategy involves changing attitudes of and relationships between individuals in these groups. You believe that changing people’s attitudes about people of “the other” ethnicity through education will encourage peace on a larger scale. Your TOC clearly spells out what you perceive to be the link between what you plan to do (educate) and what you expect to happen (peace).

Your TOC can be articulated in various formats depicting “IF we do this, THEN this will happen.” Here, we depict this TOC as a model and as a sentence.

Figure 4.1: Example of a peacebuilding program Theory of Change

The above TOC example falls under the “Individual Change Theory.” As is always the case when adapting ideas from general frameworks, it is important to base your TOC within the specific context in which you are working.

Points to remember

1. Your TOC does not have to reference specific program activities at the outset, though the program activities you later choose should fit into the TOC you have established. In the above example, “Changes in attitudes of educators and youth views of ‘the other’” will lead to “decreased levels of conflict and violence in the community,” among other effects.

12 See Church & Rogers, 2006, p. 118. There are also others who look at a TOC as a conceptual model and use it much like a results framework or logical framework. This is not the meaning used in this chapter.

13 There are a number of resources available online with more detail and examples of TOCs. One such resource is the Center for Theory of Change: http://www.theoryofchange.org/.

http://www.theoryofchange.org/



2. Being able to write your TOC in a clear and convincing manner is important and may take time. Prepare to have your TOC challenged. In the previous example, people may take exception to the third part of the theory, that teaching young people to be more tolerant will in-turn make their adult family members more tolerant. Similarly, if your TOC does not reference specific or general program activities, you will eventually need to include more details about how change will occur. When confronted, you must be prepared to defend your TOC and/or amend to ensure it’s clear and convincing to project stakeholders. One way to defend your TOC is to cite existing supporting evidence of the logic applied in your TOC. A defendable TOC should also have plans for collecting data that tests your theory and if the data disproves your theory, the program should be ready to adapt your programming to reflect the reality.

3. You should make sure your TOC is reflected throughout your project.14 Your activities and events, their outputs and outcomes, should be aligned to your theory of how peace will be built. If your project has several objectives, you will likely need multiple TOCs to describe the expected changes under each objective. As your program activities develop, return to your TOCs to make sure they fit. If not, develop new TOCs or adjust program activities to be aligned. It may seem obvious, but it also is important to make sure you do not adopt conflicting TOCs.

Some additional examples of TOCs from peacebuilding projects are as follows.

IF communities have a greater understanding of the drivers of conflict and violent extremism, THEN they will be more likely to engage in processes that build social capital and cohesion and that address community grievances.

IF networks of community peace leaders are strengthened to work collaboratively with each other and local government, THEN local cross-border conflict management will be strengthened.

Develop Project Goals and Outcomes from Your Theory of

Change

Once you have developed your TOC, the next steps are as follows.

1. Map your theories to strategic goals.

Your TOC says that doing X will lead to Y (directly or indirectly). Therefore, Y represents your project goal.

Goals, when achieved, should point directly to your ultimate vision of peace.

2. Map your theories to specific types of change.

Types of change include (but are not limited to): attitude, knowledge, skills, behaviors, maintenance, prevention, process, and structure.15

Changes can be at the individual, group, or institutional levels.

14 Church & Rogers, 2006, p. 14.

15 Church & Rogers, 2006, pp. 18–19.

Developing and explaining a TOC

This is often overlooked in peacebuilding initiatives, yet it is essential for both developing initiatives with a clear peacebuilding aim and for monitoring your progress toward achieving the intended change.



3. Identify specific outcomes for each type of change.

If you are going to change skills, what skills will be changed? Prevention: What will you stop from happening? Maintenance: What will you strive to keep the same? Behaviors: What will people do differently?

For example, here is the TOC we presented earlier in the chapter.

IF young people have a better understanding of the need to be tolerant of

other ethnic groups, THEN their attitudes and views of the “other”

communities will change and have a trickle-down effect on adult family

members. THEN communities will be more tolerant of one another and the

levels of conflict and violence will decrease.

Goals16 developed from this TOC could include increased tolerance for customs and traditions of “the other” among youth and their family members, and reduced reports of conflict between groups to community leadership.

The types of change indicated by the above examples of goals are an attitude change17 (tolerance) and a knowledge change (familiarity and understanding).

The specific outcomes we would expect to see that would indicate the goal has been reached include youth taking the initiative to organize joint events such as sports and cultural events, and fewer community members expressing fear of “the other.”

Below are the same steps, summarized in a table.

Table 4.2: Steps to develop goals from project Theory of Change

Step 1: Goals derived from TOC

Step 2: Type

of change18 Step 3: Anticipated outcomes

a. Reduced reports of conflict between groups to community leadership from 100 to 10 per month by 2016

Attitudes Decreased fear of “the other”

Reduction in distaste for customs and traditional practices of “the other”

Increase in acceptance of “the other” as part of the community

Decrease in blaming “the other” for past grievances

16 The goals stated here include specific targets (how much will change by when). Your initial goals do not have to have specific targets. They can be added (or at least refined) after your baseline study (evaluation that tells you your starting point). Targets should not be chosen arbitrarily, but based on what is reasonable and feasible (as opposed to idealistic or unrealistic).

17 From what we know about this (hypothetical) TOC, this effort also could have a complementary TOC about changes in attitudes leading to improved relationships, leading to peace. In this case, “relationship” would be the “type” of change we are promoting.

18 Church & Rogers, 2006, pp. 18–23, provides examples of how to align specific types of change with specific peacebuilding TOCs.

More on goal setting

A strategic goal should:

1. State the most ambitious impact that an organization can effect (long term)

2. Express the highest level for which the project can reasonably be expected to be held accountable

3. Not include the means of achieving the goal

4. Be as specific as possible and stated as an accomplishment




Step 2: Type

of change18 Step 3: Anticipated outcomes

b. Increased tolerance for customs and traditions of “the other” among youth and their family members

Knowledge and attitudes

Young people are familiar with specific customs of “the other”

Increase in understanding of why it is important to respect customs and traditions of “the other” community

Young people and family members are more accepting and respectful of customs and traditions of “the other” community

These goals won’t be achieved for some time (several years). However, outcomes can be evaluated periodically. If we see a marked decrease in community members expressing fear after year one, we might know we are on track toward meeting our goal. Or the opposite could be true. Either way, it is better to know when we are on or off track with meeting our goal than to wait until the intervention is over and “hope” we have been successful.

These outcomes will be very important for moving to the next stage. They will be integrated into your chosen conceptual model.

At this stage in the process, if you do not yet have specific project activities in mind, establishing goals and objectives will help you develop your project activities.

Tying Situation Analysis and Theory of Change to MERL

System Design

Figure 4.2, on the next page, illustrates how situation analysis and your TOC are connected to design of a good MERL system. Starting with the situation analysis, each level in the flow chart gets more specific toward designing a good system.

Level of intervention shows the type of change sought as a result of understanding

the nature of the conflict and context, as well as perceived, on-the-ground needs. One change sought in this example is “individual personal change” (as opposed to system-wide or community-level changes).

Areas of intervention/strategies further breaks down the type of change sought

into five specific changes, one of them being “healing and recovery.”

Proposed program activity and target of intervention shows the types of

activities this intervention will offer to promote healing and recovery. The first activity involves establishing trauma centers in five communities with a target of reaching more people than are currently being served by such centers (even if current outreach is zero). The second is to target certain key people (armed group leaders) who are perhaps in greater need of healing and recovery or whose successful healing and recovery is perhaps more important to overall peacebuilding goals than targeting others.

Examples of key changes sought proposes even more specific types of changes

the programs will seek. The first is an output (simply getting people to the trauma centers) and the second is an outcome (improved emotional wellbeing) that should result from applying the knowledge and experiences gained from the trauma counseling.

Examples of ties to MERL system design describes generally how the

organization will monitor and/or measure progress towards achieving these goals.



Figure 4.2: Linking situation analysis to MERL in peacebuilding

Current Situation

Conflict analysis, context analysis, needs assessment, other baseline data

Level of intervention

Areas of intervention/ strategies

Proposed program activity and target of intervention Examples of key changes

sought Examples of ties to MERL system design More people Key people

Individual personal change

(TOC category)

Healing and recovery

(type of change)

Establish trauma healing (TH) centers in 5 communities

Provide trauma counseling for armed group leaders

(specific outcomes)

Increased access to trauma care

Improved emotional well-being

Monitor the number of people who received quality trauma counseling

Evaluate emotional health of targeted populations

Perceptions and attitudes

Facilitate a number of joint social events between two communities with tensions

Hold outreach activities to engage leaders of ex-combatant youth

Increased tolerance between communities

Reduced violence

Monitor how the community perceives changes in their relationships

Evaluate if change in perception has led to reduced violence

Skills and capacity

Train peace committee structures on mediation skills

Train key leaders in the community in conflict negotiation

Improved capacity in mediation skills

Conflicts successfully mitigated

Monitor if training resulted in real skills

Evaluate if the improvement in skills actually resulted in a reduction in conflict

Individual behavior

Establish women’s micro-enterprise groups

Train enterprise mentors in loan making

Increased capacity to make loans

Increased household income

Improved economic recovery

Monitor the number of women engaged in micro-enterprises

Evaluate if micro-enterprise program result in a significant increase in household income

Relationships Hold joint music events for communities with historical conflicts

Gather chiefs of both communities to discuss conflict

Improved social relations between the communities

Improved level of trust

Reduced conflict

Monitor participation from both groups in joint events

Evaluate the effectiveness of social activities in building trust

Evaluate if increased perception of trust resulted in reducing conflict



Level of intervention

Areas of intervention/ strategies

Proposed program activity and target of intervention Examples of key changes

sought Examples of ties to MERL system design More people Key people

Socio-political change

Group behavior/ relationships

Design a mechanism to improve access of marginalized communities to rangeland resources during the dry season

Conduct outreach to key community leaders to participate with government in designing the system

Increased joint planning between citizens and government

Reduced incidence of conflict between marginalized pastoral communities and rangeland management authorities

Monitor if citizens’ perspectives are being incorporated into government plans

Evaluate if there has actually been improved access to rangeland by marginalized communities

Evaluate if improved access led to a reduction in conflict

Public opinion Mobilize NGOs to disseminate information on a reform policy

Train media practitioners on fair coverage

Improved public opinion on the reform policy

Increased number of unbiased media stories

Reform passed

Monitor public opinion on the policy

Evaluate if media training led to better media coverage

Evaluate to what extent advocacy activities helped pass reform

Institutional change

Advocate for the inclusion of a peacebuilding curricula for teachers in the national teacher training curricula

Train teachers in the curricula

Improved capacity of teachers to help youth mitigate conflict

Reduced violence between school-going youth from different ethnic backgrounds

Monitor teachers’ ability to teach youth how to mitigate conflicts

Evaluate if youth exposed to the curriculum reduced violence using in those skills compared to schools where the curriculum was not taught

Structural change

Lobby for the decentralization of land boards

Reduce the minimum qualifications for leaders vying for leadership posts on land boards

Improved access to land management boards for citizens

Equitable access to leadership positions for marginalized communities leadership

Monitor the number of people from marginalized communities vying for leadership positions in the next elections

Evaluate if land boards have more representative structures

Evaluate the quality of decisions of land boards in terms of equity

Desired vision/societal change




TOCs are a set of beliefs about how change happens.

Peter Woodrow’s 10 types of TOCs that can be used in peacebuilding programs are: individual change, healthy relationships and connections, withdrawal of the war resources, reduction of violence, root causes/justice, institutional development, political elites, grassroots mobilization, economics, and public attitudes.

When developing a TOC, it is important to understand: 1) What is making the conflict worse? 2) What conditions could reduce the conflict? 3) Who else is doing peacebuilding, and how can our project complement ongoing

efforts?

It is important to ensure that your TOC is reflected throughout the program.

Strategic project goals and outcomes should be informed by your TOC.

Learning Activity: Describe your project’s TOC

Based on your situation analysis findings, describe your project’s TOC (refer to the 10 types of TOCs in Table 4.1). _______________________________________________

______________________________________________________________

______________________________________________________________

______________________________________________________________

______________________________________________________________

Then, review your project’s goals to ensure they are aligned with the TOC and vision of peace.


Chapter 5: Conceptual Frameworks and Assumptions | page 32

Chapter 5: Conceptual Frameworks and Assumptions

Chapter Objectives In this chapter readers will:

Understand what conceptual frameworks are and how to develop them

Review examples of conceptual frameworks for peacebuilding programs (results framework, logical framework)

Understand the relevance of conceptual frameworks to program design and monitoring, evaluation, reporting, and learning (MERL)

Learn the role of assumptions

Learning Activities

Create a results framework

Create a logical framework

What are conceptual frameworks?

A conceptual framework is a summary description of how an organization/project thinks change will be brought about. The framework describes the process of planned change and can be presented as a table (e.g., the logical framework) or as a diagram.

Conceptual frameworks enable projects to explain the ideas and reasoning behind their Theory of Change (TOC) and link project activities to overall peacebuilding goals.

Describing your projects through conceptual frameworks helps you and your organization think about (and design/present) project activities and their results as contributing to larger peacebuilding efforts. Conceptual frameworks can help you expand on peacebuilding approaches to further define goals that relate to your TOC and proposed outcomes.

Conceptual frameworks can be used at any stage of the project cycle, even if your project is already in the midst of program activities, but has never clearly laid out how they relate to peacebuilding goals. You can start by populating the framework with the information you do have and fill in the pieces you do not.

Also, revisiting a conceptual framework periodically during the project cycle is a good way to make sure you are still on track as minor (or major) changes are made to programs, evaluations, objectives, etc. throughout the life of the project.

There are many types of conceptual frameworks used by development and peacebuilding initiatives. This chapter presents two that you may find useful:

Results framework

Logical framework (also called a “logframe”)

Each of these models, while different in appearance and structure, include the same four key components,19 as listed in Figure 5.1. Though they may be named differently in different models, (such as “outcomes” instead of “objectives”) for clarity we will label these components “goals,” “objectives,” “outputs,” and “activities” in each of their corresponding sections in the frameworks we are about to describe.

19 Adapted from Church & Rogers, 2006, p. 30.



Figure 5.1: Four levels of conceptual frameworks

In Chapter 4’s learning activity, we wrote program goals and objectives based on our

TOC. Using these as a starting point, we can begin to list short-term program outputs and define the activities that will shape them. Once we’ve established clear goals (keeping in mind that every stage of this process should be collaborative, i.e., goals of all stakeholders should

be considered), it is easier to define specific activities and the resulting outputs.

Advantages and Challenges of Using Conceptual

Frameworks20

Conceptual frameworks bring with them the following advantages. They:

Help a project focus on specific objectives/outcomes

Highlight the key linkages in the TOC that underpin the intervention

Help establish an evidence-based approach to MERL

Offer a systematic approach for program implementers to plan their strategies and to select interventions that are most likely to address targeted problems

Help guide corrective adjustments to activities, reallocating resources, and reevaluating targeted objectives or underlying assumptions

Provide a model for communicating about the resources, activities, and outcomes to project staff, development partners, or other stakeholders

Help build consensus, coordination, and ownership of the project among key stakeholders

Serve as the basic accountability tool for developing an evaluation approach

20 Adapted from International Bank for Reconstruction and Development & World Bank, 2012, pp. 8–10, 41.



But, conceptual frameworks also bring about some challenges.

They require that staff and stakeholders make an up-front investment of time and resources at the start of an intervention.

Interventions’ effects can be difficult to measure fully, especially the unintended consequences that likely were not captured in the framework.

They can become overly complicated, especially if the project indicators associated with the framework comprise a mix of qualitative and quantitative measures being used to measure complex programs (such as peacebuilding programs).

The Results Framework

The results framework organizes your activities into distinct stages, outlining what

happens (your results) in the short-, intermediate-, and long-term within four levels, as shown in Figure 5.2.21

Figure 5.2: Four levels of a results framework

In the previous chapter, we developed desired goals and outcomes from our TOC. The results framework can be used to build on them (and add more or make adjustments) and to tie in the related activities (inputs and processes) and outputs. Below we describe in more detail the four components of the results framework.

Inputs and processes (analogous to project “activities” in Figure 5.1) are the resources

and methods employed to conduct an activity, project, and/or program.

Inputs can be:

Physical, such as equipment rental or purchase

Material, such as supplies and provisions

Human, such as labor costs for salaries, technical assistance, and staff

Financial, such as travel, per diem costs, direct costs, and indirect costs

Processes are the methods or courses of action selected to conduct the work, such as training, organizing, publishing, lobbying, service provision, and message promotion. Direct results from inputs and processes are generally seen quickly (0–2 years) and often are measured through monitoring activities.

Outputs are information, products, or results produced by carrying out activities or

projects. Outputs relate to completion of activities and are the type of results over which managers have a high degree of influence. Outputs reflect what you hoped to produce from a particular input or process (or set of either). For example: You decide the process you want to use is to train people. The “number of people trained” is the result at the input/process level, while “knowledge level increased” would be the result at the output level, the assumption

21 Inputs and processes are sometimes segregated into two separate levels.



being that if you train people they will increase their knowledge on a given subject.22 Outputs usually reflect a result achieved in a relatively short time period (0–2 years) and are often measured through monitoring activities.

Outcomes are broad changes in development conditions. Outcomes help us answer the “so

what?” question. For example: We trained 100 people and increased their knowledge, but did they change their behavior? Outcomes often reflect change in attitudes and/or perceptions, emotional well-being of individuals, household income, access to rangeland or water for marginalized communities, and other changes and help us analyze how our activities and projects scale up or contribute towards these development outcomes. Outcomes usually reflect a result achieved over an intermediate time period (2–5 years).

Impacts are the overall and long-term effects of an intervention. Impacts are the ultimate

result, such as peaceful co-existence of previously conflicting communities. Impacts usually reflect a result achieved over a longer time period (5–10+ years).

A results framework organizes these four levels of results so you can see how inputs and processes lead to outputs, outputs lead to outcomes, and outcomes lead to impacts. If you choose to use this framework, you will later construct/select indicators (in Chapter 6) to help you measure attainment of each of these stages. Generally, a results framework will be accompanied by a separate MERL and indicator plan. Table 5.1 describes the type of information normally present in a results framework and how it is structured, and Table 5.2 shows the PEACE II results framework as an example.

Learning Activity: Results framework

Take what you already know about your program (it might be the activities, desired impacts, some of everything) and fill in the sections of the results framework in Table 5.3 for which you have relevant information. Then see if you can fill in any missing pieces (or add on to those you have).

Does this framework “make sense” to you? Does it seem like it will help structure your program and give guidance for later MERL? How does it align with the project’s TOC(s) that you developed earlier?

22 However, it is worth noting that some may consider “number of people trained” as an output indicator, with “knowledge level increased” as a shorter-term outcome indicator. The levels at which you consider indicators depends on several factors, such as project length, overall goal/objective, donor definitions.



Table 5.1: Example of a results framework

Inputs/Processes Outputs Outcomes Impacts (Goals)

Resources and processes used to produce a change/result

Examples

Inputs delivered or implemented:

Staff hired

Grants awarded

Funds disbursed

Materials/equipment purchased

Processes conducted:

Situational analysis completed

Appropriate fora, alliances, committees established

Stakeholders mobilized

Communities mobilized

Trainings conducted

Meetings held

Networks established

Advocacy (radio, print, dramas) messages developed

Short-term change, effects, and results

Examples

Knowledge/awareness, understanding change:

Increased understanding of the conflict dynamics

Increased demand for grassroots peacebuilding

Capacity/skills change:

Improved skills and abilities to negotiate conflicts

Stakeholder engagement and dialogue expanded

Access/expansion change:

Networks for peace supported

Quality change:

Laws and policies developed which promote stability

Intermediate change, effects, and results

Examples

Proportion/coverage change:

Increased action by an enlarged constituency for demanding and managing grassroots peacebuilding

Attitude change:

Increase acceptance of members of ‘the other’ community

Behavior change:

Improved pastoral and land use management

Actual policy change

Laws and policies implemented which promote stability (e.g., land use rights and management)

Long term change, effects, and results

Examples

Political change:

Reduction in violent conflict

Human rights, socio-cultural, and empowerment change:

Increased civil society participation in political processes

Socio-economic and/or health status change:

Improved livelihoods (income, health, education)

Resource management change:

Increased legitimacy, transparency, and accountability in governance

Project Level (implementation/progress monitoring)

Program Level (progress/results monitoring)

Strategic Level (Effectiveness) (results/impact monitoring)



Table 5.2: Example of PEACE III results framework

Inputs/Processes Outputs Outcomes Impacts

Conduct participatory learning and action workshops with community representatives to identify and prioritize peace dividends

Create space for dialogues on cultural norms and practices

Identify and train trauma healing (TH) counselors

Conduct TH sessions

Assess community peacebuilding capacities to identify capacity building needs

Facilitate formation and/or strengthening of peace networks

Conduct Appreciative Inquiry/strengths, weaknesses, opportunities, and threats (SWOT) analysis with local government peace institutions

Targeted capacity building interventions for government peace institutions

Develop customized capacity assessment tool for PEACE III partners

Identify capacity building priorities for PEACE III partners

Peace dividends constructed

Communities sensitized on the role cultural practices play in conflict and peacebuilding

Cadre of trauma healers with knowledge and insight on who can assist communities withstand conflict shocks

Targeted capacity support extended to communities

At least one peace network per corridor established and strengthened

Joint visioning and planning conducted and joint activity plans developed

Increased number of cross-border mechanisms established or strengthened

Increased local government participation in peacebuilding initiatives (initiated by PEACE III)

Organizational Capacity Assessment conducted for local partners

Targeted capacity support extended to implementing partners

1.1: Communities are more open to social reconciliation

1.2: Communities’ peacebuilding capacities mobilized

1.3: Local governments partner with their cross-border counterparts and with communities in conflict management

1.4: Capacity of Horn of Africa NGO partners to support cross border conflicts management increased

Strengthened cross-border conflict management



Table 5.3: Blank results framework for the Learning Activity

Inputs/Processes Outputs Outcomes Impacts



Logical Frameworks

A logical framework or logframe is similar to a results framework in that it explains

how a project’s day-to-day activities will achieve results. One main difference between a logframe and a results framework is that the latter describes the results as if the program had already been completed while a logframe describes what will happen in the future.23

A logframe promotes good project design by clearly stating the defined project logic and components. The logframe is usually formed as a chart that shows a hierarchy of:

Four levels of the causal relationship: activities, outputs, purpose, and goal

Indicators24 of performance

Means of verifying the indicators

Important risks and assumptions

The logframe provides a summary of what the project aims to achieve and how, what the main assumptions are, and a basis for developing the activity’s MERL system. Progression from one level to the next is based on “if/and/then” logic; for example, if the activity at the lowest level takes place and the assumptions hold true (or the risks are not realized), then the expected output at the next level can be achieved.

Basic Logframe Outline (Hierarchy of Objectives)

The basic structure of a logframe is provided in Table 5.4. For each of the four levels of

causal relationships (goals, purpose, outputs, and activities), you explain:

The direct or indirect measures that will verify to what extent the objectives have been

fulfilled (the objectively verifiable indicators)

The source of the information for the measurements or verification specified in the

indicators column (the means of verification)

The important events, conditions, or decisions necessarily outside the control of the project, but which must remain favorable for the project objective to be attained (the

assumptions or external factors)

Table 5.4: Logframe structure25

Objectively verifiable indicators

Means of verification

Assumptions (external factors)

Goal (goal)

Long-term impact sought/overall vision

Qualitative/ quantitative evidence to judge achievement of the goal (e.g., strengthen cross-border conflict management)

Specific sources of data to quantify or assess indicators of the goal (e.g., percentage of community respondents who perceive effectiveness in local conflict-management institutions)

External factor necessary to sustain the objectives in the long term (e.g., stability of operating environment)


24 We will go into more detail about constructing indicators in Chapter 6.

25 Adapted from DFID. 1997. Guidance on Humanitarian Assistance. Available at https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/404342/Humanitarian-Response-Funding-Guidelines-2015.pdf.

https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/404342/Humanitarian-Response-Funding-Guidelines-2015.pdf







Purpose (objective/result)

Intended immediate impact of the project

(purpose to goal)

Qualitative/ quantitative evidence to judge achievement of the purpose (e.g., openness to social reconciliation)

Specific sources of data to quantify or assess indicators of the purpose (e.g., level of community openness to social reconciliation)

External conditions necessary for project purpose to contribute to the achievement of the project goal (e.g., willingness of communities to devote their time to participate

in the project)

Outputs (output/ intermediate result)

Tangible/intangible result(s) to achieve the purpose expected from the project

(output to purpose)

Qualitative/ quantitative evidence to judge achievement of the outputs (e.g., construct peace dividends)

Specific sources of data to quantify or assess indicators of the outputs (e.g., number of people accessing peace dividends)

External factors that, if present, can restrict progress from outputs in achieving project purpose (e.g., conducive environment to support engagement with communities during program implementation period)

Activities (activities)

Actions needed to achieve each output of the project (these are things that the project can control)

(activity to output)

Evidence for monitoring and reporting implementation in line with the logframe (e.g., conduct Participatory Learning and Action to identify and prioritize peace dividends)

Specific sources of data to confirm implementation of planned activities (e.g., number of peace dividends completed)

External factors that, if present, can restrict progress from activities in achieving outputs (e.g., conducive environment to support engagement with communities during program implementation period)

The Role of Assumptions

Assumptions are the “external factors [that] are not influenced by the project, but may

affect its implementation and long term sustainability.”26 Identifying assumptions is critical at the project’s design stage. If you are designing the project using a logframe, the assumptions are included in framework itself. If you are using a results framework, you can still state your assumptions for each goal, objective, output, and activity in a simple list or table, such as:

Goal 1: Assumption:

Objective 1: Assumption:

Output 1: Assumption:

Assumptions can come in different forms.

Examples of things we assume will (or will not) take place include:

Approval of community-based organizations (CBOs) to be funded under the program by the funder

26 Better Evaluation. 2015. Logframe [web page]. Available at http://betterevaluation.org/evaluation-options/logframe.

http://betterevaluation.org/evaluation-options/logframe

http://betterevaluation.org/evaluation-options/logframe



Communities in target areas will support the peacebuilding initiatives

Cross-border local authorities will work together to promote peace in their areas

Local, national, and regional peacebuilding entities will work together to promote peace in target areas (vertical linkages)

There will be no new conflict flare-ups or natural disasters that inhibit the work

Assumptions that we take for granted or believe to be true—or we believe are very likely to be true—that affect results include:

Partner organizations will value capacity building and will make time for institutional strengthening activities

Enough people will be interested in supporting peacebuilding efforts

The program will mobilize a critical mass to effectively advance peacebuilding initiatives

We can get off track in our planning, budgeting, and problem solving if we don’t list our assumptions or have assumptions that are wrong.

We need to deeply think about the things we are “assuming” because they can affect the program results.

We need to write down our really important assumptions.

We need to test our assumptions to be sure that our assumptions are based in reality.

You can assess the risks coming from assumptions using Figure 5.3. Note that if an assumption is very likely to prevent the program from taking off or advancing, you either need to redesign the program or address the assumption directly to make it true.

Apart from redesigning a program based on the likelihood of an assumption preventing the execution of such a program at the start of a program, programs also could be redesigned during the course of the program if foreseen assumptions increasingly become likely to affect the intended outcome of such a program. Assumptions could give grounds for redesigning programs.

Figure 5.3: Assessing risks associated with program assumptions27




Learning Activity: Logical framework

Take what you already know about your program (it might be the activities, desired impacts, some of everything) and fill in the logframe sections in Table 5.5 for which you have relevant information. Then see if you can fill in any missing pieces (or add on to those you have).

Does this framework “make sense” to you? Does it help to structure your program and give guidance for later MERL? How does it align with the project’s TOC(s) that you had developed earlier?

Table 5.5: Blank logframe




Goal

Purpose

Outputs

Activities


Conceptual models help present a project’s TOC by showing how activities at the lowest level lead to achieving the project goal.

There are many types of conceptual models in use today, but two common ones are the results framework and the logical framework.

All models include four levels: activities, outputs, outcomes, and impact, though terminology and how each type of conceptual model is structured will differ.

Conceptual models provide program implementers with many advantages, but also present several challenges. For example, they require investment of time and other resources at the program design stage and they do not account for unintended consequences that result from the project.

It is important to recognize and test the risk associated with your project assumptions.


Chapter 6: Indicators | page 43

Chapter 6: Indicators


Learn how indicators are used to answer questions about your project's outputs, outcomes, and impact

Learn how to develop and assess indicators

Learn how to use an indicator plan

Get familiar with the U.S. Government Foreign Policy Indicators for Conflict Prevention and Mitigation

Be introduced to the PEACE III indicator protocols to help determine indicators that apply to your project

Learning Activity

Indicator plan

Overview

Indicators are the building blocks of monitoring and evaluation. Indicators signal change.

The Organization for Economic Cooperation and Development (OECD)28 defines an indicator as a “quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement, to reflect the changes connected to an intervention, or to help assess the performance of a development actor.”

Indicators are not just anything your project wants to measure, and every measure is not an indicator. Indicators are meant to reduce a large amount of data down to its simplest form. For example, when you want to buy a car, unless the engine has been replaced, the best indicator of a car’s condition is its odometer because it tells you how far the car has driven and, therefore, how much wear and tear the engine has undergone.

Indicators also are not the same as goals, objectives, results, or targets. They are developed to measure results at the four levels of results depicted in conceptual models: activities/input, output, outcome/objectives/results/purpose, and goals/impact. Indicators do not specify a particular level of achievement; thus, words like improved, increased, gained, and decreased do not normally belong in an indicator. Rather, indicators describe the unit of information to be measured over time, for example, “change in knowledge,” which could increase or decrease, but this is further specified in the indicator target. When compared with targets, indicators may signal the need for management action and help us determine if objectives are being met.

Indicators can be especially useful when straightforward data cannot tell us what we need to know. Instead, we can ask a series of questions that indicate the answer to our question. In other words, when a question cannot be answered directly by one piece of data, we collect one or more indicators instead that give us a good approximation of what the answer may be. Table 6.1 provides an example of overall project questions and their illustrative indicators.

28 OECD 2002.



Table 6.1: How indicators help answer project questions

Question Indicator

1. How many trainings were held?

1. Number of trainings held

2. Were trainings effective?

2a. Percentage of participants correctly answering 8 of 10 knowledge-based questions on the quiz at the end of each training

2b. Percentage of participants self-reporting improved understanding of [the topic] after the training

2c. Percentage of participants self-reporting that they did [x thing that they learned at the training] within 6 weeks of the training

3. Has trust increased between groups A and B?

3a. Percentage of individuals from group A indicating they would do business with a member of group B after the training

3b. Percentage of businesses in region A operated by individuals from group B after 3 years

3c. Percentage of parents/guardians from group A indicating they would allow their children to be educated with a member of group B after the training

Indicator 1 in Table 6.1, about the number of trainings held, is straightforward because the question it answers is straightforward. This indicator represents output data we can track through training sign-in sheets or registration records. Questions 2 and 3 are less straightforward. Often a single indicator is not sufficient to answer our questions. Sometimes we want to look at the issue from multiple perspectives; other times the issue at hand is too complex to be addressed with a single line of inquiry. For example, indicators 3a and 3b could be used to answer the question “Have business opportunities increased for people in group B?” In fact, if we see positive change in these indicators, they might have little to do with trust building and more to do with increased business opportunities or a better enabling environment.

This brings up another point: Indicators do exactly as their name implies, they indicate the answer to a question that cannot be directly answered. It is important when both selecting indicators and analyzing data to carefully consider if an indicator’s ability to show a result is caused to some extent by the project-sponsored activities or whether results demonstrated by the indicators do not stem from project activities.

We ask a particular question because we think it will tell us something about our intended program outputs, outcomes, or impacts. For example, we would ask a question about trust building if a goal of our program is to increase trust among opposing groups. This also presumes that we did something specific in our program that aimed to build trust (directly or indirectly). But, even if the indicators we choose all show positive results (there is an increase in people doing business across groups and there are more businesses in region A operated by people from group B), we still can’t be certain that those changes happened because of increased trust.

Also, even if they are the result of increased trust among groups, we can’t be certain that our program efforts directly led to that change. Perhaps there was a government-run program operating in the same region with the same goal of trust building that was actually responsible for the change. Perhaps a sub-group or leader within the community that had been supporting division and separation of the groups is no longer active. The point is that it is very difficult to control for all the factors that influence change in individuals and communities.



The primary function of indicators is to show us what changes have occurred, to what degree, and over what period of time. They may not tell us for certain how or why a change has occurred, thus it is important to supplement indicator data with documented qualitative data to help tell this story of change.

How to Develop Indicators

Indicators are written to measure the goals and objectives29 we have chosen (and their relevant outputs, inputs, and processes). Indicators can be written for overarching program goals and specific intervention objectives. They can contain up to seven specific pieces of information:30 1. The thing to be measured/changed 2. The unit of measure that informs the

change 3. The starting point (baseline

information) 4. The scope, size, or degree of

intended change 5. The quality or standard of change

desired 6. The target population 7. The timeframe

We can divide indicators into two basic categories: quantitative and qualitative.

Quantitative indicators measure

information in specific amounts or quantities, such as the number of people who did something, the number of times something specific happened, or the amount of increase or decrease seen in some discrete activity. An example for a peacebuilding program may be “Number of community members participating in trauma healing (TH) initiatives.”

Qualitative indicators include more subjective information, such as opinions, beliefs, or

perceptions, and typically are not able to be quantified in the way quantitative indicators are because of the way data is collected, such as through open-ended or guided questions during focus groups or key informant interviews. For example, a qualitative indicator in a peacebuilding program may be “Improved sense of security among TH participants.” To gather data for this indicator, you are unable to survey all or a sample of participants, rather you speak with individual or groups of community members to understand the general sense of how they feel about security after participating in TH. This qualitative indicator complements the straightforward count of participants in TH with participants’ sense of security as a result of those sessions.

29 See Chapter 4 for more on determining goals and objectives.


Example of an indicator with five of the seven pieces of information

Percentage of cattle herders in Karamoja who report disputes over grazing land to their local council one or more times per month in the last 3 months.

1. The thing to be measured: who report discputes over grazing land to their local council

2. Unit of measure: percentage 5. Quality or standard of change desired: one or

more times per month 6. Target population: cattle herders in Karamoja 7. Timeframe: by December 2016

Note: Once baseline data is collected, the project can establish the starting point and size of change that it wants to achieve within the specified timeframe. For example, the project can at baseline establish that currently only 5% of cattle herders report disputes to their local council one or more times per month. Taking into consideration various project factors (e.g., planned interventions, duration, funds) the project can decide that the size of change it can reasonably achieve is raising the percentage by

25%, thus the target of 30% for this indicator.



One way of looking at the difference between quantitative and qualitative indicators is that quantitative data tells us what happened and how much or how often it happened, while qualitative data tells us how what people thought about it and what impact it had on them. It is highly recommended to use a mixture of qualitative and quantitative indicators to measure the attainment of your objectives.

Table 6.2: Examples of qualitative and quantitative indicators and data sources

Example indicator Potential data source

Qualitative

Community openness to social reconciliation Focus groups

Improved sense of security among TH participants


TH participants less inclined to partake in violence


Quantitative

Number of peace dividend initiatives benefitting two or more conflicting communities

Project monitoring documents

Performance of community conflict management structures

Pact’s Community Performance Index

Number of initiatives led by community peace actors to address local conflicts


Number of cross-border peace initiatives that involve local government


Percentage of community peace initiatives that receive tangible local-government support


Review of government actions

Number of new linkages among local peace-building organizations

Pact’s Organizational Network Analysis

Percentage of community respondents who perceive effectiveness in local conflict management institutions

Community-wide, face-to-face survey

The U.S. Department of State’s Standard Foreign Assistance Indicators31 are a good place to search for indicators that have been tested and used. Most U.S. Government-funded projects will be required to use some of these indicators. PEACE III has adopted several of them for use in the program.

Developing or choosing the right indicators is the heart of designing a practical MERL system. This process can be tedious and exacting, so you cannot expect to sit down in one afternoon and develop all the indicators your project needs. It should be an iterative process involving brainstorming a range of possibilities, seeking opinions from various staff and stakeholders, and assuming different perspectives as you develop indicators.

31 See http://www.state.gov/documents/organization/231895.xlsx, specifically the conflict mitigation and reconciliation indicator definitions.

http://www.state.gov/documents/organization/231895.xlsx



Qualities of a Good Indicator

Once you’ve gathered a large list of potential indicators, you should refine them. In addition to being appropriate for measuring achievement of your project objectives, all indicators should meet three specific quality standards: reliability, feasibility, and utility.32

Reliability is the extent to which one can reasonably assume that they can acquire correct

or accurate information. If it is likely that two or more people seeking the same information will come back with different results/answers (repeatability), the indicator may not be reliable. An indicator may be deemed unreliable if:

It comes from generally unreliable sources (e.g., subjective assertions)

The question asked can be interpreted in multiple ways (e.g., the act of “participating” in an activity can be interpreted in different ways by different people)

Factors such as time of day or time of year of collection are likely to influence answers (e.g., during the dry season people will be supportive of activities that involve encroaching on other communities’ grazing land)

People are prone to lie about it, not know the answer, or give an inaccurate answer (e.g., which individuals were involved in the cattle raid?)

Feasibility33 is the extent to which the information needed can be readily acquired. It

requires that you be able to access the source of information and the specific data you need. Barriers to access include:

Security or safety concerns

Sensitivity of the topics

Confidentiality of information sought

Unwillingness of sources to participate

Physical barriers (e.g., information is in a location that is difficult to get to, that is prone to severe weather)

Utility is the extent to which the information is actually useful. How much will what we learn

help us to make or adjust programmatic or strategic decisions? When considering whether an indicator has utility, imagine having the results in front of you and asking yourself, “What will I do with this information?” Individual indicators might not, by themselves, inform decision-making, but may be considered collectively for utility in some instance.

Another way projects use to assess the quality of indicators is based on the five criterion:

Specific, Measurable, Achievable, Relevant, and Time-bound (SMART). Table

6.3 describes the criteria for SMART indicators.

Table 6.3: SMART indicator criteria34

S Specific: Is the indicator specific enough to measure progress toward results?

M Measurable: Is the indicator a reliable and clear measure of results?

A Attainable: Are the results in which the indicator seeks to chart progress realistic?

R Relevant: Is the indicator relevant to the intended outputs and outcomes?

T Time-bound: Are data available at reasonable cost and effort?

32 See Church & Rogers, 2006, p. 45.

33 In situations where tensions and/or conflict are high, one cannot underestimate the importance of indicator feasibility. See Chapter 11.

34 Adapted from UNDP, 2009, p. 63.



Table 6.4 provides examples of indicators that are well formed (“good”) and poorly formed (“bad”). Note that indicator wording does not need to capture the direction of change you are hoping for. Rather, this would be visible in the targets set—the targets would show increases—and the actual achieved figures could be increases or decreases.

Table 6.4: Examples of good and bad indicators

Good indicators Bad indicators

Number of community members involved in peacebuilding

Increased number of community members involved in peacebuilding

Number of workshops conducted (disaggregated by type of workshop)

Number and type(s) of workshops conducted

Number of conflict incidents reported Number of people killed in violent conflict35

Amount of rainfall Level of rainfall

Level of beneficiary satisfaction Degree of happiness

Indicator-Related Terms

Table 6.5: Other terms used in relation to indicators

Term Definition

Baseline This is a record of what exists in an area prior to the start of a project. It is primarily a benchmark for the future. The baseline values establish the starting point from which change can be measured.

Targets These are values against which the actual program/project achievements are measured. They describe the magnitude or level of results expected to be achieved. Targets should be realistic and, where possible, informed by past achievements.

Proxy indicators These are alternate measures used to stand in for another indicator when obtaining direct information is too difficult, time consuming, or sensitive. For example, household consumption of maize could be a proxy indicator for household income.

Indicator protocols

These are instruction sheets that describe the indicator in precise terms and identify the plans for data collection, analysis, reporting, and review. Examples of precise information contained in an indicator protocol are indicator definition; unit of measure; disaggregation; data collection, collation, analysis, reporting, storage, and use; data quality; and targets.

Sources of verification

These are sources you have or use to confirm or substantiate the data, for example attendance lists and event reports.

Unit of measure This specifies what exactly the indicator will measure or count, for example number of events funded by the U.S. Government or number of people trained.

Disaggregation This refers to how the indicator data will be broken down at the time of reporting to stakeholders, for example by location, gender, or age.

35 This may be considered a “bad indicator” from an early warning perspective. This indicator has less value in terms of informing program action to “save lives” when compared to the number of incidences reported. The assumption here is that tracking occurrence of incidences can help a program take timely action to focus on hotspots, thereby reducing the likelihood of escalation and loss of lives.



Indicator Definitions

In order to ensure that our indicators are not subject to misinterpretation and are clearly understood by everyone who will use them, it is important for us to define each indicator. An indicator definition clarifies the key terminologies used within the indicator to ensure data collectors and users measure or interpret the same thing. Using the example indicator that we developed earlier in the chapter, “percentage of cattle herders in Karamoja who report disputes over grazing land to their local council one or more times per month by December 2016,” it is important to define what the project means by “cattle herders in Karamoja” and “report disputes to the local council.” Doing so will help ensure there is common understanding of which group of people within the Karamoja community the project will count as “cattle herders” and what the project will count as an “incident” that has been “reported” to the “local council.” Below is an example of an indicator definition.

This indicator will count cattle herders from among the Pokot and Turkana communities who are residents and derive their livelihood from keeping and grazing cows within the administrative boundaries of Pokot and Turkana counties. A dispute refers to a disagreement between two cattle-keeping communities over issues of grazing land that may or may not result in fighting between the two communities. Reporting to a local council means formally informing the local authority responsible for ensuring equitable use of grazing land within these communities about a dispute over grazing land with members of the other community.

In addition to defining the indicator, it is important to specify the disaggregation that will be used when reporting on each indicator, for example location, name of community, and type of local authority.

Collecting Indicator Data

Once we have developed and defined the set of indicators we want to track and assess, they are integrated into the project’s data collection instruments, such as event documentation; reporting templates; household surveys; and observation, focus group, or interview protocols.

All of the information about the indicator is tracked in the project’s indicator plan. Table 6.6 provides an example from a peacebuilding program.

Note: Implementers of U.S. Agency for International Development (USAID) programs will be asked to complete Performance Indicator Reference Sheets (PIRSs) as part of their MERL plan.36 PIRSs are a short table for each indicator detailing the indicator definition, unit of measure, source, disaggregation, and other key pieces of information. PIRSs are invaluable resources for all MERL and program staff to understand exactly what is committed to be measured and how to follow the protocols established.

36 See more on PIRSs at https://www.usaid.gov/sites/default/files/documents/1870/LocalSystemsFramework.pdf.

https://www.usaid.gov/sites/default/files/documents/1870/LocalSystemsFramework.pdf



Table 6.6: Example indicator plan

Indicator (type)

Baseline and Target 2015 Definition and disaggregation

Data source(s)

Unit(s) of measure

Data collection tool(s)

Data collection frequency

Reporting frequency

Goal: Strengthened local cross-border management

Percentage of community respondents who perceive that the local conflict management institutions are effective

(Type: outcome)

Baseline: 5%

Target for 2015: 10%

Definition: Community respondents are defined as those living in PEACE III target areas. The respondents could be people from the same ethnic group/clan living in either rural or urban areas. PEACE III takes success in managing conflicts as a proxy measure for effectiveness in conflict management.

Perception of effectiveness is measured by survey respondents expressing how successful local conflict management institutions are in managing conflict on a scale of 1–4, denoting very unsuccessful, unsuccessful, successful, and very successful.

Local conflict management institutions denotes the following types of institutions:

Local peace committees

Government security committees

Traditional conflict management institutions

Community-based organizations (CBOs)

Disaggregation: Location (cluster, corridor), age, youth, gender, and type of local conflict management institution

Baseline survey, midterm survey, endline survey, perception survey

% of total community respondents

Household survey questionnaire

Focus group discussion guide

Baseline, midterm, endline

Baseline, midterm, endline




Indicators are not the same as goals, objectives, results, or targets.

Indicators describe the unit of information to be measured over time.

The primary function of indicators is to show us what changes have occurred, to what degree, and over what period of time.

Indicators can be classified in two basic categories: quantitative and qualitative.

Quantitative indicators measure information in specific amounts or quantities, such as the number of people who did something or the number of times something specific happened.

Qualitative indicators include more subjective information, such as opinions, beliefs, or perceptions, and are not typically able to be quantified in the way quantitative indicators are because of the way the data is collected, such as through open-ended or guided questions during focus groups or key informant interviews.

Criteria based on the following five characteristics are used to assess the quality of the indicators: Specific, Measurable, Achievable, Relevant, and Time-bound (SMART).

In addition, all indicators should meet three specific quality standards: reliability, feasibility, and utility.

Learning Activity: Indicator plan

Refer to the indicators and targets in your award document. Read through the indicator definitions found in the indicator protocols that are in the award document or that you are given for this exercise, and reflect on your project objectives/activities. Review the activities currently linked to each indicator and update indicators and the ways of measuring them as necessary, taking into consideration the following.

The appropriateness of the activities/individuals that you are counting to report on this indicator: Do they fit with the definition in the project’s indicator protocol of types of activities to be counted?

Are your targets realistic given your available budget, staffing, award period, and past experience? Please update targets accordingly for your final award.

Prepare to submit to your funder an updated indicator plan that’s well aligned to your funded project.


Chapter 7: Monitoring | page 52

Chapter 7: Monitoring


What monitoring is and how it helps peacebuilding efforts

Three types of monitoring

How to develop a monitoring plan

Learning Activity

Develop a monitoring plan

What is monitoring?

Monitoring is an activity carried out throughout the life of the intervention that provides

project stakeholders with concrete information from which to make strategic decisions. In peacebuilding, we often monitor three specific areas: the conflict/context, our program implementation, and our assumptions. In other words, we use monitoring to periodically answer the following questions.

Has the conflict context changed?

Have we done what we planned to do by this point?

Has what we’ve done had the effect we expected it to?

Let’s look at our example Theory of Change (TOC) and goals from Chapter 4, with added sample indicators, as shown in Table 7.1.

Table 7.1: Example TOC with goals, outcomes, and sample indicators

TOC: IF young people have a better understanding of the need to be tolerant of other ethnic groups, THEN their attitudes and views of the “other” communities will change and have a trickle-down effect on adult family members. THEN communities will be more tolerant of one another and the levels of conflict and violence will decrease.


Step 2: Type of change37

Step 3: Anticipated outcomes

Sample indicators (more and others are possible)

a. Reduced reports of conflict between groups to community leadership from 100 to 10 per month by 2016

Attitudes Decreased fear of “the other” % change in # of members of Group A who self-report fear of members of Group B

Reduction in distaste for customs and traditional practices of “the other”

% change in # of members of Group A who self-report that they disagree with the customs or traditions of Group B

Increase in acceptance of “the other” as part of the community

% change in # of members of Group A who, when asked to describe their community, mention individuals or groups from Group B

Decrease in blaming “the other” for past grievances

% change in # of members of Group A who self-report that past conflict with members of Group B was at least in some part their own fault

37 Church & Rogers 2006 provides examples of how to align specific types of change with specific peacebuilding

TOCs. See www.sfcg.org/programmes/ilr/ilt_manualpage.html, Chapter 2, pp. 18–23.

http://www.sfcg.org/programmes/ilr/ilt_manualpage.html



TOC: IF young people have a better understanding of the need to be tolerant of other ethnic groups, THEN their attitudes and views of the “other” communities will change and have a trickle-down effect on adult family members. THEN communities will be more tolerant of one another and the levels of conflict and violence will decrease.


Step 2: Type of change37

Step 3: Anticipated outcomes

Sample indicators (more and others are possible)

b. Increased tolerance for customs and traditions of “the other” among youth and their family members

Knowledge and attitudes

Young people are familiar with specific customs of “the other”

% change in # of young people of Group A who can name 2–3 specific customs of Group B

Increase in understanding of why it is important to respect customs and traditions of “the other” community

% change in # of members of Group A who self-report that they understand the importance of respecting the customs and traditions of Group B

Young people and family members are more accepting and respectful of customs and traditions of “the other” community

% change in # of members of Group A who self-report that they do respect customs and traditions of Group B

How might we monitor progress toward our goals? We would probably track program participation (e.g., monthly, quarterly) and resources being used (e.g., cost, staffing) to be sure we are not currently in danger of surpassing budgets. Tracking progress toward specific indicators is likely to be done through process (periodic) evaluations (see Chapter 8).

Conflict and Context Monitoring

Because the environments where we conduct peacebuilding activities are highly dynamic and often changing, it is important to monitor the context and conflict throughout an intervention (see Chapter 3). Doing so helps us know whether our activities are still appropriate and if we need to make minor or major changes, either immediately or in the future.

Table 7.2: Example conflict/context monitoring outcomes and potential resulting program changes

If monitoring shows us that… We might have to…

Violence has escalated Increase security measures

Postpone or cancel the project

Change the focus and/or TOC

Relevant legislation has been passed Adapt or develop new interventions to meet new requirements or better align with legislative goals

Types or sources of conflict have changed

Re-evaluate our strategic value: Are our objectives still in line with what is needed for peacebuilding?

Deciding when and how to monitor these changes depends on resources available (does someone have time to monitor this monthly? quarterly?) and your perception of how quickly the context may change. How fragile is the situation? Are there particular triggers or flashpoints (specific events that would impact the conflict positively or negatively) that are likely to occur? Are there patterns to watch for (e.g., movement of populations, frequency of aid delivery) that would indicate a change in the conflict? In ideal circumstances, how often



should we monitor the situation? Focusing on key contextual elements (rather than trying to cover everything) will help you design a context-relevant monitoring plan.

The Measuring Progress in Conflict Environments (MPICE) framework38 is a good reference for identifying the specific questions you will want to answer through conflict monitoring. Suggested methods39 for getting the information you need to monitor a conflict include:

Qualitative methods

- Content analysis: surveying newspapers, blogs, and other public media sources (secondary sources)

- Expert knowledge: panels of local program staff, local or national leaders or others in positions of authority, journalists, researchers, people involved with other local groups or community organizations, or anyone with a direct interest in knowing what is happening on the ground (including those involved in the conflict itself)

Quantitative methods

- Primary data: information that can be collected and quantified (counted) related to security, economic development, living conditions, etc.

- Survey or polling data:

Primary sources: Public opinion surveys, including surveying local program staff

Secondary sources: Datasets or information from international organizations that monitor crises and publish relevant information publically (Keep in mind that these broader monitoring tools cannot replace the valuable information you can learn through monitoring your specific context locally.)

Implementation Monitoring

Regardless of the duration of the intervention, it is important to monitor step-by-step progress toward achieving ultimate goals in terms of project administration, inputs, processes (activities), and outputs.

Generally speaking, implementation monitoring:

Compares what has been planned against what has been accomplished

Is used to assess and validate what has happened in the past and set the stage for what has to happen in the future

Is quantitative in nature, measuring how much or how many of a thing happened

Example questions implementation monitoring answers include: How many trainings were held and how many people completed the training? How much money was spent on each of X activities? How many times did Y happen?

For example, consider a project that has a plan to deliver 20 trainings to community leaders in years 2–4 of a five-year project. At the end of year 2, monitoring data might show that the

38 See Dziedzic et al. 2008.

39 Ibid. See the full report for descriptions of the pros and cons of each of these methods.

Note on using external sources of information

Accessing reliable data is a considerable challenge in fragile environments. When getting information (e.g., data, stories) from sources external to your organization, be careful to consider its quality and any potential biases. Is the data source reliable? Does the source have reason to give out false information? Were ethical and/or statistically sound practices used to collect the information? See Chapter 11 for more information on data quality.



project has delivered two trainings, which may be on target with the project plan or may indicate that the team is behind on holding these trainings. Or perhaps 15 trainings have been held by the end of year 2, which could indicate higher than anticipated demand for the activity, in which case the organization may want to adjust its planning or search for more funding, staffing, or opportunity to increase the number of trainings it can hold in subsequent years.

The previous example was of monitoring the magnitude of outputs (how many of this,

how much of that). We can similarly monitor progress toward achieving changes.

If a program’s target is to increase “something” from 10% to 75% over 5 years, we may use monitoring tools to look for incremental improvements every six months or a year, for example, increasing to 20% after year 1 or 35% after year 2.

Achievement of planned project stages also is important to monitor. Interventions often are planned incrementally (e.g., first introduce a savings product, then offer business development training, then offer business loans) and subsequent activities may each require

significant planning and preparation. Monitoring the implementation of each project

stage is important for informing how and when to plan and adjust future stages. In the

examples in Table 7.3, it may be wise to monitor each activity even more closely, such as every four to six months, to check progress toward each specific activity’s target.

Table 7.3: Example changes in project plans based on implementation monitoring outcomes

Objective to be monitored If the target is not met we may have to…

If the target is exceeded we may have to…

Year 1: 200 community members complete trauma healing (TH) sessions

Understand why we may be struggling to reach 200 participants

Expand and lengthen the duration of this particular intervention until we get 200 participants

Review our community mobilization strategy to draw more members to attend the TH sessions

Review current budget to identify budget savings and determine whether we can reach even more people with existing resources

Review our strategy for year 2 to provide follow on support to those who completed the sessions to see how they are coping with trauma

Year 2: Facilitate formation of 20 support groups for individuals who have completed the TH sessions

Review strategy for mobilizing TH participants to join support groups

Establish reasons individuals are not ready to join support groups and identify alternative strategies to provide the support (e.g., one-on-one follow up)

Make budget amendments to identify resources for the extra support groups

Re-evaluate the strategy to prioritize areas in which support groups will be formed in year 2

Source additional funds for support groups in year 3

Year 3: Train 40 community members as TH facilitators

Review strategy for identifying additional community members who can make good TH facilitators

Develop interim strategies to address the shortfall and ensure program results are achieved (e.g., using project staff)

Source additional funds to form additional TH session groups

Pair up TH facilitators to lead the group TH sessions



Personal narratives, stories, and anecdotes also can provide valuable monitoring

information, including for report writing, adding a human element to complement quantitative data. However, make sure your program is not reactionary based on a small number of accounts. If one program officer constantly complains about a particular problem (or success), it may or may not be worth looking into. However, several program officers reporting the same problem (or success) with or without prompting is even stronger evidence that something needs to be addressed.

Funders often have requirements for monitoring and expect regular reports (e.g., every three or six months), such as budgetary reports and status updates. While they may provide you with a set of metrics on which they want to see reports, most funders will be willing to work with you to come up with metrics that both meet their needs and the project’s needs.

Assumption Monitoring

In Chapter 5, we talked about the importance of collecting information about assumptions in the baseline study to help ensure that programs are appropriately designed from the outset. Regular monitoring of these important assumptions (e.g., about the population, governing regulations) is particularly important in fragile states because of how quickly and unexpectedly changes can occur. Examples of such changes include population migration patterns, leadership (which can dictate changes in laws), food security, economic landscape, and allegiances. Many of the methods previously described for monitoring conflict can be used to monitor assumptions. Table 7.4 uses several examples to illustrate assumptions that could change and how they might affect our interventions. Project staff should be part of the monitoring process and the learning that goes along with it (via, e.g., reports, staff meetings, bulletins) in order to identify the appropriate responses to changes in assumptions.

Table 7.4: Example: Potential project changes based on assumption shifts

If we assume… But this happens… We might have to…

A population with whom we want to work lives in a particular area

The populations has moved out or a new population has moved in

Move project locations

Expand or shrink size of intervention

Change the intervention to meet the needs of the new population

Communities are willing to devote their time to participate in the program

Communities are not willing to devote time to participate in the program

Meet with local leaders to establish why communities are not willing to devote their time to the program.

Identify ways of addressing the issues that are causing the community to stay away from the program

There is political will to support the engagement of local CBOS and NGOs in conflict management and peace building initiatives

Political restrictions hamper the program’s ability to engage with local CBOs and NGOs in conflict management and peacebuilding initiatives

Prioritize building relationships with the local and national government to ensure the program can advocate for the space for local peace actors

Propose to work with those CBOs/NGOs that operate within the government’s legal framework

It is important to note that monitoring can tell us when changes in our assumptions occur, but it does not necessarily tell us if and how to react (as in the “We might have to…” column in the table not being labeled “We should…” or “We must…”).



Monitoring Plans

In order to appropriately monitor progress against targets, each project should have a monitoring, evaluation, reporting and learning plan. A monitoring plan should include the elements in the table below: type of monitoring (it is a good idea to include all types of monitoring), indictors, decisions to be informed by data, how often to monitor the data point, and how data is collected and by whom. For crafting indicators, please refer to Chapter 6.

Table 7.5: Monitoring plan outline with examples of the three monitoring types discussed previously

Type of monitoring

Monitoring data to collect (indicators)

Decisions to be informed by data

How often to monitor

How data is collected and by whom

Conflict # of cross-border raids reported

What level of security should we maintain?

Should we maintain or postpone any key activities?

Weekly

Monthly

Interns monitoring specific news sources

Meetings with local hospital administrator

Implemen-tation

# of local leaders trained

% of those involved deeming training a “success”

Should trainings continue and at what rate?

Should training curricula be adjusted?

Quarterly Written evaluations completed by participants after training and read by program staff

Written evaluations completed by training facilitators

Assumptions # of people from “the other” community living in the target area

Should we prepare for increased incidences of violence between the two communities?

Should we increase programming efforts related to this violence?

Quarterly Staff conversations with community elders about community participation in peacebuilding initiatives


Monitoring:

Is an ongoing process, done at regular intervals throughout the life of a project

Is aligned with baseline and evaluation studies, and should be planned accordingly

Tells us what has happened and what is happening, but does NOT tell us what to do about it; Considering “what if” scenarios at the beginning of a project cycle and engaging project staff in monitoring activities will help the learning and decision-making process.

Can involve collecting information in a variety of ways, including quantitative data kept by program staff and administrators, expert and local opinions, survey data, and media content

Helps us look back at what has happened and look forward to what needs to happen, given what has already happened

Is important for tracking progress toward targets: Given where we started and where we hope to end up, has adequate progress been made?

Can be resource intensive, so it should focus on processes and activities that are essential for immediate learning, planning, and reporting needs



Learning Activity: Develop a monitoring plan

Write out a specific plan for how you will report on and learn from your project results (what decisions will be informed by the data). It is easy to lose track of monitoring without a detailed plan in place to follow. Determine in advance who will be responsible for reviewing monitoring reports and the protocols for acting on any important findings. Use the template below to prepare your plan.

Type of monitoring

Monitoring data to collect (indicators)

Decisions to be informed by the data

How often to monitor

How data is collected and by whom

Conflict

Implemen-tation

Assumptions


Chapter 8: Evaluation | page 59

Chapter 8: Evaluation


The definition of evaluation and why it is important

Different types of evaluations

Evaluation frameworks and questions

Components of an evaluation Terms of Reference (TOR)

Learning Activities

Create an evaluation plan

What is evaluation and why is it important?40

The previous chapters identified the measures and methods you can use to track data for your organization or project. However, without analysis from an evaluation angle, that monitoring data cannot tell managers why results are or are not being achieved. To get this often-crucial decision-making information, organizations must conduct evaluations to analyze the longer-term, higher-level outcomes or impacts in their programs.

Measuring success in peacebuilding programs requires going beyond just counting activities, such as the number of trauma healing (TH) activities held. Rather, we must understand what outcomes have occurred as a result of conducting TH activities in the project communities. Evaluation can focus on the change we created in the lives and minds of others.

Evaluation is most valuable when a project wants to look not only at results on a cursory level, but seeks to understand the underlying reasons why change is occurring or not occurring in the field, then uses that information to learn and adapt both its actions and its conceptual framework.

In contrast to monitoring, which provides ongoing structured information, evaluation is periodic. Evaluation focuses on why results are or are not being achieved, unintended consequences, or issues of interpretation, relevance, effectiveness, efficiency, impact, or sustainability.

An effective monitoring, evaluation, reporting, and learning (MERL) system includes a program and schedule for evaluation that is based on a learning agenda (see Chapter 9) in combination with an adaptive management approach (committed to changing course and approaches if necessary based on findings from monitoring and evaluation [M&E] data) to ensure the re-examination of management decisions/activities based on new information or learning.

Defining Evaluation

There is no universal definition for the term evaluation. British mathematician and

academician Michael Scriven (1991), one of the founders of evaluation as a field, noted nearly 60 different synonyms, based on such verbs as appraise, assess, critique, examine, grade, inspect, and judge.

40 This chapter is merely an introduction to evaluation. To read about evaluation in much greater detail before planning for an evaluation, please review Pact’s MERL Module 3, Field Guide for Evaluation: How to Develop Effective Terms of Reference, available at http://www.pactworld.org/library/field-guide-evaluation-how-develop-effective-terms-reference. Portions of this chapter are taken directly from Module 3.

http://www.pactworld.org/library/field-guide-evaluation-how-develop-effective-terms-reference

http://www.pactworld.org/library/field-guide-evaluation-how-develop-effective-terms-reference



As managers and leaders of evaluations, it is important for you to understand how others may understand the term. A common language for evaluation helps us all communicate better.

In this chapter, we will present several common definitions. None is particularly better than another. Instead, each emphasizes a different aspect of evaluation, including its purpose and utility. Understanding the similarities and differences among these definitions will directly help us manage evaluation work in our communities.

According to Michael Quinn Patton (1997, p. 23), a leader in the field of program evaluation, evaluation is “the systematic collection of information about the activities, characteristics, and results of programs to make judgments about the program, improve or further develop program effectiveness, inform decisions about future programming, and/or increase understanding.”

According to the Organization for Economic Co-operation and Development

(OECD; 2002, pp. 21–22), evaluation is “the systematic and objective assessment of an ongoing or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact, and sustainability.”

According to the U.S. Agency for International Development (USAID; 2011, p. 2),

evaluation is “The systematic collection and analysis of information about the characteristics and outcomes of programs and projects as a basis for judgments, to improve effectiveness, and/or inform decisions about programming.”

In common, all the preceding definitions assert that evaluation is systematic,

that it is specific to a program or project, and that it can answer different

types of questions.

Systematic

First, evaluation is systematic. In other words, evaluation is grounded in a system, method, or plan. To arrive at credible conclusions, a high-quality evaluation uses consistent methods that are clearly outlined in the evaluation design. In this way, evaluation is closely related to research, and both research and evaluation use many of the same tools.

Specific

Next, evaluation is specific to a program or project; this is what distinguishes evaluation from research (see more below). For example, someone might investigate whether children who live near the garbage dump get sick more often than children who live far from the dump. This is research, but it is not evaluation. Another person could study whether children who attend a certain nutrition program get sick less often. Both studies are research, but only the second example is evaluation because it is specific to a project and the first is not.

Versatile

Last, the three definitions also show that evaluation can answer many different types of questions. For instance, an evaluation may ask:

Did the program improve the well-being of community residents?

Were resources used effectively?

What factors were most important to the success of the intervention?

Why did the program fail?



Monitoring versus Evaluation

Sometimes the terms monitoring and evaluation are used synonymously. Although it is important to both monitor a project and evaluate it, the two activities are not the same; Table 8.1 shows some of these differences. Monitoring is primarily intended to provide information about a project’s operations and outputs. Impact or performance evaluations generally look at a project on a broader and higher level, assessing whether it is meeting strategic goals. There is overlap; for example, monitoring data can be used in evaluations and some evaluations are primarily oriented toward assessing operations or processes (often called process evaluations). Consequently, monitoring and evaluation often are linked and are complementary.

Table 8.1: Characteristics of monitoring and evaluation compared41

Characteristic Monitoring Evaluation

Subject Operational management issues addressed

Usually focused on strategic aspects

Character Systematic Subject and methods flexible

Frequency Continuous Periodic

Primary client Program managers Stakeholders and external audience

Party conducting

Internal, but should also be shared with funders, stakeholders, and communities/beneficiaries

Can be external or internal

Approach Utility Objectivity

Transparency

Methodology Rapid appraisal methods

Routine data collection tools

Rigorous research methodologies

Sophisticated tools

Primary focus Focus on operational efficiency, outputs, and outcomes

Focus on relevancy, outcomes, impact, and sustainability

Objectives To identify and resolve implementation problems

To assess progress toward objectives

Learning for adaptations

To check outcomes and impact

To verify developmental hypothesis

To document successes and lessons learned

Evaluation versus Research

Both research and evaluation systematically seek answers to questions; in fact, they use many of the same techniques to answer those questions. However, their purposes sometimes differ (see Table 8.2). Research usually seeks to create new and generalizable knowledge or understanding, while evaluation seeks to inform decisions and judgments about a specific project.

41 Adapted from Jaszczolt et al. 2003.



Table 8.2: Differences between research and evaluation42

Factor Research Evaluation

Purpose To add to knowledge in the field

To develop laws and theories

To make judgments

To provide information for decision-making

Who sets agenda or focus

Researchers

Academic institutions

Stakeholders and evaluator jointly

Generalizability of results

Important to add to theory/contribute to field

Across the program areas, but not necessarily beyond the program

Focus is on particulars of the program or policy and the context

Intended use of results

Usually for publication and knowledge sharing

Usually directly affects the project’s activities or decisions of stakeholders in development of future projects

Why evaluate?

Demanding time and resources, evaluation often competes with resources that are needed to implement programs or deliver services. Many managers ask the question, “Why evaluate?” The Patton, OECD, and USAID definitions, above, suggest clear reasons:

To measure a program’s value or benefits

To improve a program or make it more effective

To better understand a program

To inform decisions about future programs

Other reasons to evaluate include:

To demonstrate to planners, funders, and other decision-makers that program activities have achieved measurable improvements

To understand whether and where resources are being used efficiently and where resources may need to be used differently

To be accountable to funders and community members

To learn about which interventions work and which do not, and why

Knowing why a program is being evaluated is essential to the evaluation’s success. After all, evaluations are meant to be used. How an evaluation is used depends on what questions have been asked, the reasons for evaluating the program, funder requirements, timing, and other factors. Evaluation reports sometimes sit on shelves gathering dust. However, if we are clear about an evaluation’s purpose, if the evaluation is conducted systematically, and if the right questions have been asked about the program during the evaluation, the results should be useful and actionable.

Types of Evaluation

Types of evaluation vary by purpose and program stage. The five main types are:

Formative

42 Fitzpatrick et al. 2011.

Summative Process Outcome Impact



Most useful during program design and early in the implementation phase, formative

evaluations examine how a program, policy, or project is implemented, whether or not the

program Theory of Change (TOC) corresponds with its actuality, and what immediate consequences the implementation produces.

Summative evaluation is the final assessment at the end of a project. A summative

evaluation determines the extent to which anticipated outcomes were produced. It is intended to provide information about the program’s worth. Results help decide whether to continue or end a program.43

Sometimes called an implementation evaluation, a process evaluation looks at

whether a program has been implemented as intended: whether activities are taking place, whom they reach, who is conducting them, and whether inputs have been sufficient.

Outcome evaluation, also known as performance evaluation, examines a project’s

short-term, intermediate, and long-term outcomes. While process evaluation may examine the number of people receiving services and the quality of those services, outcome evaluation measures the changes that may have resulted in people’s attitudes, beliefs, behaviors, and health outcomes. Outcome evaluation also may study changes in the environment, such as policy and regulatory changes.44 Outcome evaluations usually occur at midterm or endline.

The most rigorous types of evaluation are impact evaluations, which use statistical

methods and comparison groups to attribute change to a particular project or intervention. USAID defines impact evaluations as evaluations that:

Measure the change in a development outcome that is attributable to a defined intervention; impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual [examination of what would have happened if the intervention did not exist] to control for factors other than the intervention that might account for the observed change.45

The counterfactual is what differentiates the outcome evaluation from an impact evaluation.

Internal and External Evaluation

Whether an evaluation is internal or external depends on who is conducting it. An internal evaluation is conducted primarily by a member of the organization implementing the project. An external evaluation is led by a consultant or other person who does not regularly work for the organization.

Internal evaluations may allow for a more complex, multi-stage evaluation design and can take advantage of in-house staff members’ understanding of the project, either to produce the evaluation more efficiently or to yield more nuanced findings.

External evaluations can be (or can be perceived as) more objective and can bring additional expertise and external perspective that can add value to the evaluation.

Which type of evaluation uses resources the most efficiently depends on an organization’s capacity. Often, evaluation involves both internal staff and external consultants in a joint effort that can leverage the strengths of each.

43 World Bank 2007.

44 U.S. Centers for Disease Control and Prevention (CDC) 1999.

45 USAID 2011.



Involving Stakeholders in Evaluation

It is important to involve stakeholders—representatives of all the people with an interest in the project—in all stages of the evaluation process. Stakeholders fall into three general categories:46

People involved in program operations (e.g., staff, partners, funders)

People served by or affected by the program (e.g., clients, community members, officials)

People who intend to use the evaluation results (e.g., staff, funders, government, general public)

It is important to identify and understand stakeholders, whether they are beneficiaries, project implementers, funders, or another audience. Research has demonstrated the value of the personal factor, and a key component to successful use of evaluations is “the presence of an identifiable individual or group of people who personally care about the evaluation and the finding it generates. Where such a person or group was present, evaluations were used; where the personal factor was absent, there was a correspondingly marked absence of evaluation impact.”47

Barriers to Evaluation

If evaluation is important, why does it not always happen or happen well? What stands in the way? Key factors are:

Lack of time, knowledge, and skills

Lack of resources for evaluation, including a limited budget

Poor project design, for example, evaluation activities not being integrated into project design

Start-up activities competing with baseline measurement or delaying baseline measurement

Project capacity overwhelmed by complex or overly ambitious evaluation designs

Fear of the consequences of negative findings

The perception of M&E as “police work” or “auditing,” that is, a fault-finding exercise

Arguments by stakeholders that M&E resources would be better spent on program expansion

Difficulty convincing others how useful evaluation will be as a learning exercise

The perception that because no baseline data was collected, it is too late to evaluate

Barriers to program evaluation are worth overcoming so we can learn what works and what does not enables us to better serve the needs of our communities.

Planning for an Evaluation

Evaluation objectives

Determining the components of your evaluation objectives will be the most challenging part of the planning process. It is easy to become overwhelmed by all of the possible “things” that could be measured. It is neither advisable nor feasible to measure everything. Instead, consider prioritizing your objectives. Ask:

What is most important for each of your audiences to learn and why?

What will be done with the knowledge gained?

46 CDC, 1999.

47 Patton, 1997, p. 44.



There are several useful frameworks available to help you choose your priorities and evaluation questions in a systematic way. Framework 1 is more common across a range of different types of development programs; however, Framework 2 is applicable specifically for peacebuilding programs.

Framework 1: Criteria for evaluating development assistance48

The OECD Development Assistance Committee criteria are:

Relevance Effectiveness Efficiency Impact Sustainability

Relevance “The extent to which the aid activity is suited to the priorities and policies of the

target group, recipient and funder.” This criterion is used to determine if (and to what degree) the activity is (still) aligned with ultimate goals and objectives (near- and long-term) and whether those objectives have changed (as the situation has evolved) or remained the same.

Example question: To what extent does the concentration of aid on peacebuilding correspond to the needs in the communities?

Effectiveness “A measure of the extent to which an aid activity attains its objectives.” This

criterion is focused on determining if and why or why not the activity is (or is not) working in a way such that the activity’s goals have been or are on the right track to being achieved.

Example question: To what extent has the aid contributed to reduced incidents of violence?

Efficiency A measure of “the outputs—qualitative and quantitative—in relation to the

inputs.” This criterion examines cost-effectiveness and timeliness (internally and as compared to other approaches).

Example question: Have implementation techniques allowed the same results to be achieved with less budget?

Impact “The positive and negative changes produced by a development intervention,

directly or indirectly, intended or unintended.” This criterion looks deeper into the effects of interventions and considers higher-level outcomes, impacts, and peacebuilding goals. An example question for a peacebuilding program could be: What is the long-term impact of TH initiatives on peace and reconciliation in project areas?

Sustainability Measures “whether the benefits of an activity are likely to continue after

funder funding has been withdrawn. Projects need to be environmentally as well as financially sustainable.” This criterion takes a longer-term view than even impact to look at whether interventions resulted in perpetual long-term changes. This is the most difficult of the criteria in this model to assess.

Example question: To what extent has the aid contributed to durable peace in the communities?

Framework 2: Conflict transformation framework49

The following framework, taken from Church and Rogers (2006), follows the same evaluation criteria as Framework 1, but includes detailed examples of data to evaluate for peacebuilding programs particularly (organized along themes).

48 Framework information and all quotes taken from OECD Development Assistance Committee, n.d. 49 Church & Rogers, 2006, as modified from Church, C., & J. Shouldice. 2002. The Evaluation of Conflict Resolution Interventions: Framing the State of Play. Belfast, U.K.: INCORE at Ulster University. p. 100.



Theme 1: Goals and assumptions This theme explores the justifications for specific

interventions and their methodologies.

Relevance

Attempts to discover whether a given intervention is appropriate for the situation (context and conflict), which has likely changed since initial program design

Assesses whether the intervention is still the appropriate means by which to achieve ultimate peacebuilding goals

Closely related to conflict and context assessment and monitoring

Important in highly volatile environments

Should be pared with outcome identification (Theme 3) to truly understand the intervention’s effectiveness

Is more often included as part of formative evaluations than summative evaluations

Strategic Alignment

Determines whether program activities and intended outcomes are aligned with organizational values and mission

Primarily used to validate internal assumptions about whether a project fits within an organization’s mandate (at a local or international level) or if it has gone off course

Is distinct from looking at an intervention’s alignment with national strategic peacebuilding efforts

Theme 2: Effectiveness and efficiency This theme looks at what was done and the

extent to which it was done efficiently.

Management and administration

Explores communication effectiveness, team dynamics and supervision, logistical effectiveness, and other administrative and managerial processes

Determines how well the program is operating (or was operated) and whether the overall process and individual components were efficient

Useful to consider if resources could have been more-efficiently used if the project was done another way

Cost accountability

Examines the cost-effectiveness of the intervention

Asks if resources were spent wisely (efficiently) and were accounted for appropriately

Practical applications include budget management, funder reporting, and cost projecting

Goal is not simply spending as little money as possible, but the best possible use of resources given the challenges and realities of operating in a time of conflict

Implementation process appraisal

Focused on the quality processes employed in the intervention

Examines how well things are done with respect to known or established standards

Looking in-depth at how activities are carried out is critical for establishing what does and does not work; identification of poor processes can explain poor results

Valuable for projects promoting innovative activities or models



Theme 3: Range of results This theme measures outputs, outcomes, and impacts of

the intervention(s).

Output identification

Measures near-term, (often) tangible results

Determines what happened and to what degree (how much/many)

Very commonly collected data; useful for meeting funder requirements and monitoring specific, countable targets

Can have reasonable and reliable information earlier in project timeline than outcome or impact assessments

Outcome identification

Assesses the changes brought about by the intervention

Steps beyond “what happened” to “what happened next” and “to whom”

Looks at both positive and negative, intentional and unintentional outcomes

Essential to look both at outcomes related to program objectives and other possible outcomes not initially considered50

Impact assessment51

Evaluates the role of an intervention’s outcomes in the larger conflict

Determines the extent to which the outcomes of activities and processes lead to changes in the overall conflict.

Should include both positive and negative, direct and indirect, intentional and unintentional consequences

Adaptability of change

Explores the extent to which change is both sustainable and adaptable

Asks whether the initial changes seen will be able to withstand and adjust with the changing environment52

Requires an extended program cycle to observe changes and adaptation to new conflict scenarios

Writing an Evaluation Purpose Statement

Deciding on the purpose of the evaluation should begin only after the program logic is clear and stakeholders are engaged. At its most basic, the purpose will be to see whether the program is having the desired results. But, the formal statement of purpose goes one step further; it helps answer the question, “Why do we want to know these answers?”

50 Looking only at anticipated program outcomes is a criticism of many M&E practices. Looking for more open-ended views of change helps ensure that results don’t just include what we looked for or wanted to happen. We can miss out on other outcomes simply by not asking questions outside the scope of our objectives. See Stave, 2011.

51 It is important to reiterate that in this context impact assessment implies looking at long-term, broader-scale changes on the community that result from activities and processes among smaller populations within that community. Identifying impacts does not necessarily prove causality; that is, even if you can prove that a large change (impact) has occurred, you cannot attribute it to a specific intervention component. The systems that influence change are vast and complex, and isolating the factors that lead to change is impossible in some situations. This distinction is important because the term impact assessment is sometimes used to describe an evaluation whose goal is attribution. Such studies are highly controlled, costly, and difficult to implement, even in stable environments.

52 For example, consider a reconciliation program that is successful in reducing people’s view of “the other” and lessens incidences of violence and feelings of animosity between the groups. If a new “other” group migrates to the region, will the same principles be applied and lessen the likeliness that violence and/or animosity will grow with the new group?



There may be a need to justify the program to policymakers or funders by proving that resources are being used efficiently. You may want to improve the program or strengthen the organizations that are a part of it. There are many possible purposes for evaluations.

But, no matter what they are, a clear and well-written purpose statement is important for clarifying the aim of the evaluation so much so that it is often required when planning evaluations and writing grant proposals.

Key questions to be addressed in the purpose statement are:

What will be evaluated?

Why are we conducting the evaluation?

How will the findings from the evaluation be used?

Another way to write the purpose statement is to complete the blanks in the following sentence.

We are conducting an evaluation of [name of program] to find out ________________________________________________ and will use that information in order to __________________.

This purpose statement is an essential part of the evaluation TOR.

Evaluation Questions

All managers have questions about the programs they manage. The following questions are the raw material for creating evaluation questions.

Is the program making a difference?

Is the course of action we’re following the best way to do things?

Are the participants benefiting from the program as expected?

An evaluation question is different than the evaluation purpose (discussed above), but the evaluation questions should help to fulfill the evaluation purpose. For example, if the purpose is to influence policymakers to fund similar programs in other parts of the country, it might be appropriate to ask:

How did the communities that received the program benefit, compared with those that did not?

How cost-effective was the program?

What elements of the program were most important in creating the desired outcomes?

On the other hand, if the evaluation purpose is to show program staff how to improve the program, you might ask:

How do participants of the program perceive it?

What are the program’s strengths and weaknesses?

Why did some program sites perform better than others?

Evaluation questions used to better understand results of peacebuilding efforts

Did we meet or exceed our written objectives? If our stated objective was not reached, why not?

Did we develop mechanisms to receive feedback on our progress throughout the project? Which feedback methods allowed us to alter strategies in a timely manner and measure our impact?

Did the campaign result in positive change in the lives of our beneficiaries?

Are there fewer conflicts in project areas?

Is there more cooperation between communities in conflict?

Have perceptions changed? Are people more tolerant or less prone to partake in conflict?

Did our work have any unintended side effects?



What is evaluation design?

The evaluation design is the plan for answering the key evaluation questions. Evaluation

design is critical to the evaluation process and should begin as soon as program planning begins. The evaluation design process should involve key stakeholders.

The design specifies:

Which people or units will be evaluated

How they will be selected

The kinds of comparisons that should be made

By what approach the comparisons will be made

The evaluation’s timing

The intervals at which groups will be studied

Overview of Evaluation Approaches

Programs can be evaluated using several approaches. There are quantitative

approaches and qualitative approaches and many ways to use them both. These

approaches are often called methods, this manual uses the term approaches, because each actually encompass a variety of specific methodologies. More detail on the various approaches can be found in Chapter 3 of Pact’s MERL Module 3; however, Table 8.3 succinctly summarizes qualitative versus quantitative approaches. Many solid evaluation designs strive to include both qualitative and quantitative approaches, called mixed methods.

Table 8.3: Features of qualitative and quantitative approaches

Approach characteristic Qualitative approaches Quantitative approaches

Aim Identify common themes and patterns in how people think about and interpret something

Classify features, count them, compare them, and construct statistical models that explain what is observed as precisely as possible

Evaluator relationship to results

May only know roughly in advance what he/she is looking for

Decides in advance what he/she is looking for

Data form Words, ideas, pictures, or objects Numbers

Research focus Focuses on fewer selected cases, people, or events

Measures a limited range of responses (forced answer) from more people

Main advantage of approach

Greater depth and detail is possible Facilitates comparison and statistical aggregation for concise conclusions

Analysis methods Categories for analysis must be developed and are specific to the particular evaluation

Uses standardized measures that fit various opinions and experiences into predetermined categories, often using questions that have been verified and tested in other programs or studies

Approach to the program

Can ask questions about the program holistically or about specific parts of a program

Views components of a program separately and uses data from the different pieces to describe the whole



Creating Evaluation Worksheets

There are two different planning documents that should be considered when planning your evaluation. First, the evaluation worksheet

provides a structured outline used to design your evaluation and describes the evaluation’s objectives, questions, and information you will

collect to answer them and how you will get that information, among other things (Table 8.4). The Terms of Reference (TOR) is another

planning tool, described after Table 8.4.

Table 8.4: Example evaluation worksheet

What needs to be evaluated? (key result area or other key tenets of program)

What evaluation questions should we be asking?

How will we obtain the data? (method) Attach survey instruments as appropriate

When will we obtain the data? (dates) Who will be involved?

Example: Progress made in reducing violence in conflict areas

Outputs (process evaluation)

What have we delivered in terms of short-term change or as an immediate result of our activities?

E.g.: # people involved in TH

Can we claim knowledge or understanding, skills, or capacity change? Were there any changes relating to improved access or quality?

E.g.: Improved level of tolerance toward other community (survey)

What evidence do we have to illustrate these changes actually occurred?

Outcomes (summative evaluation)

What has been achieved as a result of the outputs?

E.g.: Decrease in incidents of violence

Have people changed their behaviors, the way they interact with other communities, and/or their underlying attitudes?

Review of monitoring data

Quarterly Community members

Focus groups with community members

October 2015 Community leaders

Key informant interviews with local officials

October 2015 All key stakeholders and staff

Survey of community members

October to November 2015

Community members (likely a sample of community members)



The Evaluation Terms of Reference

The second planning tool, the TOR, is a comprehensive, written plan for the evaluation. After completing an evaluation plan, you are ready to create the TOR to either solicit an external evaluator to conduct an external evaluation or for your own purposes if your team is conducting an internal evaluation. Essentially, the TOR describes the scope of work. It assigns roles and responsibilities, activities, and methodologies and is more narrative in form than an evaluation plan.

Developing the TOR yields a shared understanding of the evaluation’s specific purposes, questions, objectives/themes, the design and data collection approach, the resources available, the roles and responsibilities of different evaluation team members, the timelines, and other fundamental aspects of the evaluation. The TOR facilitates clear communication of evaluation plans to people inside and outside of the organization/project.

Importantly, if the evaluation will be external, the TOR helps communicate expectations to and then managing the consultant(s). Because external evaluators are less familiar with the project than the individuals commissioning them, it is important to have a TOR that clearly sets forth all the necessary background, specifically to alert the evaluator to the questions that are most important to stakeholders.

Key components of a TOR are: 1. Overview of the evaluation 2. Brief description of the program 3. Purpose of the evaluation 4. Evaluation questions 5. Evaluation methodology/approach 6. Ethical considerations 7. Implementation information: schedule and logistics, evaluation team 8. Reporting and dissemination plan 9. Application guidelines: budget, timeline

Table 8.5: Components of a Terms of Reference53

Component Description

Overview Summarizes the type and scope of the evaluation, what is being evaluated, and where

Introduces the project to potential project partners, staff, and/or consultants

Program description

or

Organization overview

Summarizes what is being done (the interventions), the projects’ intermediary and long-term objectives (outcomes and impacts), and where and with whom the project is taking place

Describes the scale and timing of the project

Provides an overview of the implementing organization

Purpose of the evaluation:

Goal

Objectives

Questions

Audience

Methods/approach

Distinguishes the evaluation goals and objectives from those of the project

Provides further detail of the evaluation plan, including lines of inquiry (the broad questions we seek to answer) organized by theme, methodologies to be employed, and audiences

Example:

Project goal: to increase peace in community X

53 Church & Rogers, 2006 and OECD Development Assistance Committee, 2008.



Component Description

Project objective 1: to decrease instances of domestic violence in the community

Evaluation goal: to improve the processes and programs offered by our organization to promote peacebuilding

Evaluation objective 1: To determine whether co-ed educational programs are more effective at decreasing domestic violence than single-sex programs

Evaluation approach: Household survey and key informant interviews with community leaders

Ethical considerations Requiring informed consent

Ensuring data is not traceable to individuals (de-identified data sets)

Implementation plan:

Schedule and logistics

Evaluation team

Describes logistical components of the implementation plan

Describes what is needed of the evaluator or evaluation team sought, similar to a job description

Reporting and dissemination plan

Explains requirements for report format

Explains how the evaluator hands the raw data

Specifies how the report will be disseminated and to whom

Application guidelines

Budget guidelines

Timeline

Contact details

Provides information for external consultants and survey firms who wish to apply to offer their services

Explains how to submit applications and by which date


There is no one universally accepted definition of evaluation, but many of the definitions refer to evaluation as being systematic, specific to a program or project, and able to answer different types of questions.

Evaluations can be designed to take place periodically or continuously alongside implementation. Three types of periodic evaluation are: baseline, midterm, and endline.

You cannot evaluate everything during an evaluation, so it is important to use existing frameworks to identify your priorities in a systematic way.

Evaluations are often outsourced, so it is import to use a template to develop scopes of work or terms of reference with ease.

Learning Activity: Create an evaluation plan

Write an evaluation purpose statement

Reflecting on your organization’s context, write an evaluation purpose statement for your program by filling in the blanks in the following sentence.

We are conducting an evaluation of ____________________ [name of program] in order to find out __________________________________________________, and we will use the information to _____________________________________.

Crafting evaluation worksheets

Take a few minutes to think about your own program, your work on your organization’s evaluation purpose statement, and your organization’s context. Identify key potentially relevant evaluation questions for your program and jot them and all relevant information down on the evaluation worksheet in Table 8.6.



Table 8.6: Evaluation worksheet

What needs to be evaluated? (key result area or other key tenets of program)

What evaluation questions should we be asking?

How will we obtain the data? (method) Attach survey instruments as appropriate

When will we obtain the data? (dates)

Who will be involved?


Chapter 9: Learning | page 74

Chapter 9: Learning


Some best practices for organization/project learning

The basics of the adaptive management approach

How to develop a learning agenda

Learning Activity

Create a learning agenda

Overview

Learning is an essential component of strong organizations and projects. It directly builds

on monitoring and evaluation (M&E) and uses the findings from M&E activities to strengthen program design and best practices. Learning is most successful when it:

Comes in the form of regular performance data learning reviews, i.e., scheduling and holding periodic learning reviews with the technical team and partners to review performance and discuss any changes in approach that may be necessary

Also occurs post-evaluation and includes action planning (course-correction if necessary)

Involves funders, project beneficiaries, and other stakeholders

Using an Adaptive Management Approach

Adaptive management is an approach to evaluation, learning, and decision-making that

involves a cycle of planning, implementation, monitoring, evaluation, research, and subsequent re-examination of management decisions based on new information that may alter existing plans and priorities.

In its simplest form, adaptive management is action in response to learning.

Adaptive management is ideal for learning about and understanding complex systems and structures because it recognizes that systems are inherently changing and unpredictable.54 Adaptive management copes with the uncertainties by monitoring decision-making results and by re-examining choices in light of these results and based on new information that becomes available.

The ability of adaptive management to provide learning opportunities and foster programming responses to complex systems makes it especially valuable in the peacebuilding field, where contexts remain extremely dynamic. Many managers of peacebuilding programs already engage in this type of assessment and analysis. By further incorporating adaptive management processes, peacebuilding programs can formalize learning to ensure sound programming decisions.

Adaptive management requires intentionally building a series of information feedback loops

into project management during the design phase. This establishes clear pathways for sharing

knowledge and information on the project from early stages of implementation. With

feedback loops engrained into the project from the start, there will be routine opportunities

54 The U.S. Agency for International Development (USAID) has written about the local systems approach in a paper that explains how development programming can best work within existing systems. See Local Systems: A Framework for Supporting Sustained Development, available at https://www.usaid.gov/policy/local-systems-framework.

https://www.usaid.gov/policy/local-systems-framework

https://www.usaid.gov/policy/local-systems-framework



to engage in learning around contextual changes and project outcomes, which will feed

directly into project decision-making.

During the design phase, projects should identify key sources of information and create clear

mechanisms for capturing and learning from this information. Practical examples of

information sources are listed below. When considering sources, projects should keep in

mind that the goal is not to gather more information, but to carefully consider what types of

information is most useful to inform their decisions.

Consultations with key informants

Basic field visit reports

Project monitoring data

Discrete applied political economy analyses (APEAs) and/or conflict analyses

Beneficiary feedback mechanisms (like community scorecards, social audits, etc.)

Close monitoring of news and social media

As part of this process, projects should include consideration for how frequently managers

want to make substantive pivots to the project. The context of the project also should be

considered. A highly volatile context may need more frequent information review, such as on

a weekly basis, to inform resource deployment, while a more stable context might only

necessitate quarterly learning reviews of key environmental and/or programmatic

information.

The adaptive management approach requires that we periodically evaluate activities and revisit our results frameworks, hypotheses, and cause-and-effect linkages (the premises on which an organization selects activities to carry out) to ensure they are still valid based on what we have learned.

Developing a Learning Agenda

Regardless of which type of learning format an organization has planned to use, it initially needs to identify what it wants to learn, how it will learn, and how frequently the learning will be done; these are the foundations of the learning agenda.

Learning agendas are similar to evaluation plans, but differ in some key ways. While questions in the evaluation plan may be similar to those in the learning agenda, the latter enables the program/project/organization to plan for how it will ensure that improvements are made along the way based on learning from immediate processes, experiences, and activity outputs. Also, evaluations tend to be more rigorous in their methodology and analysis because findings are often published for use by stakeholders outside the organization. Learning agenda questions do not necessarily require as much rigor because findings are used internally to inform continuous program improvement.

Developing a learning agenda includes the following steps. Table 9.1 provides an example of a completed learning agenda that includes information for all these steps.

1. Determine the components of your program you want to learn about and identify what needs to be assessed. To do this, review your implementation plan, deliverables, and results framework and identify key components, sub-components, or other aspects of your program that you will analyze in terms of your organization’s efficiency to implement them and/or their effectiveness in obtaining results.

2. Clarify what you want to learn about each component you identified and determine the questions you will answer. Also review the planned deliverables, results frameworks, and indicators for this step. Example questions are:



What has been learned from the project that can contribute to improved program implementation or to building relevant knowledge in the peacebuilding field?

What do we want to know about the subject?

What was changed as a result of our program?

How do targeted stakeholders perceive our programs?

What sort of reach do we have?

How many home visits are we supporting?

How was the target population affected?

How much money did we spend?

For example, if you determine that the training component should be evaluated, you might ask what evidence is available to show that the training implemented has resulted in new ways of doing things or increased participant knowledge and skills.

3. Identify how you will obtain the data. What data do you already have to help analyze this issue, and what data do you need to be able to answer your questions? For example, will you need to facilitate a focus group discussion, hire a research consultant, hold a staff meeting, or use data from specific indicators?

4. Identify who should be involved in answering the questions and in participating in analysis of the answers.

5. Determine deadlines for obtaining the data and conducting the analysis. Do you need the information every month, each quarter, at the end of the project, etc.?

6. Identify how you plan to document the things you have learned, disseminate findings, adapt your program activities, and/or update underlying premises or results frameworks, thus altering the program design.

Ensuring Learning Processes are Participatory

To ensure that a breadth of opinion is captured relating to organizational/program performance and because learning initiatives include important capacity development and growth dimensions, they should be as participatory as possible. Stakeholder involvement in learning promotes a sense of partnership among all the key people and/or groups interested in the organization. A participatory process is essential to provide insight into programs and analysis of how well the needs of different stakeholders are being met. A variety of different perspectives is particularly helpful in analyzing unintended consequences and sustainability of activities.


Learning is an important aspect for any program that wants to continually improve programming and increase chances of achieving the desired results.

Adaptive management is one method for programs that choose to conduct learning. It involves a cycle of planning, implementation, monitoring, evaluation, research, and subsequent re-examination of management decisions.

Learning Activity: Create a learning agenda

Consider your evaluation objectives from Chapter 8 and identify the types of questions you would like answered during your program implementation. Complete the learning agenda matrix in Table 9.2 (on the next page) to address a couple of these objectives and questions.



Table 9.1: Example learning agenda

Which program components do we want to learn more about?

Which questions do we want answered or explored?

What data do we have to help us analyze this question/topic?

What further data will we need to acquire?

Who should be involved in either giving input or analyzing information?

When will we obtain the information or conduct analysis?

When and how will we disseminate the data and adapt our program?

The trauma healing (TH) activities across two communities during our five-year project

Do our TH sessions lead to increased understanding of the relation between trauma and conflict?

What kinds of resolutions do TH participants make that demonstrate willingness to forgive or reconcile with the other community(ies)?

What are we learning about the kind of community members to target for TH sessions?

Pre/post questionnaires

Most Significant Change stories from TH participants

Attendance records for the TH sessions

Interviews and surveys with TH participants and community members during mid and end term evaluations to establish actions taken towards reconciliation

Program staff

TH participants

Other community members

At the beginning and end of each TH session collect pre- and post-test data

3–6 months after completing the TH sessions, collect MSC stories

Midterm

Endline

Pre and post-test training results discussed at quarterly program review meetings

MSC stories analyzed annually and findings discussed at the annual reflection meeting

Midterm and endline assessment completed in years 3 and 5

Following midterm, results framework reviewed for remaining project



Which program components do we want to learn more about?

Which questions do we want answered or explored?






Peace dividends in corridors between two communities

Are peace dividends benefitting two or more communities?

Are peace dividends contributing to increased interaction among members of previously conflicting communities?

Are certain peace dividends more inclined to increase interaction between communities than others?

Quarterly peace dividend utilization reports

Stories of Most Significant Change resulting from the peace dividend

Context updates from program staff and implementing partners

Observation of how members from different communities are accessing the peace dividend

Interviews and surveys with participants to establish if peace dividends are changing attitudes and behaviors during the mid and end line evaluations. Are communities more willing to reconcile? What reconciliation actions have they taken?

Changing context through periodic context analysis

Program staff

Community members

Quarterly during the quarterly review meetings

6-12 months after completing construction of the peace dividend collect MSC stories from members of the two communities

Midterm and endline evaluations

Peace dividend utilization reports and context updates discussed at quarterly program review meetings

MSC stories analyzed annually and findings discussed at the annual reflection meeting

Midterm and endline evaluation completed in years 3 and 5

Following midterm, results framework reviewed for remaining project



Table 9.2: Learning agenda matrix

Which program components do we want to learn more about or evaluate?

Which questions do we want answered or evaluated/ explored?







Chapter 10: Reporting | page 80

Chapter 10: Reporting

Chapter Objectives In this chapter, you will learn:

What reporting consists of

Why reporting results, milestones, and successes to stakeholders is important

Different types of reports and communication tools typically produced by organizations

Formats for quarterly and final project reports

How to write success stories

Learning Activity

Review progress reports

Overview

Reporting, or communicating your project results, milestones, and successes to stakeholders, is an essential component to ensuring that the information you collect is put to its best possible use. It is important to know who can best use the data and how. You also need to know how to turn the data into useful knowledge that can be used by decision-makers. This chapter will show you how to easily turn your monitoring, evaluation, and learning data into reports and communication tools that can be broadly distributed among a wide range of audiences and that are useful both for your management and your funders.

Introduction to Reporting

A report is a compilation of descriptive information. It is a communication tool to present monitoring, evaluation, and research results by presenting raw data and information as knowledge. A report is an opportunity for project implementers to inform themselves and others (stakeholders, partners, funders, etc.) on the progress, difficulties encountered, successes, and lessons learned during implementation of programs and activities.

Reporting enables assessment of progress against work plans and helps focus audiences on the results of activities, enabling the improvement of subsequent work plans. Reporting helps form the basis for decision-making and learning at the program level. Reporting also communicates how effectively and efficiently the program is meeting its objectives.

Effective reporting and communication plays an important role in peacebuilding by providing accurate and unbiased information to populations affected by conflicts and by proactively delivering programs that aim to impact knowledge, attitudes, and behaviors of the population taking part in peacebuilding efforts. Reporting also can enhance receptivity, a sense of ownership and partnership when shared with community members and stakeholders, and is an effective process for promoting accountability and transparency by project partners.

A good report:

Focuses on results and accomplishments within the context of the project

Assesses performance over the past reporting period, using established indicators, schedules, baselines, and targets

States explicitly whether and how much progress or results surpassed, met, or fell short of expectations and why

Specifies actions to overcome problems and accelerate performance, where necessary

Explains the influence of comparative performance by objectives on the resources needed

Identifies the need to adjust resource allocations, indicators, or targets, where necessary



Discusses the way forward for programming in light of the findings; annual or final reports also may address prospects of successful program closeout and expected sustainability of results

Identifying Appropriate Communication Tools

Funders often require four types of communication, in addition to regular financial information:

Events and incident reports

Written progress reports (quarterly and annual)

Evaluation reports (baseline, midterm, and endline)

Final performance report

However, reporting should not end with funder requirements. Other types of communication tools you could use include:

Oral presentations/lectures

Discussion sessions/community meetings

Informal contacts and conversations

Press and media releases

Brochures and pamphlets

Formal academic papers and books

Visual presentations (e.g., videos, photos)

Internet, email, and websites

Plays, music, and dances

Use the following steps to determine the most appropriate communication tools for your organization or project’s needs.

1. Identify your audiences’ information needs. For each audience, ask yourself what key information you want to communicate.

2. Determine how you will report to each audience by selecting a tool/format that best suits the information you want to convey. Think about their primary interest in the organization/project to help you decide on the data to report.

3. Review your information database and identify what data you have to address that interest.

Reporting Schedule

Table 10.1 illustrates one way to manage various reports required by funders, internal staff, and other stakeholders. The process for writing, compiling, and submitting reporting, as shown in the table, occurs in order from left to right.



Table 10.1: Example reporting schedule

Type of report Person writing the report Time frame

Information source Reviewers Submitter Submitted to

Incident Partner organization staff

Immediately after conflict occurs

Program staff and beneficiaries

M&E manager and peacebuilding staff

Field staff from the partner organization or primary recipient

Primary recipient

Conflict assessment Sub-grantee/partner organization staff

With each quarterly report

Incident reports, APEA study, and project records

Primary recipient’s peacebuilding staff

Project lead or head from the partner organization

Primary recipient

Partner quarterly Sub-grantee/partner organization

15 days after the end of the quarter

Activity leads and activity records

Head of the partner/sub-grantee and the primary recipient M&E manager

Project lead and/or head of partner organization

Primary recipient

Quarterly programmatic

Primary recipient’s technical staff and M&E manager

30 days after the end of the quarter

Partner reports and project records/database

Chief of party (COP) or deputy chief of party (DCOP) and technical leads

COP or DCOP Funder

Annual programmatic

Primary recipient’s technical staff and M&E manager

Thirty days after the end of the project year

Partner’s/sub-recipient’s quarterly reports and project records/database

COP or DCOP and technical staff, plus headquarters staff

COP or DCOP Funder

Program/project evaluation

Evaluation consultant

Following the baseline, midterm, and final evaluations

Study data and project information

COP and M&E manager

Evaluation consultant to primary recipient and primary recipient to funder

Funder



Generic Outline for Quarterly and Annual Progress Reports

Quarterly and annual progress reports are the most common and important format for conveying information about a project to its funder(s), partners, and stakeholders. Progress reports should focus on presenting in a concise format the advances (or lack thereof) made on a project during a specific quarter or year. The report should maintain a neutral tone and should discuss any “failings” and “challenges” directly and dispassionately.

Quarterly reports often are laid out in the following manner.

1. Executive summary or highlights

This section describes and summarizes the significant activities and results during the reporting period, either in paragraph form or laid out as bulleted highlights, depending on funder requirements. While the information conveyed here should be addressed more fully elsewhere in the report, the executive summary or highlights may be the only portion of the report many people will read. So, this section, in a maximum of two pages, should briefly describe the:

Program background

Qualitative progress/impact

Quantitative progress/impact

Constraints and opportunities

Program administration

Subsequent quarter’s work plan (omit this section in an annual report)

2. Introduction/program background

This section should include standard language in one to two paragraphs about the project’s objectives, beneficiaries, and funders and what the report includes.

3. Key achievements (qualitative progress)

Provide a narrative description of the key achievements during the reporting period. Sort results by location or other geographic region or by type of project activity, as appropriate. The structure of this section should closely follow the project’s conceptual framework so the reader can easily see the relationship between activities (outputs) and the resultant achievements (outcomes).

4. Key achievements (quantitative progress)

Provide a quantitative description of the key achievements of the reporting period, and complete the indicators table provided in the reporting template with the basic data required to assess progress toward achievement of targets under each development objective.

5. Other performance monitoring updates

In a maximum of one page, describe other performance monitoring activities conducted in the reporting period not explained in above sections. If data quality issues were identified, please explain any action(s) proposed or taken to address any/all issues. Where possible, please include additional performance or learning information that was collected during routine monitoring activities (e.g., site visits, pre- and post-tests). Raw data collection materials (e.g., site visit forms) may be included as attachments.



6. Constraints, opportunities, and lessons learned

In a maximum of one page, address any significant operational challenges, including changes in scheduled or planned implementation tasks and/or issues related to coordinating tasks. Discuss changes in activities that may be required due to practical considerations “on the ground,” including any changes in scheduled or planned implementation tasks. If activity resources need realignment or new “windows of opportunity” have opened, discuss these as options. This discussion should be limited to statements of facts and conclusions that might be drawn from the facts. If issues from the previous reporting period remain significant, be frank and open in discussing any step(s) toward their resolution.

This section also can include agreements between you and your funder to take specific actions. If decisions are made (in consultation with the funder) to either change the geographic location or the strategic elements being emphasized in the activity, they should be noted here for the official record. If contract modification or amendments are needed, they can be described in the administrative review section.

7. Administrative review

In a maximum of one page, discuss the status of your program administration. During the reporting period, were there any changes in staffing/management, institutional strengthening plan actions, contract modifications or amendments to the program?

8. Progress on funder priorities

In a maximum of a half page, identify any contributions to funder priorities, such as Value for Money,55 localization of foreign aid,56 environmental impact or gender. It is appropriate to discuss how the project’s focus reflects the signature campaigns of or global initiatives for project funders.

Examples of funder priorities could include:

a. Progress on counter-terrorism and violent extremism

In a maximum of a half page, provide an update on the national or local government’s terrorism-prevention programs, while looking at effectiveness of nongovernmental, community-based initiatives and the program being implemented in countering violent extremism in the conflict areas.

b. Progress on climate change and natural resource management

Many rural populations are heavily reliant on natural resources for their livelihoods. In a paragraph, describe how violent conflict has influenced vulnerability to climate change and natural resource impacts during the reporting period for people living in affected areas. In addition, discuss stress that climate change is having on these resources and how conflict-related barriers to accessing pasture and water for livestock may increase negative coping mechanisms in response to climate-related shocks and stresses. Further, and as applicable, show how violent conflict erodes the possibility of collaborative natural resource management, which is crucial for adaptive capacity to climate change.

55 Value for Money is a DFID requirement. Guidelines are available at https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67479/DFID-approach-value-money.pdf.

56 Localization of foreign aid is a USAID requirement. See https://www.usaid.gov/sites/default/files/documents/1870/LocalSystemsFramework.pdf.

https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67479/DFID-approach-value-money.pdf





9. Following quarter’s work plan

In a maximum of one page, expand on the amount of detail from the annual work plan for the subsequent quarter, either by listing action items or offering a three-month table, calendar, or timeline describing the completion of an activity, such as in Table 10.2. You may omit this step in an annual report or final performance report.

Table 10.2: Example projected work plan for the following quarter

Activity

January to March 2015

January February March

Program start-up and cross-cutting issues

Cross-cutting and start-up activities

Result 1.1: Communities more open to social reconciliation

Activity 1.1.1: Trauma healing

Activity 1.1.2: Peace dividends

Activity 1.1.3: Cultural adaptation

Result 1.2: Communities peacebuilding capacities mobilized

Activity 1.2.1: Promote local peacebuilding leadership

Activity 1.2.2: Promote youth leadership and engagement opportunities

Activity 1.2.3: Expand the impact of peacebuilding organizations

Activity 1.2.4: Establish/strengthen peace networks

Result 1.3: Local governments partner with their cross-border counterparts and communities in conflict management

Activity 1.3.1: Promote emergence of local-government peacebuilding leadership

Activity 1.3.2: Expand impact of local-government peacebuilding initiatives

Activity1.3.3: Promote partnership between local government and peace networks

Generic Outline for Final Performance Report

Funders usually require that recipients submit a final performance report at the end of the funding cycle. Final reports often are laid out in the following manner.

1. Executive summary

This section captures the essence of the report and provides an overview of its contents. It is the last section to be written and does not exceed two pages.

2. Introduction

This section presents a very concise overview of the need for and history of this program in a couple paragraphs. It describes the results, objectives, context, and activities that were anticipated under the program during the period of agreement.



3. Methods and approach

This section briefly describes, in at most a page, the methods and approaches used to implement the program.

4. Results/impact

This section compares planned versus actual achievements. In this section, you should:

Summarize program accomplishments or failings

Present findings as to why progress toward planned results was unexpectedly positive or negative

Present findings on how well needs of different customers were met (e.g., by gender, age, ethnic group)

Present indicator results/tables and anecdotal information to support findings

Assess the value of the program’s contribution and clarify exactly how the achievement of your objectives contributed to the development outcome and impact

Review the validity of hypotheses and assumptions underlying the results framework based on lessons learned in implementation

Describe mitigating factors that disrupted what was planned and the organization’s response to the disruption

Describe the facilitating factors that helped spur results

Identify and analyze unintended consequences and effects of assistance activities

5. A summary of the prospects for sustainability

In a page or less, describe the progress made in meeting the sustainability objectives of the program, including the approaches used to build financial sustainability (e.g., local financing, cost recovery, resource diversification, corporate sponsorships). Present any cost leveraging achieved during the project. Identify what the beneficiary communities say about sustaining project services through alternative funding sources at project close.

6. Review of deliverables

Review the deliverables submitted, preferably in table format for easier reading. Explain any unfinished work and recommend whether and how it should be completed.

7. Lessons learned/best practices/recommendations for future programs

Distill lessons learned or best practices identified through program implementation that may be useful elsewhere and/or by others. Recommend how program activities and outputs could continue or what new programs should/could be developed.

Appendix 1. Fiscal report

Describe the budget in detail, including closeout matters, cost share results, and how any matching funds were used.



General Writing Tips

Use active voice, for example: - Passive: Wells were tested by 80 percent of the participants. - Active: Eighty percent of the participants tested their wells.

Use short, complete sentences.

Write in plain English, using simple words and avoiding jargon.

Be concise.

Avoid abbreviations/acronyms.

Use your name, program name, or county name rather than “this agent,” “I,” “our,” or “we.”

Write in paragraph style, with a topic sentence.

Success Stories and Photography

Success stories are human-interest pieces that illustrate project outcomes and impact by

detailing an individual’s or community members’ positive experiences as a result of the program. They include the when, what, where, how, and why of projects outcomes, similar to a newspaper article. Appendix 1 presents some general guidelines and tips on how to write success stories and take photographs, with links to specific examples of success stories from some key funders.

Success stories are written to:

Show accountability for public funds and/or funder funds

Share successes so individuals inside and outside the program can learn from our work and results

Illustrate program outcomes and impact

Demonstrate the “human feel” that numbers often don’t show

Success stories are used:

To assess team progress in relation to planned activities

To share information among existing partners and generate interest among potential partners

To help partners and the general public better understand the program’s work and value

For funder reporting purposes, including for funders to report to their stakeholders and superiors

To post on program-area websites

As a source of news stories for local media

To document activities and accomplishments of program and staff

Photographs are images that convey events or ideas. Photographs are an important

component of success stories and help bring them to life.




Reporting entails compilation and communicating useful data and information relating to the program, including for use in decision-making.

A good report is able to convey required information to the person who needs it when he/she needs it.

The most important program reporting tools include quarterly and annual progress reports, final performance reports, events and incident report forms (both written and phone submissions).

Main sections of the quarterly and annual progress reports are the executive summary/highlights, introduction/background, key achievements (qualitative progress/impact), key achievements (quantitative progress/impact), constraints/opportunities/lessons learned, progress on any special funder priority areas, and subsequent quarter’s work plan (omitted in annual progress reports).

It is important to observe reporting schedules to ensure information is conveyed to those who need it on time.

Success stories with photographs helps communicate change in ways that conventional reports do not.

Learning Activity: Review progress reports

Review past and recent quarterly and annual progress reports submitted by your program based on the guidance provided in this chapter. Share with the group some of the obvious mistakes seen in the report and discuss how best to fix these mistakes.


Chapter 11: Data Quality and Ethics | page 89

Chapter 11: Data Quality and Ethics


Key data quality management concepts and assessment criteria

Steps for addressing common anticipated data quality issues during routine data management processes

Factors or tips to remember when creating a good data management system

Learning Activities

Review key data quality issues and indicator plans

The Importance of Ensuring Quality Data

Data quality refers to the accuracy and precision of the information collected against

monitoring and evaluation (M&E) standards. It focuses on ensuring that the process of data capture, verification, and analysis is of a standard that meets the requirements of an internal or external data quality assessment (DQA) or audit.

We can select the best indicators and write the best protocols, but if tools are not properly used and if standards of reporting set by the organization are not respected in terms both of quality and timeliness, data could be at risk of poor value and/or could be over/under/or miscounted.

Commonly used Criteria for Assessing Data Quality

There are five key criteria commonly used to assess data quality: 1. Validity 2. Reliability 3. Timeliness 4. Precision 5. Integrity

Issues and risks relating to data quality need to be thought through and documented to ensure quality standards are developed and maintained. Thus each organization or project needs to develop and document how it checks the following.

1. If the data adequately represent performance and are thus valid

Specifically, we need to check if there are any issues with:

Face validity: Is there a solid, logical relation between the activity or program and what is being measured, or are there significant uncontrollable factors?

Data transcription: Are steps being taken to limit transcription error (e.g., double keying of data for large surveys, electronic edit checking program to clean data, random checks of data entered by supervisors)?

Sampling methodology: If we report on data where we use a sample (rather than census- or case study-derived data), we need to be sure there are no significant measurement or representation errors (i.e., our samples are representative, the survey questions are clear, response rates were sufficient, the population was appropriate, sampling methods accurate). We also need to be sure that anyone who is collecting data is appropriately trained and that we account for potential bias on the data collector’s behalf.



2. If the data collection processes are stable and consistent over time

and are thus reliable

Are data collection procedures consistent (e.g., from one reporting period to the other, from location to location, is the same instrument being used)? Are we checking the data to ensure it is correct and free from errors? Are data problems reported? Would we get the same results if someone else went out again to collect the same data in the same way?

3. If the data is collected frequently enough and is current, and thus timely

Is data coming in on a frequently enough to inform program management decisions? For example, are you receiving training feedback data before planning the next round of training (so as to use learning from those training to shape the upcoming training?) Do people report when they are supposed to? Data that is required today but not received until tomorrow might not still be helpful.

4. If the data have an acceptable margin of error and are thus precise

Is the margin of error less than the expected change being measured?

5. If there is data integrity

Are data free from bias or manipulation? Are there procedures in place to reduce the possibility that data are manipulated for political or personal reasons? Does an independent review, assessment, or audit of the data take place?

Data Management

Data management process involves collecting, collating, analyzing, reporting, using, and storing project activity information. There can be no monitoring, evaluation, reporting, and learning (MERL) without a good data management system, with the components described below.

1. Data source

This is where the data originates, such as:

Training sign in sheets

Village Elders records

District peace committee meeting minutes

Community peace agreement records

Interview transcripts

Questionnaire responses

2. Data collection

Data collection is the process of gathering data generated from the various activities implemented by an organization and relevant to an organization’s MERL framework. Data collection involves obtaining data from original sources and transferring it into tools (paper or electronic) from which it can be analyzed or transferred to another data analysis system for analysis and reporting. Many times, data quality is compromised at this stage; therefore, you should exercise caution during data entry—garbage in, garbage out! See Chapter 12 for tips on collecting data through electronic means.



3. Data collation

Data collation refers to aggregation of data from different sources, such as different program sites or different field workers who collect the data, into summarized formats. Collation can be done electronically using MS Excel spreadsheets or databases or manually using paper-based systems. For example, you likely want to sum up female and male participants of different dialogue sessions across regional sites/partners in the quarter to get the total dialogues conducted and total female and male participants who attended peace dialogues implemented by the organization in the quarter.

4. Data analysis

Data analysis involves reviewing and manipulating data to assess progress made toward desired objectives and targets. Analysis enables data users to:

Associate variables (test underlying theories or assumptions)

Predict relationships (cause and effect/outcomes)

Indicate confidence in results

5. Reporting

Reporting entails compiling descriptive data and presenting raw data and information generated from data analysis as useful knowledge. Reporting provides an opportunity for project implementers and others to learn about the progress, problems, difficulties encountered, successes, and lessons from implementing program/project activities. Remember, if it is not reported it never happened because others do not know about it. See Chapter 10 for more on reporting.

6. Data use

Data use refers to making timely data-driven decisions that relate to program/project implementation. Information need to be available when required so it serves the purpose for which it was collected, then is used for learning.

7. Data storage

Data storage entails securely keeping and maintaining information or data collected for current and future reference. Data storage is done best when both electronic and hard copy (original data source documents) are well organized and kept safe.

Getting Started with Developing Systems to Address Data

Quality Issues

The indicator plan and/or protocol provides a starting point for identifying potential issues that may affect data quality.

To get it right from the start, it is absolutely essential that all indicators are defined upfront to avoid potential problems with data validity or reliability that could arise from indicator definitions being unclear. Indicator protocols allow programs to address common data validity and reliability problems by providing standard definitions of program indicators and related units of measure.



Staff responsible for MERL must ensure common indicator understanding and interpretation by program team members who would be involved in collecting or analyzing indicator data coming through the system at any given time. This could best be done through a team meeting or partners/stakeholders MERL training. It is also important for the program manager(s) to consider and determine the likely risks to managing the individual indicator data right from source to data use.

The “data quality issues” section of the standard USAID performance indicator reference sheet (PIRS)57 allows programs to document all key data-management processes and the anticipated data quality risks that may affect data use. This also allows the program MERL and/or program managers to pay attention to key data quality considerations, such as timeliness of the data, which is addressed by detailing the frequency and timing of data acquisitions and analysis. Data integrity also is addressed to some extent by the provision on the indicator protocol to assign responsibility for data storage along with protocols and systems necessary to ensure that data remains sure and uncorrupted. However, additional issues, such as policies addressing data ethics within the organization, are still required to ensure that data integrity is addressed comprehensively.

Improving Data Quality along the Data-Management Chain

To help ensure data quality at the source:

Design simple-to-use instruments and correctly implement them

Include data providers (community stakeholders) and data processors in decisions about feasible review processes and drafting of instruments

Ensure all personnel are trained in their assigned task; use the same trainer for all trainings, if possible

Develop and document instructions for the data collectors and on the collection forms, as well as computer or gadget for data-collection procedures

Develop an appropriate sample

Ensure all data collection, entry, and analysis needs are available (e.g., pens, paper, forms, computers, phones, chargers)

When using technology, make sure to have simple scripts and forms that users will find easy to understand

To help ensure data quality during collection:

Develop specific instructions for data collection

Routinely check to see if instructions are being followed

Identify what to do if you (or someone) wants to make a change to the data collection process or if you have problems during data collection (change management process)

Check to see if people follow the change management process

Train data collectors on how to collect information

57USAID Performance Indicator Reference Sheet (PIRS): https://www.usaid.gov/sites/default/files/documents/1870/LocalSystemsFramework.pdf

10 tips for developing a good data management system

1. Keep it simple 2. Make the interface attractive 3. Remember only techies read the users’

guide! 4. Craft well-defined indicators 5. Consider user needs 6. Ensure compatibility of systems 7. Ensure quality of reports generated 8. Assign a dedicated database

administrator 9. Watch out for accessibility issues

10. Conduct Back up and maintenance




Develop standard operating procedures (SOPs) for managing the collected data (moving data from one point to the next)

Develop SOPs for revising the collection tool

Communicate the process and establish processes for supportive supervision of data collectors

Conduct on-site reviews during the data collection process

To help ensure data quality during collation:

Develop checklists and sign off on key steps

Conduct reviews during the data entry process

Create an electronic or manual format that includes a data verification process by a second individual who is not entering the data

Randomly sample data and verify

Ensure problems are reported, documented, corrected, communicated, and tracked back to the source of the problem

To help ensure data quality during analysis:

Run audit report for review by experts with knowledge for reasonableness

Ensure analysis techniques meet the requirements for proper use

Disclose all conditions/assumptions affecting data interpretations

Ensure that appropriate analysis that is management useful to program managers is undertaken routinely

Document changes made to the dataset, including creation of new variables and how they were calculated

To help ensure data quality during reporting:

Make sure you have all the data required to back up your report

Synthesize results for the appropriate audience

Maintain integrity in reporting: don’t leave out key information

Engage multiple report reviewers within the organization prior to dissemination

Protect confidentiality in reports/communication tools

Review data and provide feedback with stakeholders

Provide appropriate and easy-to-understand reports based on your audiences’ information needs

To help ensure data quality during use:

Consistently promote use of the data

When possible, provide data in terms of spatial or graphic representation

Routinely make available data at key decision points (e.g., staff meetings, program review)

Understand the data’s strengths and weaknesses and incorporate results appropriately

Build in processes to review how data has been used for decision-making over time

Take corrective action to enhance data use where necessary

To help ensure data quality during storage:

Take note of cumulative data and carefully track and store using a secure, accessible, and well-organized database

Remember to periodically back up incoming data, if possible in different locations; web-based storage systems are preferred

If using raw data, remember to keep it with metadata information for easy retrieval for future use



Hard copy or primary information sources need be well filed for future reference and possible DQAs

Stored data access needs to be available only to authorized team members as per the data management protocol; this ensures data remain safely secured

Consider implementing basic system maintenance tools like “restore” and “check data integrity” commands

Data Ethics

Data ethics refers to the rules or standards governing the conduct of a person collecting,

collating, reporting, or using data. Common ethical considerations that programs should make in relation to data include the following.

Ensure that program participants/beneficiaries are provided with sufficient information to enable them to make informed decisions about their participation in data collection efforts. No participant/beneficiary should be coerced to give information.

Participants need to be made aware of how much privacy/confidentiality/anonymity they can expect for their responses. This is especially important for peacebuilding programs where safety is of the utmost concern.

Programs should take steps to ensure that data is not misrepresented or falsified by anyone involved in the data-management process.


Accuracy and precision are important attributes to ensure data is of good quality.

The five commonly used criteria for assessing data quality are validity, reliability, timeliness, precision, and integrity.

Data-management processes involve collecting, collating, analyzing, reporting, using, and storing project activity information.

Indicator plans and protocols are the most important tools to start with when assessing data quality.

The team leader needs to ensure that indicators are clearly defined to ensure that everyone involved in data collection and reporting have the same and the correct understanding of all indicators and data collection methods.

A good data-management system should be simple with attractive interphase, user friendly, compatible, accessible, able to produce quality reports, and easy to maintain.

Data ethics must be observed by all involved with data along the data-management chain.

Learning Activity: Review key data quality issues and indicator plans

Divide participants into groups and ask them to discuss and identify key data quality issues likely to affect their indicator data reporting.

Review partner indicator plans to ensure they include measures for addressing any data quality issues moving forward and have strong indicator definitions.


Chapter 12: Mobile Technology for Data Collection | page 95

Chapter 12: Mobile Technology for Data Collection


About short message service (SMS)-based data collection versus general packet radio service (GPRS) data collection

The benefits of using mobile technology

How to develop a mobile technology plan

When to use and not use mobile technology

How to capture GPS coordinates

Learning Activities

Brainstorming on mobile technology for data collection

Evaluating experience with mobile data collection

Overview

Pact’s MERL Module 4: Mobile Technology Handbook58 covers in great detail the use of mobile technology for data collection. This chapter is an overview of the different types of mobile data collection platforms available and considerations to make when deciding how to use mobile technology in a peacebuilding project. Much of this chapter is drawn directly from MERL Module 4.

Mobile technology already has proven itself a powerful and efficient tool that accelerates

achievement of project objectives and, ultimately, of development goals. Efficiency and

data quality gains have been accepted as the norm for many applications, and the

frontier of possibilities expands daily.

What is information communication technology for

development?

Information communication technology for development (ICT4D) is the application of common, everyday technologies to the development field. This chapter focuses on mobile technology (specifically, cellphones and tablets) and how they are used around the world to accelerate development program goals. Projects from diverse technical areas have adopted mobile technology to perform:

Survey data collection

Routine monitoring

Dissemination of general or targeted information

Awareness and sensitization campaigns

Beneficiary/patient tracking

Surveillance and reporting

Crowdsourcing

58 Pact. 2013. MERL Module 4: Mobile Technology Handbook. Available at http://www.pactworld.org/blog/pact%E2%80%99s-mobile-technology-handbook-debuts

Logistics management information systems

Mobile banking and cash transfers

Coordinating service delivery or referral services

Conflict early warning systems

Public dialogue

http://www.pactworld.org/blog/pact%E2%80%99s-mobile-technology-handbook-debuts



Advantages of Mobile Data Technology

The advantages of using mobile data technology over traditional, paper-based methods include the following.

Faster, more accessible data: Data collected in the field with mobile devices can be submitted and stored online (in the cloud) or on central office servers, allowing for real-time access to data for analysis and decision-making.

More reliable data: Collecting data in the field with mobile devices eliminates the need to enter data from paper into a database, thereby reducing the opportunity for error or loss.

Cleaner data: Mobile applications embed skip logic that directs respondents to specific sections of the questionnaire based on previous responses (e.g., different questions for males and females). Thus, if skip patterns are correct, data are never entered where they don’t belong.

Ability to monitor data collection: By monitoring submitted survey GPS locations and time stamps you can track field worker performance and quality of incoming data.

Richer data: Integrated GPS coordinate collection, camera capabilities, and audio capture broaden the scope and type of information that can be collected.

Built-in data visualization: In some systems, online interfaces permit data visualization or have dashboards that allow data to be manipulated, customized, summarized, or downloaded.

More than just data: Mobile devices support a range of capabilities beyond data collection (e.g., mobile banking, patient tracking, crowdsourcing, communication).

Mobile data collection in conflict/post-conflict environments

When working in a fragile, conflict location, data collectors collecting data on paper have to either arrange for transportation of the data to a centralized location for aggregation and analysis or ask for data entry to be done then emailed to the central project location. Removing this step reduces risks for, e.g., data loss, longer data entry time, reduced safety of the personnel needed to transport the data.

The real-time nature of mobile data collection is beneficial for peacebuilding programs. Because these programs operate in environments with ever-changing contexts, having real-time data allows programs to make decisions and reorient strategy if necessary.

A MERL officer with the Pact PEACE III project touts the benefit of using mobile data collection. “We benefit a lot from the real-time nature of mobile data collection as this helps us to track partner activities in real-time and hence be able to make decisions faster. Also real-time incident reporting helps us quickly obtain information on trouble hotspots allowing us to make quick decisions. The mobile data platform also includes visualization feature that offers different views of the data such as maps and graphs allowing us to quickly get some insight of the data in real-time as this gets reported.”



SMS versus GPRS

Data can be transmitted in two main ways using mobile technology: SMS and GPRS.

About using SMS (basic cell phones)

SMS is a system for sending short messages of a fixed length, traditionally a maximum of 160 characters, in English, with other lengths in other languages. Select SMS for your data collection if:

You are using basic phones (e.g., if your initiative is using the personal phones of data collectors who do not own more technologically advanced mobile devices, specifically Java enabled feature phones or smartphones

Data submissions are typically 160 characters or fewer in length

Total data volume is low

Data collectors lack internet access

Your project is a small one-off initiative that will not be scaled up

Data collectors need to send their messages at no cost, that is, if you need the ability to reverse-bill SMS charges

About using GPRS (internet)

GPRS is a data-transmission system similar to SMS, but without limits on the number of characters or transmission size. This is more commonly known simply as “the internet” on phones. Select GPRS for your data collection if:

Data collectors are equipped with more technologically advanced phones (e.g., Java-enabled feature phones or smartphones)

You need to submit large volumes of data at one time

Total data volume is high

Multimedia data (e.g., photos, videos) will be transmitted

Internet access on the phone or via intermittent connection to Wi-Fi is not an issue for data collectors

Example: Using SMS in a peacebuilding project

Setting up an SMS-short code can be an effective way of receiving conflict monitoring or other such data best gathered from community crowdsourcing. If community members are made aware of the code, they may text the number about incidents they see, often at no expense to them. There are ways you can ensure that your project pays for the SMS messages that are sent in, rather than making the respondents pay for that cost out of their own phone credits.

A word of caution: If you work on a project that crosses country borders, it may be difficult to set up an SMS system. This is because you would have to set up short codes with various service providers in the different countries and would not receive the data in one central location. In addition, using SMS is only appropriate when you have few data items to collect.

Taking GPS Locations

The GPS receiver on a mobile phone can collect location data even when there is no mobile data connection. However, getting an accurate GPS location requires clear view of the sky and good satellite signal reception. Avoid collecting GPS locations when inside buildings or when standing next to large obstructions. If you do not have a clear view of the sky, your location will not be recorded accurately.



Figure 12.1: Screenshot of PEACE III event form on iFormBuilder (at right)

Developing Your Mobile Technology

Strategy and Why it is Important

Developing a strategy for incorporating mobile technology into your work processes is critical. The strategy will ensure that the human, material, and financial resources you will need are available; that your technology is appropriate for the project and its context; and that mobile technology will enhance program management or implementation.

Example: Using GPRS (internet) in a peacebuilding project

Pact’s PEACE III project collects data via the mobile platform iFormBuilder (see Figure 12.1). The project uses mobile data for event and conflict incident reporting. Using mobile technology for data collection has increased the speed in which data is received from each event, removed the burdensome step of data entry, allowed for collecting cleaner data through complex validation controls, and enabled the project to collect richer data, including GPS coordinates and images. The GPS coordinates have enabled the project to look at trends in the data from different perspectives using geographical information systems (GISs).

Many GPRS mobile platforms, such as iFormBuilder and others, provide an offline data collection mode. This means that data collectors in remote areas with no internet or Global System for Mobile Communications (GSM) connectivity can collect data, store it in the phones’ memory, then upload/“send” the data later when they get to an area with Wi-Fi or GPRS. This is especially important for areas without any or with poor mobile service coverage.

A word of caution: Sometimes you may need to change platforms mid-project if it does not suit your needs. For example, the type of form you use or data you collect may be better suited for one mobile platform over another. Free trial periods (before purchasing software) to test forms are ideal to allow you to extensively test all the software features before purchasing. Also, while most mobile platforms allow offline data collection, it is crucial that users are well trained on how to go about this to minimize data loss.

In hostile places, using a mobile phone could endanger the life of the data collector. For example, using the mobile platform to take pictures in events where military and/or militia members attend might accelerate tensions because data collectors may be viewed as spies or worse. Extreme caution must be taken in these circumstances. In instances where security may be a concern, use discretion while collecting data in unsafe locations and delay collecting data if the situation is too insecure.

Mobile phones also have their limitations. Smartphones regularly run out of power, and storage space is limited. For week-long activities in areas where there are few charging sockets, it is important to plan ahead and have backup power (e.g., extra batteries, solar charging stations) or to switch off the phone when it is not in use to conserve power. In the early stages of the program, PEACE III experienced this challenge, when a data collector gave the feedback that “the phone went off on day one.” Another partner was unable to use the mobile app because the phone’s internal memory was full, so the app could not launch.



When developing a strategy, it is important to involve all stakeholders, including those who will collect the data, use or analyze data, and manage the process. Chapter 2 of Pact’s MERL Module 4 discusses how to get started and what questions you should be asking stakeholders as you develop the strategic plan.

Use the following steps to help you begin to develop your organization or project’s mobile technology strategy.

1. Brainstorm the areas where mobile technology can improve your project, keeping in mind the sensitivity of the context and logistical limitations in which it operates.

2. Identify the data you would like to collect. 3. Identify and understand your mobile technology users. 4. Consider data-management processes. 5. Conduct a mobile technology feasibility scan. 6. Decide on the appropriate mobile technology platform. 7. Develop your mobile technology strategy document.

1. Brainstorm how mobile data collection can improve your project.

It is important to consider first how mobile technology will be useful for your project’s MERL system. Think about some problems you have and how mobile technology can solve them. For example:

Slow data processing: “Whenever we conduct surveys, data clerks are tasked with entering all the data. Given that this is in addition to their other work, it takes up to four months to obtain a completed dataset. This delays our ability to respond and make databased programming decisions.”

Poor data quality: “Too many times, our data collectors don’t follow skip patterns or questionnaire instructions. As a result, our data is poor.”

Lack of geographic data: “USAID has asked us to map services delivered to beneficiaries. We need to begin to collect GPS data.”

Learning Activity: Brainstorming on mobile technology for data collection

Referencing the brainstorming examples below, what type of organization or project data could be most efficiently collected using mobile technology? How might you integrate mobile technology into your current data collection activities?

Pact Myanmar Brainstorming

Goal: Pact Myanmar wanted to begin using mobile technologies to monitor and track women’s savings groups outcomes.

Problem: Every six months, data are collected from women’s savings groups on savings, loans, and dividends—a huge endeavor, given that there are more than 2,000 women’s savings groups in Myanmar.

Solution: Mobile technology made data collection much more efficient.

PEACE III Brainstorming

Goal: PEACE III partners with local implementers working in the cross-border locations to bring peace in these neighboring communities through trauma healing, cultural practices adaptation, and peace dividends.

Problem: To achieve its objectives, PEACE III needed to track partner activities and conflict incidences in real time. The program also wanted to collect GPS coordinates to enable mapping of partner activities and reported incidences.

Solution: Mobile technology enabled collection and reporting of real-time data and GPS coordinates.

(Learning activity continued on next page)



Goal:

Problem:

Solution:

2. Identify your data.

Based on your brainstorming, think about the data you would like to collect.

Indicators: If you are starting a new project, consider which indicators make the most sense to collect using mobile technology.

Tools: If your project is already underway, think about which data collection tool(s) could most appropriately be adapted for mobile data collection.

New information: If you could access data for analysis much more quickly after collection, as is possible with mobile technology, consider whether there are data that would be useful to have that you may not otherwise have collected (e.g., conflict data).

New data types: Remember that with mobile technology you can collect not only text and numerical information, but also photos, videos, GPS coordinates, audio, and biometric data.

Remember: Mobile technology should simplify your life. For example, think of the large volume of data that are waiting to be entered or data that people have to travel far to submit paper survey forms.

3. Identify and understand users.

As part of the process of deciding whether mobile technology will enhance your data collection and management processes, you need to identify and understand the end users of your mobile technology. Knowing your users will help you:

Obtain the right type and number of mobile devices and appropriate software: How many mobile devices are needed? What language will the software need to use? Under what conditions will the devices be used (e.g., access to power sources, rain or dust that might damage the devices)? What level of technology is appropriate for the end users? Also, consider their ages, experience with phones, literacy, and eyesight.

Tailor training to the users: The most effective training content and delivery is responsive to end users’ level of education, culture, and previous exposure to similar technology.

Get buy-in from the people who will be using the technology: Securing end users’ commitment to mobile technology during the design phase is the best way to



ensure the success of your rollout. Exposing end users to the technology early on will test any assumptions you have made about the appropriateness of mobile technology. Early user involvement also has the potential to greatly improve the level of participation and, ultimately, the speed and quality of implementation. It also is highly important to ensure gender equity when introducing technology. Obtaining buy-in from both male and female users is crucial. Additionally, sensitization to the technology may require different techniques with men and women, depending on existing usage levels.

4. Consider data-management processes.

Once you identify the relevant questions, data collection tools, and data types for your mobile initiative, consider your data management processes to determine which platform will be the most cost effective and best suited to your strategy for incorporating mobile technology into your work.

Frequency: How frequently will the data be collected (e.g., every month)?

Data collectors: How many people are collecting data and how many will require mobile devices (e.g., 10 field workers, all need mobile devices)?

The tool: How many items/fields are on the survey/data collection tool (e.g., survey is 15 pages long with 65 items)?

5. Conduct a mobile technology feasibility scan.

After you have a general idea of who will be using mobile data technology in your organization and how, you will want to conduct an initial feasibility scan to help you understand the mobile infrastructure and the cultural, political, and economic environment. This scan will help you ensure that your strategy is appropriate and realistic. Ask the following questions.

How widespread is network coverage? Which mobile service provider(s) make sense to use?

If coverage is poor in the targeted area, are Wi-Fi hotspots nearby, or is better coverage available elsewhere?

What data plans are available, and how much do they cost?

Are people able to keep their phones charged across the project area?

What are the data security and management issues to be aware of? Is privacy a consideration?

What are the legal and regulatory frameworks that might cause concern?

How will the use of mobile technology impact gender equity, ethnic relations, and access to power and resources? Will the use of mobile technology benefit some groups more than others?

Will using mobile technology bring unwanted attention to respondents or field workers?

Will privacy concerns or other factors bias the data in ways that a traditional approach would not?

6. Decide on the appropriate mobile technology platform.

After defining your mobile strategy objectives, identifying end users, and conducting a feasibility scan, the next step is deciding which mobile device and platform are most appropriate for your needs. Mobile platforms, the systems that receive data from mobile devices, provide a variety of features and functions, including data storage, verification, analysis, and reporting. See more in Chapter 3 of MERL Module 4.

With so many different mobile devices on the market, each with different plans and features, choosing a mobile device can be overwhelming. To quickly narrow your options, consider the



nature of your mobile initiative (one-off versus long-term), data requirements, method of transmission, power sources, and budget. Table 12.1 outlines the types of mobile devices you could purchase and the functionality of each.

Table 12.1: Mobile devices from basic to advanced

Basic Phone Feature Phone

Smartphone (Android) Tablets

Description Inexpensive cellphones with voice and SMS capabilities

Mid-range cell phones

Can access internet

Some have GPS capability and a camera

Advanced cell phones

Can access the internet

Most have GPS and a camera

Low-cost tablets

Come with Wi-Fi and may come with 2G/3G/4G

Pros Ubiquitous, cheap, known

Low power use

Variety of features available

Can transmit GPRS data

Moderate power use

Software can be updated and upgraded

Can use any language

Larger screens make for good visibility and ease of entering data

Low cost of some models

Cons Moving parts of flip phones and slider phones affect reliability and durability

Creating texts on basic keyboard can be challenging

Not all have GPS

Language compatibility issues

Small screen

Heavy power user, needs recharging frequently

Might be targeted by thieves

Heavy power user, needs recharging frequently

Might be targeted by thieves

Some are more expensive than other mobile devices

Manufacturers Nokia Nokia, Siemens, Sony Ericsson (Java phones)

Samsung, LG, ZTE, Vodaphone, HTC, Apple

Aakash, Ubistlate, Google Nexus, Samsung, Apple

A number of mobile platforms offer a broad range of features and different pricing plans; several open-source options come at no cost.

To determine which platform is right for your mobile initiative, consider your needs and overall project budget. To a certain extent, your project’s data needs and the frequency and type of data collection outlined in your mobile strategy will help you narrow your options.

Please keep in mind that platforms change rapidly and that features and price structures evolve over time; the latest information is always available on the platform’s website. When deciding among mobile platforms, consider phone requirements; the data entry interface and transmission methods permitted; data storage, analysis, and reporting features; miscellaneous features; and the pricing structure.

Table 12.2 shows the most commonly used mobile platforms on the market today; however, new options open daily. For additional details on the different mobile platforms, please see Chapter 3 of MERL Module 4.



Table 12.2: Commonly used mobile platforms

Features Mobenzi Commcare Magpi DataWinners Frontline SMS iFormBuilder

Language Does not work in all alphabets yet

Not alphabet restricted





SMS No Yes Yes, at pro level Yes, costs more Yes No

Forms Yes Yes Yes Yes Yes, in old version Yes

Type of phones for forms

Java, Android, Blackberry, Windows

Java, Android Java, Android, Blackberry, Windows

Android Android and Apple

Web portal entry No Yes Yes Yes No Yes

Skip logic Yes, and repeat logic If you can code Yes No No Yes

Ease of survey setup interface

Low Low High Medium-high High High

Other features Images, GIS Audio, images, GIS GIS GIS -- Audio, Images, GIS and ESRI connector, attach files from phone memory, API, RFID

Survey Setup Support

Survey setup support Do it yourself (DIY; complicated) or set-up support (expensive)

DIY DIY DIY DIY

Price Price models differ, so which is more cost effective will depend on the size of the surveys, how frequently they will be conducted, etc.

To learn more Site: www.mobenziresearcher.com

Login: www.researchconsole.com

Phone list: http://www.mobenzi.com/researcher/Help/handset-support

Site: http://www.commcar

ehq.org/home/

Phone list: https://wiki.commcarehq.org/pages/viewpage.action?pageId=4718838

Site: https://www.magpi.com/login/auth

Site: https://www.datawinners.com/en/home/

Site: www.frontlinesms.org

Device list: http://www.frontlinesms.com/the-software/requirements/

Site: https://www.iformbuilder.com/

http://www.mobenziresearcher.com/

http://www.mobenziresearcher.com/

http://www.researchconsole.com/

http://www.researchconsole.com/

http://www.mobenzi.com/researcher/Help/handset-support



http://www.commcarehq.org/home/

http://www.commcarehq.org/home/

https://wiki.commcarehq.org/pages/viewpage.action?pageId=4718838




https://www.magpi.com/login/auth

https://www.magpi.com/login/auth

https://www.datawinners.com/en/home/

https://www.datawinners.com/en/home/

http://www.frontlinesms.org/

http://www.frontlinesms.com/the-software/requirements/




https://www.iformbuilder.com/

https://www.iformbuilder.com/



7. Develop your mobile technology strategy document.

The aforementioned examples highlight the need for projects to document procedures and

policies. A well-crafted mobile data collection strategy document should describe:

Purpose of the strategy document, including the challenge that mobile technology use will help you to solve

Goals and objectives of the mobile technology strategy

Data to be collected via mobile devices

Your planned mobile technology feasibility scan, including: - Available network coverage in the areas where you will be working - Comfort and experience level of mobile device users - Data plans and costs of data plans - Costs of mobile devices - Security issues and management - Internal and external technical support required

Tools and instruments selected and rationale behind selection

General data collection procedures

Reports/dashboards that will be generated

Integration of data into other systems

Data management and security plan

When You Should Continue Collecting Data on Paper

In some cases, it may not make sense to use mobile technology, such as follows.

Status quo satisfactory: When your data is not urgent and your current system is already meeting your standards for quality and efficiency, changing data collection methods may not make sense.

Security concerns: If stakeholders have concerns about the security of data leaving the country to an international data host, collecting data on paper may continue to make sense. The cost of having separate servers host data in country can be prohibitive for small data collection exercises.

For qualitative data collection: When you are collecting a lot of qualitative data, using mobile devices for numeric or even QWERTY-keypad input will limit the amount of information that can be entered quickly. If audio capture is possible, it may mitigate this issue if the survey is limited in size. Otherwise, paper data collection is most flexible.

Use of mobile devices impractical or unsafe: In some areas, using mobiles may make data collectors more vulnerable to violence or surveillance. Elsewhere, entering data on a mobile device as an interview is in process may be distracting for the respondent or even potentially bias responses, such as when the respondent is concerned for his or her privacy or security. Taking photos or recording video with phones can exacerbate these concerns.

It is important to consider the type of data you are collecting and who will be collecting it to make sure that the use of mobile technology is appropriate for the context.



Develop a Budget

There are costs associated with mobile data technologies. The most cost-effective platform depends on the amount of data you collect, number of users, and number of survey items.

Costs of mobile versus paper data collection should be considered when deciding on the mode of data collection you will use. Table 12.3 outlines the considerations for both types of data collection mode.

Table 12.3: Data collection comparison: mobile technology vs. traditional paper

Cost of mobile data collection Cost of paper-based data collection

Number of mobiles (e.g., total number of data collectors plus backups or replacement mobiles)

Cost of submitting data (e.g., per SMS text message, or, if over the internet, per gigabyte)

Charging devices

Survey/database design

Technology support for survey setup (if needed)

Training of data collectors

Mobile platform (e.g., database, data verification, data analysis and reporting functionality)

Human resources (e.g., to manage mobile provision and data collection systems and processes)

Costs for printing, pens, and paper and for transportation for a limited number of paper backup survey materials

Printing costs

Pens and paper

Transportation of paper surveys

Training of data collectors

Human resources for, e.g., data entry, transportation, supervision of data collectors

Database design, data entry, and data verification

Storage of paper surveys (particularly if dealing with sensitive information)

Costs for each platform varies: some charge an annual fee for unlimited use (iFormBuilder), while others charge per data field collected (per survey question, for example—Mobenzi’s model). Of note, there are plenty more mobile software options than only Mobenzi and iFormBuilder, and projects are encouraged to investigate and produce cost estimates of a variety of platforms before making their decision.

Costs related to powering mobiles may need to be budgeted as well. Some options to ensure reliable power include:

Chargers that plug into car cigarette lighter sockets

External battery sources

Spare internal batteries

Solar chargers




Use of mobile technology has proven to be a powerful and efficient tool for collecting data that can accelerate achievement of project objectives and, ultimately, of development goals.

Projects from diverse technical areas successfully have adopted mobile technology to perform survey data collection, routine M&E, patient tracking, crowdsourcing, and conflict early warning systems, among other data collection needs.

It is important to consider the type of data you are collecting and who will be collecting it to make sure that the use of mobile technology is appropriate for your context. Remember that mobile technology only should be used if it helps to make your data collection and reporting processes more efficient.

When planning to use mobile technology, it’s important to consider: - Your specific data needs and data management processes in order to decide which

platform will be the most appropriate - Identify and understand the users of your data, including to inform type of device - Conduct an initial feasibility scan to help you understand the mobile infrastructure

and the cultural, political, and economic environment - Develop a budget, train users, and pilot the technology before wide-scale roll out

Learning Activity: Evaluating experience with mobile data collection

For programs already using mobile data collection: In small groups, discuss your experience with mobile data collection over the past few months to one year. What is working well, and what is not working well? Recommend actions to address the aspects that are not working well in close consultation with your M&E and information technology teams.

Discuss what other opportunities there may be for your organization to use mobile technology beyond existing use. What strategy will you use to expand to these other projects/areas?


Appendix 1: Success Stories Guide | page 107

Appendix 1: Success Stories Guide

Partners should report stories that either:

1. Show broad-scale sustainable peace activities/projects, i.e., no one-offs and nothing that is not being implemented at scale; the writer may focus on an individual beneficiary to highlight their work, but the greater project must impact more than an individual or small group of people

2. Relate to the peacebuilding project they work on

3. Highlight new, cutting-edge development innovations

4. Highlight any contributions to funder priorities, for example Value for Money (DFID) or localization of foreign aid (USAID).

For more information on the USAID Reform Agenda, visit http://forward.usaid.gov or other funder priorities. For the DFID Value for Money policy, visit https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67479/DFID-approach-value-money.pdf.

All stories should point to results!

How to Write Stories

Use a compelling title. Include an intriguing lead or hook, beneficiary quotes, and powerful statistics to communicate progress.

Include a character/subject that illustrates how the program has improved people’s lives or made a difference in the community.

Consider specific photographs that “communicate” to back up the story. Make sure to caption and attribute the photograph(s).

Keep the story short and concise: 350–500 words.

Consider your target audience. Different audiences will have different points of view and needs and will respond to different ways of sharing the story.

Focus on qualitative information, but with sufficient supportive quantitative information.

Have enough details to help unpack technical information for non-technical readers.

Avoid abbreviations/acronyms and jargon.

Use quotes from beneficiaries. Do not quote activity staff.

Photographs

The process of taking and quality of a photograph can make or break a story. The following should be considered when using a photograph in a success story.

Digital photos should be shot with at least a 3-megapixel resolution and, when possible, maintain at least 300 dpi (dots per inch).

Send only graphics files (preferably JPEG/.jpg). Do not attach an MS Word document with the photo pasted into it.

Do not alter, compress, or crop photographs. Send only original images.

Do not scan images from publications or other printed materials.

Generally speaking, the larger the file, the better the quality and final result.

http://forward.usaid.gov/




Appendix 1: Success Stories Guide | page 108

Include a caption that briefly summarizes what is occurring in the photograph: who, what, when, and where.

The photo should be colorful, depict action, and feature the main story character.

Ask permission before taking a photo of someone. Check funder, country, and your organization’s requirements for taking and using photographs of human subjects.

Play with different angles and backgrounds instead of capturing the subject straight on. The most interesting and visually pleasing photographs are often those with that capture the subject at unusual angles.

Use available light instead of flash whenever possible.

Model Stories

Before sitting down to write, consider reviewing the model stories you have come across.59 This will give you a sense of what makes for a successful submission.

The Plain Writing Act of 2010 (http://www.plainlanguage.gov/index.cfm) requires the United States government and its contractors to write all new publications, forms, and publicly distributed documents in a “clear, concise, well-organized” manner that follows the best practices of plain language writing. The site also includes some great writing pointers!

59 Some examples of success stories can be found on Pact’s website: http://www.pactworld.org/features

Example of story from peacebuilding program in South Sudan: http://www.pactworld.org/blog/south-sudanese-displaced-war-peacemaker-always-ready-help

http://www.plainlanguage.gov/index.cfm

http://www.pactworld.org/features

http://www.pactworld.org/blog/south-sudanese-displaced-war-peacemaker-always-ready-help

http://www.pactworld.org/blog/south-sudanese-displaced-war-peacemaker-always-ready-help


Appendix 2: MERL Plan Template | page 109

Appendix 2: MERL Plan Template

Place organization (and funder, if required) logo here

Name of Organization/Project

Monitoring, Evaluation, Reporting, and Learning Plan

Add period covered by the plan

Date: (Insert date finalized/updated)



Contents

Abbreviations and Acronyms………………………………………………………………………………………….. 1 1. Introduction……………………………………………………………………………………………………………… 2 1.1. Mission and Vision……………………………………………………………………………………….. 2 1.2. Purpose of the MERL Plan……………………………………………………………………………. 2 1.3. Overview of Programs…………………………………………………………………………………… 2 1.4. MERL and the Project Cycle………………………………………………………………………….. 3 1.5. MERL Team………………………………………………………………………………………………… 3 1.6. Audience Analysis………………………………………………………………………………………… 4 2. Conceptual Framework……………………………………………………………………………………………….5 2.1. Organizational/Project Theory of Change………………………………………………………..6 3. Indicator Definitions, Data Collection, and Reporting Plan…………………………………………….7 4. Reporting…………………………………………………………………………………………………………………. 8 5. Data Quality and Data Verification Procedures…………………………………………………………….. 9 6. Monitoring Tools………………………………………………………………………………………………………10 7. Deliverables Schedule……………………………………………………………………………………………….. 11 8. Evaluation Plan……………………………………………………………………………………………………….. 12 Annex 1. Work Plan……………………………………………………………………………………………………… 13 Annex 2. Indicator Targets…………………………………………………………………………………………… 14



Abbreviations and Acronyms

Add a list of the abbreviations and acronyms used in your MERL plan in alphabetical order

and defined as they are used in the text. Some standard abbreviations are included below.

CBO community-based organization

CSO civil society organization

MERL monitoring, evaluation, reporting, and learning

NGO nongovernmental organization

TOC Theory of Change

US$ United States dollar

page 1



1. Introduction

1.1. Mission and Vision

State your vision and mission here. Explain each as needed for better understanding by users of the MERL plan and to show how these inform the goal of this MERL framework.

1.2. Purpose of the MERL Plan

State the purpose of having this MERL plan for your organization/project (e.g., to outline the process for carrying out effective MERL in the organization/project, to inform decisions on achievement of organizational results).

1.3. Overview of Programs

Provide an overview of all the programs/projects the organization is currently implementing and for the one(s) for which you intend to apply this MERL plan.

page 2



1.4. MERL and the Project Cycle

Edit as needed the following matrix to demonstrate how your MERL system aligns with the project cycle and with the award “prime” organization, if you are an implementing partner. Fill in each cell with information relevant to your program.

The project cycle Key activities to integrate the MERL system into the project cycle

Clarify the organization’s vision and mission Clarify the goals of the MERL system

Select policy issues of interest Select policy issues of interest

Identify stakeholders Identify MERL audiences

Identify who needs information and data from the organization, why, and when

Conduct situation analysis Set up participatory monitoring and evaluation framework (with input from the community and other key audiences)

Develop peacebuilding approach/Theory of Change (TOC)/conceptual strategy

Establish peacebuilding approach

Develop TOC, goals, and objectives

Clarify conceptual model and identify assumptions

Identify what to monitor and evaluate Develop project indicators

Implement activities Monitor progress: Collect data, report on progress

Analyze results and evaluate project Design and implement project reviews/periodic evaluations

Use results to adapt and learn Report on project results/review project objectives/strategies/approaches

1.5. MERL Team

Identify members of your MERL team and explain their roles and responsibilities. Use position titles and not individuals names. Be sure to include all staff and volunteers that have responsibilities related to MERL, not just MERL and Program officers.

page 3



1.6. Audience Analysis

Use this tool to outline who your audiences/stakeholders are and what data they need. This exercise can help your organization see if there are audiences you are currently missing and help you to avoid collecting data that you do not need. Don’t forget beneficiaries and volunteers!

Audience WHAT data is required WHY the data is required

WHEN the data is required

How this audience can participate in the MERL program/system

External audience

Internal audience

page 4



2. Conceptual Framework

Use the either the results framework or logical framework to demonstrate your expected project results. These frameworks show the logical connections between what your organization/project hopes to achieve and the activities it conducts. It gives the reader of the plan an overview of why you do what you do.

Project Results Inputs Activities Outputs Outcome Impact

R1.1

R1.2

R1.3

page 5



2.1. Organizational/Project Theory of Change

Briefly explain the Theory(ies) of Change and key assumptions on which your conceptual framework is based. In other words, tell the reader how you got to the point of showing the relationships between the inputs, outputs, outcomes, and impact results.

page 6



3. Indicator Definitions, Data Collection, and Reporting Plan

Use the following template to describe each of the indicators your organization/project will report on.

Program/ project result

Indicator

(show reference number and indicator name as they appear in your targets table)

Indicator definition and disaggregates

(show definition, disaggregation, and unit of measure as they appear in your targets table)

Data source (name of data collection tools)

Frequency of data collection

Responsi-bility for data collection

Frequency of data analysis and reporting

EXAMPLE

Goal: Strengthen the horizontal and vertical linkages within and between local, national, and regional conflict management actors within the Horn of Africa

R1.1

R1.2

R1.3

page 7



4. Reporting

Describe the types of reports the organization/project is required to prepare periodically for your MERL audiences, such as weekly updates, monthly reports, quarterly reports, and annual reports. You can use the following template to develop a reporting schedule to show when each type of report is due, who is responsible for preparing it, and to whom the reports are submitted. Annex all relevant reporting templates to this plan (e.g., those from your current funders).

Audience Communication tool selected for reporting Schedule for reporting

Responsibility for preparing report

To whom is the report submitted

EXAMPLE

Funder: Pact

Quarterly report April 5, July 5, October 5, January 5

Program manager Pact regional manager and MERL officer

Soft copy of event and incident reports

Monthly by 5th of the following month

Program officer Pact regional manager and MERL officer

Financial reports Monthly by 5th of the following month

Finance manager Pact grants manager

Final report 30 days after end of grant Program manager Pact regional manager, grants manager, and MERL officer

Final financial report and closeout audit

30 days after end of grant Finance manager Pact grants manager

page 8



5. Data Quality and Data Verification Procedures

Insert a graphic showing how data flows in your organization from the field level to the national level using the data sources or data collection tools. The graphic should show the points at which data verification is done before it is forwarded to the next level.

After drawing the graphic, describe the procedures your organization is carrying out to ensure data is of good quality at each of the following data management stages: source, collection, collation, analysis, reporting, use, and storage. If no verification and data quality management procedures have been specified before, use this plan to develop some that will guide the organization/project going forward.

page 9



6. Monitoring Tools

Include a matrix of the monitoring tools that you will use to track the progress of your

organization/project. Remember that these are not the tools for collecting data, but are

related to monitoring the organizational and operational aspects of program/project rollout.

These tools include things like conflict/context monitoring tools, work plan implementation

monitoring, and assumptions monitoring. It is important to state in your matrix how often the

information captured in the tools is collected and/or will be updated. Remember that these

data collection processes must inform operational issues, and thus must be updated as often

as the information would be useful for management. Do not waste time, money, and effort

using tools that have no operational impact when their content is reviewed! Include blank

examples of your monitoring tools as an annex to this plan.

Operational area of concern

Name of monitoring tool

Data collection frequency Update frequency

Conflict monitoring

Project implementation and expenditure

Assumptions monitoring

page 10



7. Deliverables Schedule

Refer to your award document and identify all the contractual deliverables that are due to your funder(s). Then, complete the following matrix. Update it quarterly, especially to record the actual delivery dates.

Funder staff who will receive the deliverable Deliverable(s) Due date(s)

Actual date of delivery

Means of delivery

EXAMPLE

Grants manager

Monthly financial report

Quarterly progress report

Event reports

The 5th day of the month following the event or the close of the month or quarter

November 5, 2015 (confirmed by email and currier receipts)

Emailed and hand-delivered hard copy sent to funder

page 11



page 12

8. Evaluation Plan

Include a basic evaluation plan that enables you to evaluate why you have or have not achieved the goal/results that were set. This plan allows you to look at consequences (intended or unintended), effectiveness, efficiency, outcomes, and sustainability of initiatives. Remember that evaluation looks at the overall program/project, the operations, governance, and deliverables! Basically, it helps you identify the lessons learned and what you would do better next time. Use a simple tool such as the table below to help you evaluate your overall program/project.

Type of evaluation What do we need to evaluate?

What questions do we need to ask?

How will we obtain the data?

When will we get the data? Who will do this?

EXAMPLE

Outcome evaluation

Progress made in reducing violence in conflict areas

Have people changed their behaviors, the way they interact with other communities, and/or their underlying attitudes?

One-to-one interviews with key informants

Survey questionnaires

Semi-annually and annually

MERL officer

Program staff



Annex 1. Work Plan

Attach a copy of the organization or project work plan here.

page 13



Annex 2. Indicator Targets

Attach a copy of your final approved indicator targets here.

page 14


Appendix 3: References | page 125

Appendix 3: References

Argyris, C., R. Putnam, & D. McLain Smith, D. 1985. Action Science: Concepts, methods, and skills for research and intervention. San Francisco: Jossey-Bass.

CDA Collaborative Learning Projects. 2013. Reflecting on Peace Practice: Participant Training Manual. Available at http://cdacollaborative.org/publication/reflecting-on-peace-practice-i-participant-training-manual/.

Channel Research. 2008. Evaluation of Conflict Prevention and Peace Building. Handout for INCORE University of Ulster Summer School.

Church, C., & M. Rogers. 2006. Designing for Results: Integrating Monitoring and Evaluation in Conflict Transformation Programs. Washington, D.C.: Search for Common Ground.

Church, C., & J. Shouldice. 2002. The Evaluation of Conflict Resolution Interventions: Framing the State of Play. Londonderry: Incore.

---. 2003. The Evaluation of Conflict Resolution Interventions. Part II: Emerging Practice & Theory. Londonderry: Incore.

Conflict Sensitivity Consortium. 2004. Conflict-sensitive approaches to development, humanitarian assistance and peacebuilding. Available at ttp://www.saferworld.org.uk/resources/view-resource/148-conflict-sensitive-approaches-to-development-humanitarian-assistance-and-peacebuilding.

Dziedzic, M., B. Sotirin, & J. Agoglia (Eds.). 2008. Measuring Progress in Conflict Environments (MPICE) - A Metrics Framework for Assessing Conflict Transformation and Stabilization. Washington, DC: United States Institute for Peace. Available at http://www.usip.org/files/resources/MPICE%20Aug%2008.pdf.

Fitzpatrick, J. L., J. R. Sanders, & B. R. Worthen. 2011. Program evaluation: Alternative approaches and practical guidelines (4th ed.). Upper Saddle River, NJ: Pearson Education, Inc.

Herrington, R. 2015. Emerging Practices in Design, Monitoring, and Evaluation for Education for Peacebuilding Programming (1st ed.). Washington DC: Search for Common Ground. Available at http://www.dmeforpeace.org/educateforpeace/wp-content/uploads/2015/11/EP-guide-_-Final-B-1027.pdf.

Igbo, A. 2013, Sept. 3. Ten Tips for Developing a Data Management System for Monitoring and Evaluation (M&E). The Communication Initiative Network. Available at http://www.comminit.com/job_vacancies/content/10-tips-developing-data-management-system-monitoring-and-evaluation-me.

International Bank for Reconstruction and Development & World Bank. 2012. Designing A Results Framework for Achieving Results: A How-To Guide. Washington DC: Independent Evaluation Group. Available at http://siteresources.worldbank.org/EXTEVACAPDEV/Resources/designing_results_framework.pdf.

Jaszczolt, K., T. Potkański, & S. Alwasiak. 2003. Internal Project M&E System and Development of Evaluation Capacity: Experience of the World Bank-funded Rural Development Program.

http://cdacollaborative.org/publication/reflecting-on-peace-practice-i-participant-training-manual/

http://cdacollaborative.org/publication/reflecting-on-peace-practice-i-participant-training-manual/

http://www.usip.org/files/resources/MPICE%20Aug%2008.pdf

http://www.dmeforpeace.org/educateforpeace/wp-content/uploads/2015/11/EP-guide-_-Final-B-1027.pdf

http://www.dmeforpeace.org/educateforpeace/wp-content/uploads/2015/11/EP-guide-_-Final-B-1027.pdf

http://www.comminit.com/job_vacancies/content/10-tips-developing-data-management-system-monitoring-and-evaluation-me

http://www.comminit.com/job_vacancies/content/10-tips-developing-data-management-system-monitoring-and-evaluation-me

http://siteresources.worldbank.org/EXTEVACAPDEV/Resources/designing_results_framework.pdf

http://siteresources.worldbank.org/EXTEVACAPDEV/Resources/designing_results_framework.pdf



Kusek, J.Z., & R.C. Rist. 2004. A Hand Book for Development Practitioners, Ten Steps to Results Based Monitoring and Evaluation System. Washington, DC: World Bank.

Lederach, J.P., et al. 2007. Reflective Peacebuilding. Mindanao, Philippines: John B. Kroc Institute for International Studies.

Norad, DAC Network on Development Evaluation International Network on Conflict and Fragility, & OECD. n.d. Chair’s Summary: Evaluating conflict prevention and peacebuilding activities: Lessons from the application phase. Available at http://www.oecd.org/dataoecd/8/48/48092571.pdf.

OECD. 2002. Glossary of Key Terms in Evaluation and Results Based Management. Available at http://www.oecd.org/dataoecd/29/21/2754804.pdf.

OECD Development Assistance Committee. n.d. DAC Criteria for Evaluating Development Assistance. Available at http://www.oecd.org/dataoecd/42/6/49756382.pdf.

---. 2008. Guidance on Evaluating Conflict Prevention and Peacebuilding Activities: Working Draft for Application Period. Paris: OECD. Available at: https://www.oecd.org/dac/evaluation/dcdndep/39774573.pdf.

Patton, M.Q. 1997. Utilization-focused evaluation: The new century text (3rd ed.). Thousand Oaks: SAGE Publications, Inc.

Pact. 2008. Building Monitoring and Evaluation Systems in Civil Society Advocacy Organizations (3rd ed.). MERL Module 1. Washington, DC: Pact.

---. 2013. Mobile Technology Handbook. MERL Module 4. Available at http://www.pactworld.org/blog/pact%E2%80%99s-mobile-technology-handbook-debuts.

---. 2014. Field Guide for Data Quality Management. MERL Module 2. Available at

http://www.pactworld.org/sites/default/files/DQM%20Manual_FINAL_November%202014.pdf.

---. 2014. Field Guide for Evaluation. MERL Module 3. Available at http://www.pactworld.org/documents/field-guide-evaluation.

PEACE III Program technical proposal to USAID

Sartorius, R. & Carver, C. n.d. Monitoring, Evaluation and Learning for Fragile States and Peacebuilding Programs. Arlington, VA: Social Impact.

Segone, M. (Ed.). 2009. Country-Led Monitoring and Evaluation Systems: Better evidence, better policies, better development results. Available at http://mymande.org/sites/default/files/images/Country_ledMEsystems.pdf.

Scriven, M. 1991. Evaluation thesaurus. 4th ed. Newbury Park, CA: SAGE Publications, Inc.

Stave, S. E. 2011. Measuring peacebuilding: Challenges, tools, actions. Norwegian Resource Centre.

Social Impact. 2009. Strengthening Monitoring and Evaluation for Fragile States and Peace Building Programs [PowerPoint presentation].

U.K. Department for International Development (DFID). 1997. Guidance on Humanitarian Assistance. Available at

http://www.oecd.org/dataoecd/8/48/48092571.pdf



https://www.oecd.org/dac/evaluation/dcdndep/39774573.pdf



http://www.pactworld.org/sites/default/files/DQM%20Manual_FINAL_November%202014.pdf

http://www.pactworld.org/sites/default/files/DQM%20Manual_FINAL_November%202014.pdf

http://www.pactworld.org/documents/field-guide-evaluation

http://mymande.org/sites/default/files/images/Country_ledMEsystems.pdf



https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/404342/Humanitarian-Response-Funding-Guidelines-2015.pdf.

---. 2009. Political Economy Analysis How to Note. Available at: https://www.odi.org/sites/odi.org.uk/files/odi-assets/events-documents/3797.pdf.

U.S. Government. n.d. Plain Language.gov: Improving Communication from the Federal Government to the Public [website]. Available at http://www.plainlanguage.gov/index.cfm.

United Nations Development Programme (UNDP). 2009. Handbook on Planning, Monitoring and Evaluating for Development Results. New York: UNDP. Available at http://web.undp.org/evaluation/evaluations/handbook/english/documents/pme-handbook.pdf.

University of Reading Statistical Services Center. 1998. Project Data Archiving: Lessons from a Case Study.

U.S. Centers for Disease Control and Prevention. 1999. Morbidity and Mortality Weekly Report (Vol. 48). Atlanta: Centers for Disease Control and Prevention.

USAID. 2011. USAID evaluation policy. Washington, DC: USAID. Available at http://www.usaid.gov/evaluation/policy.

---. 2016. USAID Forward [web page]. Available at http://www.usaid.gov/results-and-data/progress-data/usaid-forward.

World Bank. 2007. International Program for Development Evaluation Training (IPDET) Handbook. Washington, DC: World Bank.

Bibliography for Peace and Conflict Terms

Call, C. 2008. Knowing Peace When You See It: Setting Standards for Peacebuilding Success. Civil Wars 10(2): 173–194. Available at http://www.informaworld.com/smpp/content~content=a793148283~tab=send.

Collier, P., & A. Hoeffler. 2004. Greed and Grievance in Civil War. Oxford Economic Papers 56(4): 563–595.

European Union External Action. n.d. Conflict Prevent, Peace building and Mediation [web page]. Available at http://eeas.europa.eu/cfsp/conflict_prevention/index_en.htm.

Fearon, J., & D. Laitin. 2003. Ethnicity, Insurgency, and Civil War. American Political Science Review 97(1): 75–90.

Lawry-White, S. 2002. Review of the UK Government Approach to Peacebuilding. London: DFID. Available at https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67946/ev646.pdf.

Le Billon, P. 2005. Fuelling War: Natural resources and armed conflict. The Adelphi series edition no. 373. Available at https://www.iiss.org/en/publications/adelphi/by%20year/2005-1a3b/fuelling-war--natural-resources-and-armed-conflict-ebea.



https://www.odi.org/sites/odi.org.uk/files/odi-assets/events-documents/3797.pdf

http://www.plainlanguage.gov/index.cfm

http://web.undp.org/evaluation/evaluations/handbook/english/documents/pme-handbook.pdf

http://web.undp.org/evaluation/evaluations/handbook/english/documents/pme-handbook.pdf

http://www.usaid.gov/evaluation/policy

http://www.usaid.gov/results-and-data/progress-data/usaid-forward

http://www.usaid.gov/results-and-data/progress-data/usaid-forward

http://www.informaworld.com/smpp/content~content=a793148283~tab=send

http://eeas.europa.eu/cfsp/conflict_prevention/index_en.htm

https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67946/ev646.pdf

https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67946/ev646.pdf

https://www.iiss.org/en/publications/adelphi/by%20year/2005-1a3b/fuelling-war--natural-resources-and-armed-conflict-ebea

https://www.iiss.org/en/publications/adelphi/by%20year/2005-1a3b/fuelling-war--natural-resources-and-armed-conflict-ebea



Leonhardt, M. 2000. Toward a Unified Methodology: Reframing PCIA. In Bergdorf Handbook for Conflict Transformation. Available at http://www.berghof-foundation.org/fileadmin/redaktion/Publications/Handbook/Dialogue_Chapters/dialogue1_leonhardt.pdf.

United Nations. 2008. United Nations Peacekeeping Operations Principles and Guidelines. Available at http://www.un.org/en/peacekeeping/documents/capstone_eng.pdf.

United Nations Development Programme. n.d. Bureau for Crisis Prevention and Recovery Overview [web page]. Available at http://www.undp.org/content/undp/en/home/ourwork/crisispreventionandrecovery/overview.html.

United Nations General Assembly and United Nations Security Council. 2007. Report of the Peacebuilding Commission On Its First Session, June 2006 - June 2007, A/62/137-S/2007/458, 4. Available at http://www.securitycouncilreport.org/atf/cf/%7B65BFCF9B-6D27-4E9C-8CD3-CF6E4FF96FF9%7D/PBC%20A62137-S2007458.pdf.

USAID. 2005. Fragile States Strategy. Available at http://pdf.usaid.gov/pdf_docs/PDACA999.pdf.

---. 2012. Peacebuilding and Reconciliation. Available at https://www.usaid.gov/what-we-do/working-crises-and-conflict/peacebuilding-and-reconciliation.

Williams, A.Z. 1999. Sierra Leone: The Political Economy of Civil War, 1991–98. Third World Quarterly 20(1): 143–162.

World Bank. 2008. State and Peace-Building Fund. Available at http://www.worldbank.org/en/programs/state-and-peace-building-fund.

http://www.berghof-foundation.org/fileadmin/redaktion/Publications/Handbook/Dialogue_Chapters/dialogue1_leonhardt.pdf



http://www.un.org/en/peacekeeping/documents/capstone_eng.pdf

http://www.undp.org/content/undp/en/home/ourwork/crisispreventionandrecovery/overview.html

http://www.undp.org/content/undp/en/home/ourwork/crisispreventionandrecovery/overview.html

http://www.securitycouncilreport.org/atf/cf/%7B65BFCF9B-6D27-4E9C-8CD3-CF6E4FF96FF9%7D/PBC%20A62137-S2007458.pdf

http://www.securitycouncilreport.org/atf/cf/%7B65BFCF9B-6D27-4E9C-8CD3-CF6E4FF96FF9%7D/PBC%20A62137-S2007458.pdf

http://pdf.usaid.gov/pdf_docs/PDACA999.pdf

https://www.usaid.gov/what-we-do/working-crises-and-conflict/peacebuilding-and-reconciliation

https://www.usaid.gov/what-we-do/working-crises-and-conflict/peacebuilding-and-reconciliation

http://www.worldbank.org/en/programs/state-and-peace-building-fund

monitoring, evaluation, reporting, and learning (merl) for ... · undp united nations development...

Documents