Beyond logical frameworks to program impact pathways

CIIFAD M&E Workshop5 November 2011 135 Emerson Hall

Sunny S. Kim, MPH PhD candidate

Division of Nutritional Sciences

Role of research in evaluation

• Evaluation usually focuses on an internal situation, such as collecting data about specific programs, with no intent to generalize the results to other settings and situations. Evaluation determines the merit, worth, or value of things.

• Evaluation asks what should be, which makes reference to the standards and criteria of evaluations.

• Program-related research usually conducted with to the intent to generalize the findings from a sample to a larger population.

• Research asks why, which indicates a theoretical basis.

Basic steps of evaluation

• Identifying program goals and objectives

• Mapping the program

• Selecting an evaluation design– Sample size, control group, sampling frame

• Selecting indicators

• Determining data collection strategy

• Developing data collection instruments

• Analyzing the data

• Communicating the results

Program theory vs. logic model

• Program theory explains why a program is expected to work, and a logic model illustrates a program theory.

• Program logic model is a picture of how your organization does its work… it links the theoretical assumptions of the program and program activities/processes and outcomes.

Translating program theory…

Program theory has also been referred to as:

• program logic or intervention logic• theory-based or theory-driven evaluation • theory of change• theory-of-action• impact pathway analysis

all ways of developing a causal modal, linking program inputs and activities to intended or observed outcomes

Translating program theory…

• Program theories are often captured in a series of “if-then” statements.

• And between the “if” and “then,” there should be “how” and “why”, i.e. evidence or well-established connections (established theory, previous results, published research, feedback from participants).

IF… THEN…

IF… THEN…

+ CONTEXT

… into logic models• a commonly-used tool for illustrating program theory. • a picture of describing the sequence of activities to

bring about change and how these activities are linked to the results the program is expected to achieve.

• often presented in a flow chart that illustrates the linkages between program components and outcomes:

Inputs Resources, supplies, staffActivities Processes, techniques, tools, events, technology, actions Outputs Direct results of activities (size and/or scope) Outcomes Changes in attitudes, behaviors, knowledge, skills, status,

or level of functioningImpact Organizational, population, community, and/or system

level changes (improved conditions, increased capacity, and/or policy changes)

Program Logic Model Template

… into logic models • “Logic models come in as many sizes and shapes as the

programs they represent.” [W.F. Kellogg Foundation, 2001, p.7]

• No best logic model, only the model that fits your program best and provides the information you need in the format that is most helpful.

Why use a logic model? • Design and planning helps to explain and illustrate program

concepts and approach for key stakeholders (advocate and train); finds “gaps” in the theory or logic of a program and work to resolve them .

• Implementation and management focuses attention on the most important connections between actions and results for monitoring, improved programming, and reporting.

• Evaluation, communication, and marketing presents progress toward goals; provides a way to involve and engage stakeholders in the design, processes, and use of evaluation.

Why use a logic model? • Donor requirements:

Indicators

Means of verification

Assumptions

Goals

Objectives

Outputs

Activities

Resources

Goal: Health status improved

Strategic Objective: Use of key services and behaviors improved

Intermediate Result 1: Access and availability of services and supplies increased

Intermediate Result 2: Quality of services improved

Intermediate Result 3: Demand for key services and behaviors increased

Intermediate Result 4: Social and policy environment enabled

Log frame (OECD/DAC, USAID, CIDA…)

Results framework(USAID…)

Logic model example

Intervention model of Programa de Nutrición Infantil (PNI)

Maternal and child health and nutrition

Small-scale economic activities

Strengthening of civil society

Basic sanitation and sanitary education

Improved health and nutrition knowledge and practices

Improved development plans of CBOs in health and sanitation

Improved basic sanitation

Increased access to health services

Improved health and nutritional status of children <3 years

Improved health and nutritional status of pregnant women

Strategies: Information Education Social communication

FINAL OUTCOMES

INTERMEDIATE OUTCOMES

COMPONENTS

Intermediate Result 1: Improved nutrition, health knowledge and practices of those attended in the target communities

Strategic Objective: Improved health and nutritional status of children <3 years and pregnant women

Intermediate Result 3: Improved basic sanitation in the target communities

Intermediate Result 4: Community-based organizations applied their improved knowledge on health and sanitation in the formulation of their development plans

GOAL: Reduced rates of child malnutrition

Intermediate Result 2: Increased access to health services of the target population

Results framework of Programa de Nutrición Infantil (PNI)

From boxes to arrows…

• While “filling in” the logic models with the program components , we must not neglect the mechanisms and linkages… don’t forget about your program theory or impact pathways.

“The theme/issue is the training that you give through the participatory methdology to the mothers. If you just give the theory, it is the theme. The workshop is the practice. So if that day, for example, the mothers didn’t bring food to prepare their workshop, then the [health] promoter only gives the theory. They discuss, socialize, and receive only the theories in the small groups… Then the workshop returns another day to learn about doing. For example, the health themes, they make their workshop do dramatizations. The mothers do their dramatizations, it tells the priority, what happens and everything, and so they are going to learn.”

[Regional level, ex-ADRA adviser]

Description of mothers workshops

Impact pathway of mothers workshops

Husbands permit/ support women to participate

CHAs raise topic of health and care for children (i.e., diarrhea and infections; respiratory illness; early stimulation) and reproductive health and prenatal care

CHAs raise topic of self esteem and family violence

CHA raise topic of hygiene and sanitation

CHAs convene pregnant women and mothers of children <3 years to participate in workshops

CHAs and mothers discuss ideas and experiences

CHAs and mothers demonstrate recipes

Mothers understand nutritional needs of children and pregnant women and know feeding and cooking practices

Mothers recall nutritional needs and feeding and cooking practices

Mothers implement feeding practices and recipes

Nutritional status of pregnant women improved

CHAs raise topic of child feeding (i.e., BF and CF; balanced diets; feeding sick children) and nutrition for pregnant women

Children receive appropriate care and health attention in a timely manner

CHAs and mothers discuss ideas and experiences

Pregnant women consume adequate and appropriate foods

CHAs and mothers perform dramatizations

Mothers understand child health, reproductive health and prenatal care issues and know appropriate practices

Mothers recall child health, reproductive health and prenatal care issues and appropriate practices

Mothers implement care practices and seek health services

Pregnant women receive appropriate care and health attention in a timely manner

CHAs and mothers discuss ideas and experiences

CHAs and mothers perform dramatizations

Mothers understand issues of self esteem and violence and know how to address them

Mothers recall information on self esteem and violence

Mothers implement practices related to self esteem and violence

Children exposed to safe and caring home environments

CHAs and mothers discuss ideas and experiences

CHAs and mothers demonstrate practices

Mothers understand issues of hygiene and sanitation and know appropriate practices

Mothers recall information on hygiene and sanitation

Mothers implement hygiene and sanitation practices

Children exposed to safe and clean home environments and less exposure to contamination

Child nutrition and growth improved

Household food availability and access

Quality health service access and availability

Influencing factors –facilitators

Children consume adequate and appropriate foods

Household access to clean water

Women see incentive to participate (e.g. food baskets or small economic activities)

Use of local language and readily accessible foods

Husbands are sensitized about domestic violence

Repeated reminders or internal/external stimulus (e.g. home visits, concern for child’s future)

?

Main references• Kellogg Foundation. (2004). Logic model development guide: Using

logic models to bring together planning, evaluation, and action. Battle Creek, MI: W.K. Kellogg Foundation.

• Rogers, P.J. (2000). Program theory: Not whether programs work but how they work. In D.L. Stufflebeam, G.F. Madaus, & T. Kellaghan (Eds), Evaluation models, 209-232. Boston, MA: Kluwer Academic Publishers.

• Rogers, P.J. (2008). Using program theory to evaluate complicated and complex aspects of interventions. Evaluation, 14(1): 29-48.

• Taylor-Powell, E. & Henert, E. (2008). Developing a logic model: teaching and training guide. Madison, WI: University of Wisconsin-Extension.