Understanding Why, When, and What it will take to do Operations and/or

Implementation Research in a Program

Experiences from CSHGP and TRAction ProjectsFlorence Nyangara, PhD, MCHIP/ICFPre-CORE Spring Meeting on OR/IR

April 22nd, 2013

Objectives of this OR/IR Workshop

1. Establish a common understanding of terms used in program-based research (learning by doing) - OR/IR

2. Share knowledge and skills on why and how to identify & formulate OR/IR research questions, and decide on research approaches that fit the questions

3. Recognize what it will take to undertake OR/IR within your program

4. Understand major issues surrounding OR/IR within community-based health programs i.e. by NGOs

Caveat!

The 4-hour workshop is NOT intended to make you OR/IR experts!

BUT, we hope you will leave here feeling that OR/IR can be done in your program

Workshop Format

Interactive and participatory with: Limited presentations just to overviews on taxonomy

of OR/IR, and about CSHGP & TRAction projectc

Discussions, demonstrations, and illustrative examples from existing implementations (CSHGP/TRAction)

Open discussions of common issues, unknowns, and future directions of OR/IR types of research

Brainstorming Activity 1

What is Operations Research (OR)? What is Implementation Research (IR)?

Instructions: Members of each table take 5 minutes to define what they understand by OR and IR terms; each group identifies 1 person to present the definitions to the audience!!

Terms Used in Program Research – Are You Confused?

Monitoring and Evaluation - routine

Implementation Research (IR) - Field

Implementation Science (IS)

Process Evaluation

Operations Research (OR)

Impact Evaluation

Within Routine settings: Clinical, community, Policy

Others e.g., Case studies

Overlap in Terminologies in Public Health Programs’ Research

Operations Research

OTHER (IS): Health Systems Research (HSR), QI

Implementation

Research

Imple

men

tatio

n

Scien

ce

Another Way of Understanding OR versus IR!Operations Research Implementation Research

Focus Addresses specific problems within a program by testing a feasible solution (problem must be what managers can address (control)

More general research inquiry focusing on "what is happening and why" in all program aspects -design, implementation, administration, operation, services, & outcomes

Approach Uses scientific techniques -wide range - descriptive to experimental for impact assessments - i.e. evaluative studies

Uses scientific techniques: -------- - - Describes program experiences - Assesses and explains, what/why- No experimental studies, i.e. not designed to compare “what is happening Vs. what would have happened

Purpose Effectiveness, efficiency, quality, access, scale-up, sustainability

Identify common implementation problems, their determinants, develop solutions, identify which solutions need further research, and test implementation strategies to address them and improve access, scale-up, sustainability

Present examples on OR and IR

OR – CSHGP – Africare example IR – TRAction

Overview of the Child Survival & Health OR Grants 30 CSHGP/OR studies imbedded within diverse programs implemented

by iNGOs including MNCH, Nutrition, family planning, Malaria, etc to support national (i.e. MOH) efforts to reduce maternal & child mortality in low resourced countries

CSHGP mainly focus at community level health systems, remote areas, and vulnerable populations to address inequities and improve overall MNCH outcomes but links all levels of the HS (national community)

CSHGP develops new partnerships with local communities, CBOs, research institutions, and national stakeholders to design and implement responsive programs

Strong partnerships between iNGOs, North/South Universities, MOH, several CBOs, and a few donor agencies i.e. UNICEF and WFP

10

Key Guiding Principles for Successful OR?

1. Contextual relevance (i.e. aligned with MOH policy/strategy/plan)

2. Use-focused results (i.e., how results will inform/influence/change)

3. Diverse and strategic partnerships 4. Stakeholder engagement and learning

Examples of Programming Barriers Identified for CSHGP/OR

Inadequate human capacity, i.e. limited availability of skilled and motivated CHWs

Limited accessibility to services i.e.: Physical inaccessibility (i.e. distance to facility) Financial – inability to pay Cultural, etc

Poor quality of available services i.e. lack of coordination of services and continuum of care

Poor information and monitoring systems i.e.: Limited availability and use of evidence-based data for

decision-making

Examples of Solutions tested through CSHGP/OR

Maternity Waiting Homes (MWH) Incentivize, motivate and retain community-based human

capacity e.g. Community Health Workers (CHW) Integrate service delivery within the health sector and/or

across other relevant sectors (i.e. Agriculture, education) System Integration or coordination for continuum of care Strengthen local community structures and build their capacity

(i.e. commitment, resources, skills) Strengthen Community-based Health Information Systems for

informed decision making

Illustrative CSHGP/OR Example

Maternity Waiting Homes (MWH) by Africare, in Liberia (2010 Cohort)

Key Partners: University of Michigan, MOHSW, County Health Teams

Programming Barrier addressed by Africare/OR

Problem: physical inaccessibility (long distance, poor roads to facility) to get Maternal and Newborn services (i.e. Skilled Birth)

Policy: MOH’s national policy directive promotes only facility based skilled births (no home delivery!)

Africare, Liberia --------ContinuedSolution: establish Maternity Waiting

Homes (MWH) structures near/close to a health facility for pregnant women to stay 2-3 weeks before delivery due date and post-delivery if they choose to rest/need further observation)

Relevance: aligned with MOH’s national policy directive for only facility based skilled births. Therefore findings likely to be used to inform the implementation of this policy

Results: increase facility-based births and access to other MNC services and improved outcomes

IR by TRAction Project

How OR is done?

OR Steps and Processes: CSHGP/SCALE

Problem Analysis in consultation with key

stakeholders

Develop Solution (Develop

objectives & questions)

Conduct Formative Research

Refine the Intervention mix

i.e. concepts, messages, survey

tools, training materials, etc

Incorporate findings to program operations

and monitor

Ongoing Analysis and Use of Information

Implement the study design & conduct

ongoing monitoring & documentation

Evaluative / Summative Phase

Final Evaluation of Study - Reflect on Findings with

Stakeholders

Scale-up successful interventions

Problem Analysis and Contextualization

Identify and define the operational problem (barrier) hindering uptake of interventions

Identify subsystems affected by the problem Analyze and narrow the problem (i.e. lit. review,

experience, context, observations, etc) Prioritize the specific aspects of the problem for

OR that if addressed might bring the most impact

Solution/Intervention Development

What are the objectives of proposed solution? What is the rationale for selecting this solution? Identify & prioritize specify decision variables i.e.

Training methods, amount of ORS packets Specify constraints on the solution i.e. SES Specify the key intervention mix (package) Specify research questions Develop a conceptual model -- important

Describe the OR Interventions in Detail

Intervention components must be described in detail for replication or scale-up

Without details we are often left knowing very little about the details of an intervention or the functional relationship between the components of the intervention and outcomes

Question?Instructions: Using the MWH intervention by

AFRicare as an example, please list 3 key components that must be included in MWH package?

1.2.3.

Some ResponseInstructions: Using the MWH intervention by

AFRicare as an example, please list 3 key components that must be included in MWH package?

1. Structure (building)2. Community outreach3. Training (CHWs, facility staff, etc)

Objectives and research questions: GOOD

Objectives Research questionsObjective #1: To develop content of intervention to deliver PPFP at household level

Q 1.1: What are main barriers to promotion and acceptance of PPFP, and how can they be addressed?Q 1.2: What is optimal timing and content of PP home visits by community midwives?

Objectives and Research questions for the Formative Research concern the specific intervention content

Objectives and research questions: GOOD

Objectives Research questionsObjective #1: To develop content of intervention to deliver PPFP at household level

Q 1.1: What are main barriers to promotion and acceptance of PPFP, and how can they be addressed?Q 1.2: What is optimal timing and content of PP home visits by community midwives?

Objective #2: To measure quality and impact of intervention

Q 2.1: Can adequate supply of contraceptives be maintained at community level?Q 2.2: What is quality of counseling by CM?Q 2.3: What is effect of intervention on LAM and modern contraceptive use?

Objectives and research questions: BAD

Objectives Research questionsObjective #1: To conduct formative research using focus groups

Q 1.1: Why do people have wrong beliefs about family planning?Q 1.2: Why do people think that babies need extra water in addition to breast milk?

Too specificQuestions anticipate a certain answerAssumption that community is wrong

“To conduct FR” can’t be an objective Methods shouldn’t be in the objectives

Objectives and research questions: BAD

Objectives Research questionsObjective #1: To conduct formative research using focus groups

Q 1.1: Why do people have wrong beliefs about family planning?Q 1.2: Why do people think that babies need extra water in addition to breast milk?

Objective #2: To prove that the intervention is highly effective

Q 2.1: What is the contraceptive prevalence rate?Q 2.2: What is the mean age of women who use contraceptives?Q 2.3: How much lower is the maternal mortality rate in the intervention group compared to the comparison group, from the household survey?Q 2.4: What is the cost-effectiveness of contraceptive distribution by midwives?

Objectives and research questions: BAD

Research questionsQ 2.1: What is the contraceptive prevalence rate?Q 2.2: What is the mean age of women who use contraceptives?Q 2.3: How much lower is the maternal mortality rate in the intervention group compared to the comparison group, from the household survey?Q 2.4: What is the cost-effectiveness of contraceptive distribution by midwives?

Questions too specific

Insufficient sample size for maternal mortality

Impossible to determine given the methods

Formative Research: Objectives

Define the intervention content Design and pre-test specific intervention

components: Communication materials Counseling guidelines Packaging/instructions for delivering the services

Pilot test entire package on small scale e.g. a few villages to refine the intervention

Formative Research: Methods

Women Men, mothers-in-law

Community midwives

In-depth interviews or FG to identify facilitators /barriers

Pretesting of materials and counseling guidelines

Pilot test of PP home visits

Evaluative Research: Objectives

Assess the quality of intervention Measure coverage of intervention Measure study outcomes Implement, Evaluate, and Scale-up the Solution

Ongoing evaluation and modifications if needed Scale-up – Integrate the solution with the larger

system

Rigor/Study design: How believable is the evidence?

Non-experimental (Case study)• Exploratory to see how to implement the intervention• Ongoing documentation on how bets it can be implemented• No comparison group

Quasi-experimental (most common for CSHGP/OR)• Two groups (intervention and comparison)• Non-random selection to groups

Experimental• Two groups (intervention and Control)• Random selection of subjects (1 - CSHGP/OR using clusters)

A generic study design diagram

Source: http://hsl.lib.umn.edu/biomed/help/understanding-research-study-designs

A generic diagram: Information to include

Who was your population? Site and study participants

How did you decide who received the intervention?

What did the intervention and comparison groups receive?

What methods are you using to compare outcomes in intervention and comparison groups?

What analysis will you conduct to compare outcomes in intervention and comparison groups?

Reporting results of a study: What happened?

Source: http://www.jpgmonline.com/article.asp?issn=0022-3859;year=2007;volume=53;issue=4;spage=221;epage=227;aulast=Odabasi;type=0

Reporting results of a study: What happened?

CSHGP OR Study Design: Quasi-Experimental

Quasi-experimental (most common for OR)• Two groups (intervention and comparison)• Non-random selection to groups

Challenges of designing a rigorous OR design Difficult for program staff to design research study (limited skill &

time)

Non-random placement of program participants to - intervention areas and control areas often not comparable

Suitable control areas may not exist – other programs in control

areas or cross-over of interventions to control areas

Need to control for other factors beyond the program that might affect outcomes

Interventions are often multi-faceted – causality difficult to judge

Challenges of designing a rigorous OR design

Timing – projects are already underway and it is hard to incorporate a strong evaluation design

Scale – many projects are too small to expect to be able to demonstrate impact

Pressure for rapid results before projects ends

Ethical considerations

Cross-cutting issues and concluding thoughts

Open discussions of common issues, unknowns, and future directions of OR/IR types of research

1.What are the common issues?2. What else should we know – unknowns 3. Suggestions for Future directions of OR/IR types

of researchInstructions: Divide into 3 groups and choose 1, 2,

or 3, discuss among yourselves and with audience?

Web-based Tools and Resources

www.MCHIPNGO.net

www.coregroup.org

www.MCHIP.net

43

Reference Materials on Operations and Implementation Research