adaptive management for learning, improvement...
TRANSCRIPT
![Page 1: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/1.jpg)
Adaptive Management for Learning, Improvement & Evaluation of Health Programs
SoumyaAlva & Anne LaFondCenter for Health Information, Monitoring and Evaluation (CHIME)John Snow, Inc.September14, 2017
![Page 2: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/2.jpg)
2
OUTLINE
• What is adaptive management?• Why use adaptive management?• “CHN on the Go” – an example of JSI’s use of
adaptive management
![Page 3: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/3.jpg)
3
WHAT IS ADAPTIVE MANAGEMENT?
![Page 4: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/4.jpg)
4
EVALUATION THROUGHOUT THE PROJECT CYCLE
In some cases:Limited to Mid term evaluation
![Page 5: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/5.jpg)
5
ADAPTIVE MANAGEMENT
Evidence driven program implementation strategy
A big step beyond simple performance monitoring
Iterative approach to learning & evaluation while implementing
Interest in better implementation strategies for better outcomes
Increasingly being used in the field of public health
Data to drive adaptation
![Page 6: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/6.jpg)
6
DEVELOPMENTAL EVALUATIONAn approach to evaluation grounded in systems thinking and that supports innovation by collecting and analyzing real time data in ways that lead to informed and ongoing decision-making as part of the design development and implementation process.
- Michael Quinn Patton
![Page 7: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/7.jpg)
7
USAID LEARNING LABAdapting—or adaptive management—refers to intentionally and systematically using relevant knowledge to inform decision-making and ultimately take action. Within the development context, that action could be adjusting interventions or whole strategies, experimenting with new ways of working, scrapping programming that simply isn’t working, or scaling approaches that have demonstrated value.
- Monalisa Salib, in USAID Learning Labhttps://usaidlearninglab.org/lab-notes/adaptive-management-if-not-now-when
![Page 8: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/8.jpg)
8
PARTICIPATORY ACTION RESEARCH IN HEALTH
Participatory action research cycle. Oxford University Press, Australia, from F. Baum, 2016, The New Public Health 4th ed. Oxford University Press. www.oup.com.au
![Page 9: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/9.jpg)
9
THE ADAPTIVE MANAGEMENT CYCLE
![Page 10: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/10.jpg)
10
WHY USE ADAPTIVE MANAGEMENT?
![Page 11: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/11.jpg)
11
KEY ADVANTAGES
RELEVANT AND TIMELY DATA ARE NEEDED TO MAKE PROGRAM
IMPLEMENTATION EVIDENCE-BASED
Moving away from traditional M&E
Incorporate learning into implementation (testing implementation strategies)
Ability to develop a better program
Better tailoring of interventions to context & target populations
No need to wait for endline results, potential for early stakeholder buy-in
![Page 12: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/12.jpg)
12
HOW DOES THIS TRANSLATE TO THE REAL WORLD?
Quality ImprovementPDSA CyclesSTUDY
PLAN
DO
ACT
Organizational ManagementLearning Organization
Program Learning/ Routine Reflection
Iterative & Adaptive Program Strategy Development (e.g. developmental evaluation)
![Page 13: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/13.jpg)
13
THREE MANAGEMENT STRATEGIES
Traditional
Testing
Adaptive
• PMP• Baseline/Midline/Endline• Implementation Research
• PMP• Baseline/Midline/Endline• Implementation Research• Process Documentation &
Developmental Evaluation
• PMP• Baseline/Midline/Endline
• Management • Reporting
• Management • Reporting• Learning
• Management • Reporting• Learning• Adaptation
1
2
3
Purpose
Purpose
Purpose
![Page 14: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/14.jpg)
14
WHAT DOES IT TAKE TO MAKE THIS WORK?
Supportive funder & implementing partner
For funding and enabling team to iterate
Mixed methods –use of multiple data types
Researchers & evaluators who have a good understanding of program implementation
Stakeholders willing to engage
![Page 15: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/15.jpg)
15
JSI’S EXPERIENCEWITH ADAPTIVE MANAGEMENT:THE EXAMPLE OF “CHN ON THE GO”
![Page 16: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/16.jpg)
16
CCH & THE “CHN ON THE GO” APP
ICT via a smartphone application (CHN on the Go)
An innovative solution to address barriers in job satisfaction & motivation among Community Health Nurses (CHNs)
Designed using an HCD approach to identify needs of CHNs
Improve connectivity & career development among CHNs Increase their linkage & interactions with a professional network & supervisors
Provide clinical refreshers
Innovations for MNCH Initiative: Funded by the BMGFConcern Worldwide : Implementing Partner 2009-2016 (2 Phases)JSI: Global Research Partner with in-country research partners (Sept 2012-Dec 2016)
![Page 17: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/17.jpg)
17
PROGRAM IMPLEMENTATION• Implemented by Grameen
Foundation in Ghana
• 5 Districts: Ningo Prampram, Ada East and West (Greater Accra), and South Tongu, South Dayi (Volta)
• Approximately 50 health centers and associated CHPS
• More than 300 CHNs/supervisors
• Project timeline:• Design process: Oct ‘13-March ‘14• V1 July 2014, V5 May 2016
GHANA
VOLTA
Togo
Burkina Faso
Gulf of Guinea
Côte d’Ivoire
GREATER ACCRA
![Page 18: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/18.jpg)
18
OBJECTIVES
Effect on greater health worker motivation/satisfaction?
Clinicalrefresher
Workplanning
Emotionalsupport
Career development
Connectivity
Professionalnetwork
![Page 19: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/19.jpg)
19
Baseline and endline levels of CHN motivation & job satisfaction? Differences by district, facility type, and CHN age?
Association of CHN on the Go with greater CHN motivation/ satisfaction?
Project implemented as planned? Barriers and supportive factors affecting implementation? Expected results as foreseen in the pathways in the TOC?
1
2
3
RESEARCH QUESTIONS
![Page 20: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/20.jpg)
20
JSI’S ROLE AS GLOBAL RESEARCH PARTNER
Develop Theory of Change
Rigorous evaluation approach
Mixed method approach
Real time data for adaptive management during the life of
the project
Process documentation
Monitoring data
Review meetings with implementers and
stakeholders
RME activities to understand impact of pilot
![Page 21: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/21.jpg)
21
Learning Center
Health knowledge
Point of Care Health knowledge,
relationshipsPlanner
Targets, Work scheduling
Achievement Center Targets,
relationships
Staying Well Keeping well, relationships
AN ADAPTATION OF THE THEORY OF CHANGE
Driver (Direct outcomes (Modules & Intermediate outcomes) Direct outcomes ) Driver
Change Pathways
WHATSAPPCommunication &
Connectedness
Supervisor AppMonitoring & Supervision
FINAL OUTCOMES: Better CHN performanceBetter quality of careImproved maternal & child health
ASSUMPTION: Health workers feel unmotivated and frustrated in their jobs
SYSTEM LEVEL CHANGES: Supervision, education/learning
![Page 22: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/22.jpg)
22
COMPONENTS OF ADAPTIVE MANAGEMENT
Refine Program Strategy using HCD
Process documentation, periodic reflection and review with program team
Endline Data Collection
Baseline Data Collection
Process documentation, periodic reflection and review with program team
Human centered design (HCD)
Process documentation, periodic reflection and review with program team
![Page 23: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/23.jpg)
23
RME TimelineProject Start
May – June 2014
Project EndMay 2016
BaselineMay – Aug 2014
PD Round 1May – Aug 2014
BaselineMay – Aug 2014
Feedback SessionApril 2015
Magpi e-surveyApril – May 2015
Pop-up QuestionsJune 2015
PD Round 2July – August 2015
Magpi e-surveyDec 2015 – Jan 2016
EndlineApril 2016
Pop-up QuestionsMay 2016
Other data collection activities: • Monthly field interviews• Monthly usage monitoring dashboard
Are pathways in
the TOC being followed?
Is implementation progress in relation to plans?
Are results on app use as expected according to the CCH TOC?
Reasons why different modules were used by CHNs?
Successes & challenges faced?
![Page 24: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/24.jpg)
24
OUR APPROACH
Designed & planned when project started
Worked closely with local research partners
Worked closely with implementing program staff (Concern/ Grameen)
Aware of all aspects of program implementation
Led research but in collaboration with Concern/Grameen
Review meetings on a regular basis
Information sharing with stakeholders
Analyzed PD and monitoring data in combination with endline findings
Create engaging visualizations
![Page 25: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/25.jpg)
25
• More probing into usage of the app and specific modules by CHNs
• Helped develop newer iterations of the app
• Greater focus on technical issues faced
• Greater focus on supervision
• Easy to understand visualization
• Built stakeholder interest
HOW DID ADAPTIVE MANAGEMENT HELP?
![Page 26: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/26.jpg)
26
BENEFITS• Increased qualitative or mixed data
availability – deeper than just simple monitoring
• Ability to review program strategy
• Near real time picture of implementation
• Sharing findings with stakeholders on an ongoing basis
• Increase in appreciation of the value of iteration among program managers
• Greater interest in methodology over time
CHALLENGES• Uptake and appreciation of
iterative measurement and routine reflection among program managers
• Time
• Ability to use data
• Program theory understanding
• Tension between evaluation standards and iterative and adaptive program design
• Complexity of the theory of change/change pathway
• Time and resource investment
+
++
+
+
+
-
-
-
-
![Page 27: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/27.jpg)
27
SUMMARY OF FINDINGS
• As a job aide for CHNs (work planning, knowledge, communication)
• As a job aide for supervisors (communication, monitoring & supervision)
• Helps CHNs
• Gain confidence
• Gain respect & recognition from colleagues, supervisors & clients
• Provides emotional support for some
App addresses various needs of CHNs & supervisors
Sustainability• CHNs want to continue using the app - both current and other
districts
• GHS support to scale up, using CHNs own phones, especially in Volta
![Page 28: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/28.jpg)
28
RESOURCES• Better Evaluation http://www.betterevaluation.org/en/taxonomy/term/1429
• USAID Learning Lab, Collaborating, Learning and Adapting (CLA). https://usaidlearninglab.org/faq/collaborating-learning-and-adapting-cla
• USAID’s US Global Development Lab (Monitoring, Evaluation, Research and Learning Innovations - MERLIN). https://www.usaid.gov/GlobalDevLab/about/monitoring-evaluation-research-and-learning-innovations-program.
• Doing Development Differently. http://doingdevelopmentdifferently.com/
![Page 29: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/29.jpg)
29
QUESTIONS FOR DISCUSSION
![Page 30: Adaptive Management for Learning, Improvement …mini-university.com/wp-content/uploads/2017/09/2017...Adaptive Management for Learning, Improvement & Evaluation of Health Programs](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f07dbb77e708231d41f1ac1/html5/thumbnails/30.jpg)
30
QUESTIONS• What are some of the decisions that need to be made at different stages of
project implementation? What questions need to be answered? And for what purpose?
• 6 months after implementation of a 2-3 year project, mid-term, 6 months before end of project
• What types of data are useful at each stage to answer questions about project fidelity, implementation, strategy relevance and project performance?
• How do you make better decisions with the data available?
• What are some of the ways you can use data for decisionmaking?
• When and how do you engage stakeholders?
• How do you enable them to use data at all stages?