monitoring & evaluation
DESCRIPTION
TRANSCRIPT
Monitoring & Evaluation Using
Evidence Based Participatory Quality Improvement Techniques in
Public Health Programs
Presented by,
Mr. Rakesh. N. Pillai
Socio Research and Reform Foundation (NGO) - SRRF
1
STEPS TAKEN TO IMPROVE THE QUALITY OF A PUBLIC HEALTH PROGRAM
- AN EXAMPLE
Background
• India has 2nd leading global case load of Diabetes
• National prevalence is estimated at 8.3%*
• Diabetes Educators are not recognized as a health professional at the start of this program
India Diabetes Educator Program
2007-2011
*Source: International Diabetes Federation, Diabetes Atlas 5, 2011 2
Program Objective
• To develop a cadre of Diabetes Educators in India, creating a favorable environment to increase their acceptance in patient care
• To increase positive self efficacy of Diabetes Patients by promoting self care behaviour and achieve clinical metabolic control
Partnerships
- Society for Action in Community Health – SACH India
- Project HOPE, USA & Other implementing partners
- Eli-Lilly, BD, Bayers & Many Others
3
Program Overview
4
M&E Team - Objective
• To improve the Quality of the Program
• Set up an Evidence based Continuous Monitoring and Evaluation system
• Create a Health Management Information System (HMIS)
• Establish a mechanism for regular / quarterly evaluations and take corrective steps
• Overall to foster an environment for the success of the program
5
What is M&E ??? Note: M&E should be done at the planning phase
M&E
Document Program Activities
Summarize Figures
Establish Uniformity
Create Evidences
Disseminate Results
Generalize Results
Reduce Biases
6
QC – Seven Step Formula
Significance :
‘You can’t get good without knowing the basics’
QC- 7 step formula is the basic procedure for solving problems scientifically, rationally, efficiently and effectively.
This is a problem solving strategy which allows any individual to solve difficult problems rationally and scientifically
7
What is QC – Seven Step Formula Step Purpose Action Items
1 Understand Topic -Identify Problem -Decide topics to address
2 Understand Situation & Set targets
-Mechanisms for collecting information -Decide on targets (value and deadline)
3 Plan Activities -Decide ‘what to do’ -Decide schedule, Divide responsibilities
4 Analyse Causes -Check present values of characteristics -List possible causes, Items to tackle
5 Consider & implement counter measures
-Propose ideas for counter measures -Ways to put counter measures in place -Check details of counter measures
6 Check results -Compare results with targets -Identify tangible and intangible benefits
7 Standardize & Establish Control
Establish new standards & revise old, Decide on methods to control, Educate
8
Step 1 : Understanding the Topics – Diabetes Educator Program
» Check the roles of each departments & roles - Central Co-ordinating Office
- Finance & Admin - Program Implementing Team - Each Partner centres- MT’s & DE’s
» Check the policies & objectives of departments » List various problems by each departments » Select most appropriate problems to be tackled > Pareto Principle – 80% of problems could be
attributed to 20% of the causes Tools Used a. Cause & Effect Diagram b. Pareto Diagram 9
Components of the M&E System ?
M&E system should establish a system to;
1. map Performances of the Program
- through performance/outcome indicators
- during baseline, interim and final evaluation
2. map the Progress – through Process Indicators
- Quarterly reports
- Bi-annual reports etc
3. Detailed Implementation Plan (DIP)
4. Monitoring Cost & Time Schedules 10
» Decide on control characteristics
» Survey the past trends & existing levels
» Investigate the flow of work & Processes
» What sorts of changes occurred over time
» Stratify data as per problems
» Statistical dispersion / deviation is measured
» Prioritise
Tools Used:
Graphs, Checklists, Histograms, Scatter Diagram
Step 2 : Understand Situation & set targets – Diabetes Educator Program
11
We Strategized M & E as;
PDCA Wheel
Act Plan
Check Do
Source: TQM Approach / Kaizen Principle 12
We decided on;
a. Action Items
b. Schedule of activities
c. Activity Plan
Tools Used : Control Charts, Check lists etc.
Step 3 : Planning M&E – Diabetes Educator Program
13
Options considered for M&E
1. ‘Health Management Information Systems (HMIS)’
- Many software programs are available
2. Finally decided to make a simple, Excel sheet
- ‘Program Design Tool (PD Tool)’
14
Step 4: Analysis Questions to design ‘PD Tool’
According to
a. 5W1H [What, When, Where, Who, Why and How]
b. 4M Technique [Manpower, Machines, Materials and Methods]
• How many readings do we need?
• What period should we collect data?
• How to stratify data?
• What sampling or measurement methods?
15
Step 5: Design Counter Measures
16
Consider Problem from all sides
Collect ideas from experts
Collect ideas from various stakes in the processes
Using lateral thinking
Select countermeasure proposals
- Effect
- Feasibility
- Economy
‘PD Tool’ Definitions for Diabetes Educator Program
1. Performance Indicators
- High level indicators that lead directly to your Goal and Objectives
- Usually in Percentages
- Common for Baseline, Interim and Final
Objective Performance/Outcome Indicator
Foster an enabling environment for diabetes education and training in India
% improvement in wellbeing of Diabetes Patients
17
Definition’s & Examples ...Contd
2. Process Indicators
- Indicators that measure progress at the activity level.
- Usually simple counts
Program Specific Activities Process Indicators
Develop national curriculum for diabetes education
# of recognized curriculum developed
Conduct Master Trainer training
# of Master Trainers trained
18
Definition’s & Examples ...Contd
3. Detailed Implementation Plan (DIP)
- Based on Process Indicators / Activity based
- Map the progress every quarter through quarterly reports
Process Indicators
Target Year 1 Year 2 Year 3 Year 4
# Students certificate in Diabetes Education
3000 740 810 860 920
19
DIP System Hide blank rows
Target LOP
Year 2 Year 3 Year 4 Hide blank rows
Hide blank rows
Process Indicators
Q1 Q 2 Q 3 Q 4 Q1 Q 2 Q 3 Q 4 Q1 Q 2 Q 3 Q 4
Sep-08
Dec-08
Mar-09
Jun-09
Sep-09
Dec-09
Mar-10
Jun-10
Sep-10
Dec-10
Mar-11 Jun-11
Hide blank rows
1 # of Master Trainers trained
35 35 35
Hide blank rows Actual achieved 66 22 22 12
Hide blank rows
2
# of MT's actively engaged in conducting Diabetes Education and training 30
30
Hide blank rows Actual achieved 44
Hide blank rows
3
# Students enrolled in PG Certificate in Diabetes Education 4,325
4,325 325 500 500 500 500 500 500 500 500
Hide blank rows Actual achieved 4,542 304 464 482 480 470 500 411 448 450
Hide blank rows
4
# Students of PG Certificate in Diabetes Education presenting the Final Exam 3,000
3,000 197 80 90 100 100 200 200 547 581
Hide blank rows Actual achieved 2,855 197 80 90 100 100 200 200 547 581
20
Quarterly Report Reporting Period
Year 2011
Scoring Current Quarter green
Version 6.0, September 2010 Approvals
Expected to Date green Beneficiaries green
Project Name Diabetes Educator Project Country India 30-Dec-11 % Elapsed: 100% Project Code 05058
Goal Decrease morbidity and mortality related to diabetes in India Outcome Indicators
Indicator Baseline Year 4 Final Targe
t +/-
target +/- baseline
Data of data collection (planned or actual)
Date Result
30-Jun-11 1
average a1c for patients 30-Dec-
10 8
7.3
7.5 20% 80%
2
% improvement in well being index (WHO 5) 30-Dec-
10 61%
66%
71% -5% 5%
3
improvement in patients Self Efficacy score 30-Dec-
10 11
14
16 -200% 300%
Process Indicators Current Quarter
Expected to Date %
Description Target Actual
4 # of Master Trainers trained 0 0 189%
5 # of MT's actively engaged in training 0 0 147%
6 # Batches planned/initiated 0 0 99%
7 # Students enrolled i 0 0 105%
8 # Students presenting the Final Exam 0 0 103%
9 # Students graduating 0 0 103%
10 # Annual National Meeting 0 0 100%
11 # Partner Meetings 0 0 100%
12 # Abstracts accepted 0 0 100%
14
# Program evaluations conducted (mid/end term)
0 0 50%
15 % of students dropping out of the course 0% 0% 105%
Total Direct Beneficiaries 0 0 104% Expected to Date Direct + Indirect
213% Indirect Beneficiaries 0 0 214%
21
Timeline Set for the Program Activities Timeline
Project Name:
India Diabetes Educator Project
Version 3.0, September 2010 Country: India Start Date:
1-Jul-07 End Date:
Optional page
Activities Year 1 Year 2 Year 3
Program Specific Activities Q 1 Q 2 Q 3 Q 4 Q 1 Q 2 Q 3 Q 4 Q 1 Q 2 Q 3 Q 4
Sep-07
Dec-07
Mar-08
Jun-08
Sep-08
Dec-08
Mar-09
Jun-09
Sep-09
Dec-09
Mar-10
Jun-10
1 Develop national curriculum for diabetes education
2 Develop partnerships with leading centers for diabetes education, care and management
3 Develop regional network/TAG of Des
4 Conduct Master Trainer training
5 Master trainers to train Diabetes Educators
22
HMIS Developed
23
Hide blank rowsProgram Design Version 3.0, September 2010
Hide blank rows
Hide blank rows Location
Hide blank rows Start Date 1-Jul-07 $ award
Hide blank rows End Date 30-Dec-11
Hide blank rows
Hide blank rows
Hide blank rowsHide blank rows
Hide blank rowsFrequency
Hide blank rows1
Foster an enabling environment for diabetes education
and training in India1 average a1c for patients One time
Hide blank rows
2
Design and develop Diabetes Education distance learning
course for allied healthcare professionals benchmarked
on the best standards and incorporating Indian medical
practices
2 % improvement in well being index (WHO 5) One time
Hide blank rows
3
Increase knowledge, enhance education skills and
improve practice of allied healthcare professionals in
diabetes education, care and management
3 improvement in patients Self Efficacy score One time
Hide blank rows
4Improve training outcomes among people living with
diabetes managed by allied healthcare professionals
trained in diabetes education by Project HOPE
Hide blank rows
Hide blank rows
Hide blank rows
Hide blank rows1 Develop national curriculum for diabetes education 1 # of IDF recognized curriculum developed One time
Hide blank rows2
Develop partnerships with leading centers for diabetes
education, care and management2 # of centres adopting HOPE Diabetes Educator Program Annually
Hide blank rows3 Develop regional network/TAG of Des 3 # of regional network/TAG of Diabetes Educators formed Annually
Hide blank rows4 Conduct Master Trainer training 4 # of Master Trainers trained Baseline & final
Hide blank rows5
Master trainers to train Diabetes Educators5
# of MT's actively engaged in conducting Diabetes
Education and trainingSemi-annually
Hide blank rows6
Supervise/mentor Master Trainers6
# Batches planned/initiated for imparting HOPE PG
Certificate in Diabetes EducationSemi-annually
Hide blank rows7
Conduct annual MT meetings7
# Students enrolled in PG Certificate in Diabetes
EducationSemi-annually
Hide blank rows8
Conduct partner/donor meetings8
# Students of PG Certificate in Diabetes Education
presenting the Final ExamSemi-annually
Hide blank rows
9 Disseminate results 9
# Students graduating with HOPE's PG Certificate in
Diabetes Education with > = 70% grade in each milestone
and overall
Annually
IDEP Database
Guidelines and peers/
experts
IDEP Database
Data Source
IDEP Database
Frequency
Goal
Program Specific Activities Process Indicators
IDEP Database
IDEP Database
Donor Lilly, BD and Bayer
Objectives
IDEP Database
Problem Statement
Outcome Indicators
Networks/ MOU
Networks/ MOU
Decrease morbidity and mortality related to diabetes in India
Project Name
IndiaCountry
India Diabetes Educator Project
Diabetes is a global pandemic and India has the highest case load of any country. While patient education is key to promoting effective
self-management, no formal Diabetes Educator program exists at the national level. Poor self-management leads to increased morbidity
and mortality and diabetes is currently the second leading cause fo death nationwide.
Launch a sustainable model for forming allied health professionals as diabetes educators nationwide, in partnership with regional
centers of excellence.
Duration (# yrs)
Data Source
Program Snapshot
India
Evidences from
operations research
Evidences from
operations research
Evidences from
operations research
Step 6: Check Results
24
• Operational Research was Initiated
To evaluate the impact of HOPE-trained DEs on patient outcomes
a. Clinical parameters
b. Psychosocial parameters
c. Behavior change parameters
Results – Baseline to 6 Months Clinical Measures
Baseline
Mean (S.D)
6-Month
Mean (S.D) Significance
HbA1C 8.13 (1.54) 7.35 (1.15) < .0001
Fasting blood glucose (mg/dl) 148.7 (39.9) 130.4 (34) < .0001
Body mass index (BMI) 27.1 (4.5) 26.0 (4.1) < .0001
Total Cholesterol 174.7 (37.9) 163.4 (32.9) < .0001
Triglycerides 143.4 (74.3 ) 139 (55.8) .06
HDL Cholesterol 42.2 (11.5) 59.3 (28.2) < .0001
LDL Cholesterol 101.9 (31.2) 79.9 (36.8) < .0001
Blood
Pressure
Systolic BP
132.7 (13.8 )
129.1 (12.4) < .0001
Diastolic BP 80.9 (6.5) 79.7 (6.5) < .0001
* All analyses conducted using paired t-tests
Results – Baseline to 6 mos Psychosocial /Behavioral
Variables
Baseline
Mean (S.D)
6 mos
Mean (S.D) Significance
Well-being 61 (16.8) 65.7 (23.5) < .0001*
Self Efficacy 11.1 (3.5) 13.7 (3.6) < .0001*
% people eating 4+ fruits or
vegetable per day
14.8 % 43.9 % < .0001**
% people drinking 2+ litres of
water per day
58.1 % 70.2 % < .0001**
% people doing 30+ minutes of
exercise per day
22.8 % 71.9 % < .0001**
% people doing self monitoring 15.2 % 54.6 % < .0001**
* Analyses conducted using paired t-tests
** Analyses conducted using Chi Square
Conclusion
QC Seven Step Quality Improvement Technique is an useful tool to strategise Monitoring and
Evaluation Plans, especially in large, multicentre, Public Health Programs
Mastering this tools helps the Program Managers to Plan, Do, Check and Act upon the Objectives
finally achieving the Program Goals
27
Thank You
28