assessing health worker performance of imci in kenya

Q U A L I T Y

A S S U R A N C E

P R O J E C T

Center for Human Services • 7200 Wisconsin Avenue, Suite 600 • Bethesda, MD 20814-4811 • USA

Assessing Health WorkerPerformance of IMCI in Kenya

Q U A L I T Y A S S E S S M E N T C A S E S T U D Y

The Quality Assurance Project (QAP) is funded by the U.S.Agency for International Development (USAID), under ContractNumber HRN-C-00-96-90013. QAP serves countries eligible forUSAID assistance, on USAID Missions and Bureaus, and otheragencies and nongovernmental organizations that cooperate withUSAID. The QAP team consists of the Center for HumanServices (CHS), the prime contractor; Joint CommissionInternational (JCI); Johns Hopkins University School of Hygieneand Public Health (JHSPH), Johns Hopkins University Center forCommunication Programs (JHU/CCP); and the Johns HopkinsProgram for International Education in Reproductive Health(JHPIEGO). Together, they provide comprehensive, leading-edgetechnical expertise in the design, management, andimplementation of quality assurance programs in developingcountries. The Center for Human Services, the nonprofit affiliateof University Research Co., LLC, provides technical assistance inthe research, design, management, improvement, and monitoringof healthcare systems and service delivery in over 30 countries.

QUALITYASSURANCE

PROJECT

TEL (301) 654-8338

FAX (301) 941-8427

www.qaproject.org

About this seriesThe Case Study Series presents real applications ofQuality Assurance (QA) techniques in developing countries atvarious health system levels, from national to community. Theseries focuses on QA applications in maternal and reproduc-tive health, child survival, and infectious diseases. Each casestudy focuses on a major QA activity area, such as qualitydesign, quality improvement, communication and develop-ment of standards, and quality assessment. In some cases,more than one QA activity is presented.

Quality assessment is the measurement of the quality ofhealthcare services. A quality assessment measures thedifference between expected and actual performance toidentify opportunities for improvement. Performance stan-dards can be established for most dimensions of quality, suchas technical competence, effectiveness, efficiency, safety, andcoverage. Where standards are established, a qualityassessment measures the level of compliance with stan-dards. For dimensions of quality where standards are moredifficult to identify, such as continuity of care or accessibility, aquality assessment describes the current level of perfor-mance with the objective of improving it.

A quality assessment frequently combines various datacollection methods to overcome the intrinsic biases of eachmethod alone. These methods include direct observation ofpatient-provider encounters, staff interview, patient focusgroup, record review, and facility inspection, among others.The assessment is often the initial step in a larger process,which may include providing feedback to health workers onperformance, training and motivating staff to undertakequality improvements, and designing solutions to bridge thequality gap.

This case study describes how five Integrated Manage-ment of Childhood Illness (IMCI) trainers and supervisorsconducted an assessment of provider knowledge and skill, tocarry out IMCI at 38 facilities in two districts in Kenya.

Assessing Health Worker Performance of IMCI in Kenya

AcknowledgmentsThe Kenyan Ministry of Health implemented the workdescribed in this case study. This case study was written byYa-Shin Lin and Paula Tavrow, with contributions from DinaTowbin. Editorial and technical review was provided by LaniMarquez and Diana Silimperi.

Recommended citationLin Y. and Tavrow P. 2000. Assessing Health WorkerPerformance of IMCI in Kenya. Quality Assurance ProjectCase Study. Published for the U. S. Agency for InternationalDevelopment (USAID) by the Quality Assurance Project(QAP): Bethesda, Maryland, U.S.A.


Background

Approximately two-thirds ofKenyans are exposed to endemicmalaria transmission, including 3.5million children under five years ofage. Since the 1980s, epidemic malariahas been increasing in frequency and severity among denselypopulated and economically important areas of Kenya’sWestern Highlands, such as Bungoma District. Malaria is themost common presenting complaint at health facilities inendemic areas and accounts for nearly 30 percent of outpa-tient visits and 20 percent of inpatient admissions nationally.1

Some 26,000 children in Kenya die each year from the directconsequences of malarial infection: about 70 children eachday.2 It is estimated that 20-25 percent of childhood deaths inBungoma district are due to malaria.3

In January 1998, the Government of Kenya and USAIDinitiated the Bungoma District Malaria Initiative (BDMI),managed by the African Medical Research and EducationFoundation (AMREF), a nongovernmental organization withheadquarters in Nairobi. The goal of BDMI is to reduce malariamorbidity and mortality through a multi-faceted approach thatincludes the improved case management of children, using

■ 1

Assessing Health WorkerPerformance of IMCI in Kenya

Q U A L I T Y A S S E S S M E N T C A S E S T U D Y


the Integrated Management of Childhood Illness (IMCI)guidelines, developed by the World Health Organization(WHO) and UNICEF. IMCI provides a structure for the compre-hensive assessment, symptom classification, the treatment ofchildren, and counseling of parents or caretakers for multiplediseases, including malaria.

Though the IMCI algorithm had not been officially launched inKenya on a national scale, training of health workers in theIMCI protocol had been carried out in Bungoma and theneighboring district of Vihiga in 1996, with technical assis-tance from the U. S. Centers for Disease Control and Preven-tion (CDC). A CDC-sponsored evaluation in 1997 indicated,however, that many trained health workers in these two largelyrural districts of western Kenya were still not performing IMCIeffectively. Focus groups revealed that many providers had infact stopped performing IMCI regularly. Under the BDMI,training in IMCI in Bungoma District resumed in 1998.

To support the BDMI, the Quality Assurance Project (QAP)developed an operations research study in the same twodistricts to test the impact of systematic team problem solvingon providers’ IMCI performance. The study entailed setting upfacility-level problem-solving teams who would be coached onhow to develop, implement, and evaluate solutions to theproblem of poor compliance with IMCI standards. Dependingon each facility’s needs, these solutions could consist ofsupporting IMCI-trained providers through improvements topatient flow, reorganization of responsibilities, on-the-jobtraining, or job aids. To better understand health workers’knowledge and attitudes about IMCI and identify the mainconstraints to improved performance, QAP proposed abaseline quality assessment of health provider performanceduring sick child consultations.

2 ■

Assessing Health Worker Performance of IMCI in Kenya ■ 3

Designing and Preparing for theAssessment

Definition of performance standards and instrumentdesign. To guide the original health worker training in 1996,WHO standards for IMCI had been adapted to conditions inwestern Kenya by the CDC. In preparing for the baselineassessment, QAP reviewed the tools CDC had developed forhealth worker assessment during supervision. QAP staffdrafted three assessment instruments:

■ An observation checklist (Appendix 1)

■ A provider knowledge and competence questionnaire(Appendix 2)

■ A facility inventory checklist

After incorporating some suggestions from AMREF staff, theinstruments were pre-tested in Bungoma district hospital withthe assistance of an IMCI trainer. Questions that were misinter-preted during the pre-test were clarified, and observationprocedures were standardized.

Selection of facilities and quality assessors. Allgovernment facilities in the two districts where at least onehealth worker was trained in IMCI were selected to participatein the assessment. To explain the goals of the assessmentand the operations research intervention, QAP gave a half-dayquality assurance awareness seminar for selected members ofthe two District Health Management Teams (DHMTs) and theheads of all 38 facilities that would be assessed. After thisworkshop, the DHMTs nominated four people who had beentrained in IMCI in 1996 to conduct the assessment.

Training of data collectors. In May 1998, the fournominees were trained to use the assessment instruments,but not all had sufficient skills in IMCI to conduct the assess-ment. QAP requested that the DHMTs nominate new asses-sors who were either IMCI trainers or supervisors. Threeclinical officers and the two district nurse supervisors wereselected. They then participated in an intensive, two-daytraining session. The training included discussion of questions,

Assessing Health Worker Performance of IMCI in Kenya4 ■

role-playing, standardization of response recording from mockinterviews, and a final pre-testing of the instruments at twofacilities. To ensure interobserver reliability, the five assessorscompleted observation forms for the same provider-clientinteraction. They then discussed their differences and reachedagreement on them. They repeated this exercise until nearly90 percent interobserver reliability was achieved.

Collecting Quality Data

In June 1998, immediately after the training session, theassessment was launched. The data collectors spent 14 daysin Bungoma and 10 days in Vihiga. For the first two days ateach site, they worked in two-person teams to improveinterobserver reliability. The assessors observed 74 healthworkers’ IMCI performance in clinical sessions with 739children at 38 facilities (21 in Bungoma and 17 in Vihiga).

Each health worker was observed handling 10 sick childrenbetween the ages of two months and five years. Because ofheavy patient utilization due to the malaria season, datacollection in each facility was accomplished in a single day inmost cases. On average, the five assessors observed fourproviders at two or three facilities each day. After the 10observations, the assessors interviewed each provider using aquestionnaire on IMCI knowledge and attitude, and thenconducted a rapid inventory of the availability of necessarydrugs and supplies for IMCI. Both the questionnaire and theinventory each took about 10 minutes to complete, whileobservations took two to three minutes longer than theduration of the consultations.


Figure 1. Providers’ Performance of IMCI in 1998,by District and Year of Health Worker Training

* Dates indicate when providers were trained; N refers to the number of providers

100

80

60

40

20

0Correct

classification

Per

cent

of c

hild

ren

who

rece

ived

Correcttreatment

Sufficientcounseling

Bungoma, Trained in 1998 (N=15)*

Bungoma, Trained in 1996 (N=29)

Vihiga, Trained in 1996 (N=29)

Results: Analyzing and Using AssessmentFindings

Compilation of the quality of care data. Once the assess-ment was completed, AMREF personnel entered the data and ajoint AMREF and QAP team cleaned and analyzed them.

Findings on compliance with IMCI standards. Theobservation checklist data revealed serious deficiencies inhealth workers’ compliance with IMCI. For example, providersdid not check for all danger signs in over one-third of thechildren observed, and they did not check for all majorsymptoms in two-thirds of the children. Less than 10 percentof the children received a complete assessment, in which allthe assessment steps in the IMCI guidelines were followedaccurately. Less than one-fifth were correctly “classified” (IMCIterminology meaning diagnosed). In addition, only 60 percentof sick children received correct treatment (Figure 1).


Figure 2. Health Worker’s Competence in IMCI AssessmentMethods, by District and Year of Training

Findings on health worker IMCI knowledge. Theprovider questionnaire data indicated that providers’ knowl-edge of IMCI differed considerably by district (see Figure 2),regardless of whether they had received training in 1996 or1998.

* Dates indicate when providers were trained; N refers to the number of providers

(These results are from Appendix 2, Question 5.)

100

80

60

40

20

0Stiff neck

Per

cent

of p

rovi

ders

who

wer

e co

mpe

tent

Edema Severewasting

Skin pinch Chestindrawing

Competent inall methods

Bungoma, Trained in 1998 (N=15)*

Bungoma, Trained in 1996 (N=29)

Vihiga, Trained in 1996 (N=29)

In addition, providers expressed high frustration performingIMCI, which they found to be very complex and time-consum-ing. More than half said that IMCI took too long or made theirworkload too heavy. In addition, more than four out of fiveproviders said that drugs and supplies were often unavailable(see Figure 3).


Presentation of findings. In August 1998, QAP andAMREF conducted a one-day workshop to present thefindings to the entire DHMT from each district, some 25participants in all. Data were aggregated by district and notpresented by facility, in order to focus participants’ attention ondistrict-level opportunities for improvement. The findingsgenerated considerable discussion among the DHMT mem-bers.

Identification of opportunities for problem solving.After the presentation of findings, the DHMT members dividedinto two groups by district to discuss district-level strategiesand write brief action plans to address issues raised. Whenboth groups reconvened, the top three strategies put forwardwere:

■ Intensify supervision of health workers

■ Redirect some of the cost-sharing monies to IMCI suppliesand equipment

■ Ensure that all facilities have oral rehydration therapy (ORT)corners†

Figure 3. Top Five Difficulties Performing IMCI Reported byHealth Workers*

* These results are from Appendix 2, Question 6 (open-ended).

† An ORT corner is an area set aside in a facility where caretakers canadminister ORT to their dehydrated children over several hours.

20%

22%

29%

40%

82%

200 40 60 80 100

Drugs or supplies for IMCI not available

Workload too heavy

Takes too long

Mothers want injections

No support or help from other staff

Percent of providers who reported


Supplemental data collection. To gain additional insightinto the assessment findings, QAP and AMREF convened afocus group with 11 IMCI-trained providers in Bungomadistrict. Providers were asked to detail the problems theyfaced in trying to perform IMCI. Half acknowledged that theycould not perform IMCI regularly. They also cited numerousIMCI steps often skipped by providers, such as assessing thechild’s nutritional status. Providers also suggested strategiesto save time in order to make the IMCI guidelines morefeasible.

Follow-up with systematic team problem solving. InOctober 1998, 20 DHMT members, including IMCI supervi-sors and trainers, attended a three-week quality assurancecoach training course. During the training, DHMT memberswere presented with the baseline assessment results for eachfacility for use in setting up problem-solving teams. Over thenext six months, the DHMT coaches created teams at 23facilities and began coaching them in problem solving. As partof the week-long onsite training for each problem-solvingteam, the coaches helped facility staff understand theassessment findings.

All teams initially concentrated on the problem of healthworker implementation of the IMCI algorithm: specifically, thatall children were not receiving a full IMCI assessment andappropriate treatment. Ensuring that health workers regularlyuse the IMCI algorithm was the first step in improving thequality of health worker performance. For the next year,AMREF monitored the teams and gave guidance to thecoaches.

In March 2000, a follow-up assessment was conducted tomeasure the impact of the systematic team problem-solvingwork on IMCI performance. The facilities with teams werecompared to those without teams. The assessment resultsshowed that facilities with teams had significant improvementin IMCI case management when compared to facilities withoutteams. The difference was greatest among the teams withhigh problem-solving ability.


Quality Assessment Insights

This case study illustrates how quality assessments canprovide the impetus for district and facility-level problem-solving. Providers had received training in IMCI guidelines,but they were not implementing them consistently oraccurately, for reasons that were not well understood.Systematic team problem solving was the interventionchosen to increase compliance with IMCI guidelines,since it had been successful in other contexts. Qualityassessment provided a baseline, against which the effectof the teams’ interventions could be measured. In theprocess, the following lessons were learned:

Participation of the DHMT in the assessmentprocess ensures local commitment to improvingIMCI performance. The DHMTs participated from startto finish in every step—from the planning and pre-testingstage, through training as assessors, launching andconducting the assessment, discussing and analyzing theassessment data, addressing issues raised, settingpriorities, and using problem-solving techniques to findappropriate solutions.

Quality assessment focuses providers’ attentionon their skills and makes them feel that theirwork is important, especially at small, remotefacilities. The assessment was the first time that healthfacilities in the two districts had received feedback ontheir performance. Several providers mentioned that theyappreciated having their performance assessed andhoped assessments would occur regularly.

The methodology of quality assessment needs tobe continually adapted and refined. For the follow-up assessment in Kenya, changes were made to stream-line the assessment, including reducing the number ofcases observed to five per provider and simplifying theobservation form so assessors could more easily observe


and record their observations. Another improvement wasto initiate data entry during the same period as datacollection, so that results were obtained in no more thantwo days after collection was completed. As with the firstassessment, skilled IMCI trainers were used as assessorsto ensure data quality.

End Notes

1. Bungoma District Malaria Initiative Annual Report, 1999.

2. Wellcome News, http://www.wellcome.ac.uk/en/1/biosfginttrpinfbig.html, 1999.

3. Mid-term Review of Bungoma District Malaria Initiative(BDMI), June 2000.


APPENDIX 1

Page 1

IMCI Performance Process Assessment(Pages 1 and 2 of 4)


IMCI Performance Process Assessment(Pages 1 and 2 of 4)

APPENDIX 1

Page 2


Rapid Provider Competence AssessmentAPPENDIX 2

14 ■

■ 13

Assessing Health WorkerPerformance of IMCI in Kenya:Summary

In mid-1998, five district-level clinical trainers/supervi-sors conducted a baseline assessment of 74 providers’performance of the Integrated Management of ChildhoodIllness (IMCI) algorithm. The assessors observed the pro-viders in clinical consultations with 739 children at 38facilities in two districts of Kenya (21 facilities in Bungomadistrict and 17 in Vihiga district), with technical guidancefrom the Quality Assurance Project (QAP). Local DistrictHealth Management Teams (DHMTs) were involvedthroughout the process—from selecting the assessors,participating in the initial training, and collecting data, tointerpreting and using results for district planning. Thisparticipatory approach built local commitment to improveIMCI compliance. It also marked the first time that healthworkers saw how their facilities performed in compari-son with others.

assessing health worker performance of imci in kenya

Documents