institutionalizing and sustaining the lot quality ... 8-14.pdf ·...

57
STAR-E LQAS Institutionalizing and Sustaining the Lot Quality Assurance Methodology in Uganda: How close are we? June 26, 2014 Jerald Hage Director, Center for Innovation University of Maryland College Park, MD 20742 Joseph Valadez Professor of International Health Liverpool School of Tropical Medicine Liverpool, UK With Charles Nkolo Liverpool School of Tropical Medicine

Upload: others

Post on 04-Aug-2020

4 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

STAR-E LQAS

Institutionalizing and Sustaining the Lot Quality Assurance Methodology in

Uganda:

How close are we?

June  26,  2014  

 Jerald  Hage  

Director,  Center  for  Innovation  University  of  Maryland  College  Park,  MD  20742  

 Joseph  Valadez  

Professor  of  International  Health  Liverpool  School  of  Tropical  Medicine  

Liverpool,  UK    

With  Charles  Nkolo  

Liverpool  School  of  Tropical  Medicine  

   

Page 2: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

                                     

 

 

 

 

                           Copyright  ©  Liverpool  School  of  Tropical  Medicine  2014  All   rights  are  reserved.  This   report  and  any  attachments   to   it  may  be  confidential  and  are   intended  solely   for   the  use  of   the  organisation  to  whom  it  is  addressed.  No  part  of  this  report  may  be  reproduced,  stored  in  a  retrieval  system,  or  transmitted  in  any   form  or  by  any  means,  electronic,  mechanical,  photo-­‐copying,  recording  or  otherwise  without  the  permission  of  Liverpool  School   of   Tropical  Medicine.   The   information   contained   in   this   report   is   believed   to   be   accurate   at   the   time  of   production.  Whilst  every  care  has  been  taken  to  ensure  that  the  information  is  accurate,  Liverpool  School  of  Tropical  Medicine  can  accept  no   responsibility,   legal   or   otherwise,   for   any   errors   or   omissions   or   for   changes   to   details   given   to   the   text   or   sponsored  material.  The  views  expressed  in  this  report  are  not  necessarily  those  of  Liverpool  School  of  Tropical  Medicine.  

Page 3: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

TABLE  OF  CONTENTS  

Acronyms  ..................................................................................................................................  4  

Acknowledgements  ...................................................................................................................  5  

Executive  Summary  ...................................................................................................................  6  

Background  ...............................................................................................................................  8  

Research  Methodology  ...........................................................................................................  11  

Measurement  Section  .........................................................................................................  13  

Measuring  Institutionalization  .........................................................................................  14  

Measuring  Challenges  ......................................................................................................  16  

Measuring  Learning  .........................................................................................................  16  

Measuring  Sustainability  and  Challenges  ........................................................................  19  

Research  Findings  ....................................................................................................................  21  

The  Degree  of  Institutionalization  .......................................................................................  21  

Challenges  ...............................................................................................................................  27  

Learning  and  Challenges  ......................................................................................................  31  

Sustainability  and  Challenges  ..............................................................................................  32  

Recommendations  ..................................................................................................................  35  

Short-­‐Term  Recommendations  ...........................................................................................  35  

Long-­‐term  Recommendations  .............................................................................................  36  

Appendices  ..............................................................................................................................  37  

Appendix  A:  Discussion  of  the  Research  Design  ..................................................................  37  

Appendix  B:    DHO  and  Assistant  DHO  Interview  Questionnaires,  Focus  Group  Schedules,  and  Surveys  for  Supervisors  and  Data  Collectors  ................................................................  46  

Appendix  C  Monitoring  Report  on  LQAS/HFA  survey  data  use  by  districts  .........................  55  

References:  .............................................................................................................................  57  

 

 

Page 4: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

ACRONYMS  

 CAO       Chief  Administrative  Officer  CDC       Center  for  Disease  Control  CSF           Civil  Society  Fund  CSO       Civil  society  organizations  DHO       District  Health  Officer  GoU       Government  of  Uganda  HF       Health  Facilities  HMIS       Health  Management  Information  System  INGO       International  non-­‐government  organizations    LSTM       Liverpool  School  of  Tropical  Medicine  LQAS       Lot  Quality  Assurance  Sampling  MCH       Mother  and  child  health  MoH       Ministry  of  Health  MoLG       Ministry  of  Local  Government  MSH       Management  Sciences  for  Health  NGO       Non-­‐government  organizations  NLF       National  LQAS  Facilitators  OVC       Orphans  and  Vulnerable  Children  PACE       Program  for  Accessible  Health  Communication  SDS       Strengthening  Decentralization  for  Sustainability  TASO       The  Aids  Support  Organization  TWG       Technical  Working  Group  VHT       Village  Health  Teams    USAID       United  States  Agency  for  International  Development  

 

 

 

 

 

 

 

 

 

 

 

 

 

Page 5: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

ACKNOWLEDGEMENTS  

LSTM  would  like  to  thank  all  of  the  many  stakeholders  from  the  nine  districts    

In  particular  they  would  like  to  single  out  Esther  Sempiira,  the  national  coordinator  of  STAR-­‐E   LQAS   and   Stephen   Lwanga,   MSH   Country   Representative   and   Director   of   Management  Sciences  for  Health,  Uganda.    

Special   thanks   also   goes   to   the   7   District   Health   Officers   and   6   Assistant   DHOs   or   their  substitutes  and  the  69  supervisors  and  data  collectors  of  all  nine  participating  districts  who  took  the  time  to  answer  the  questions  and  to  discuss  possible  strategies  of  sustainability.  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Page 6: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

 

EXECUTIVE  SUMMARY  

 

The  Liverpool  School  of  Tropical  Medicine  requested  that  a  research  study  of  the  extent  of  institutionalization,  the  challenges  remaining,  and  the  potential  for  sustainability  be  made  of  the   use   of   Lot   Quality   Assurance   Sampling   (LQAS)   in   Uganda   since   2009   with   funding   by  USAID  called  STAR-­‐E  LQAS.     In  addition,   the  Management  Sciences   for  Health   (MSH)  asked  that  the  study  also  examine  how  LQAS  has  impacted  on  the  effectiveness  in  the  delivery  of  social  services.     In  particular,  they  desired  recommendations  that  could  be  implemented  in  the  remaining  nine  months  of  the  current  contract  so  as  to  improve  service  delivery.  

 

With   these   instructions,   a   research   study   of   nine   districts—Bushenyi,   Mbarara,   Kabale,  Kabarole,  Hoima,  Kamwenge,  Mbale,  Kaberamaido,  Tororo-­‐-­‐within  Uganda  was  undertaken  during  the  month  of  May  2014.    The  nine  districts  were  selected  to  maximize  variation  in  the  percentage  of  births  delivered  in  a  health  facility  within  three  regions:    eastern,  western,  and  southwestern.     The   reasons   for   the   specific   research   design,   its   advantages   and  disadvantages   are   explained   in   the   research  methods   section   and   Appendix   A.     The  main  reason   for   a   comparative   case  method   research   design   is   to   be   able   to   relate   the   use   of  LQAS   to   the   effectiveness   of  maternal   health.     Interviews  were   conducted  with   7   District  Health   Officers   (DHO)   and   6   Assistant   DHOs   to   cross-­‐check   on   institutionalization,  challenges,   learning   and   sustainability.     Similarly   31   supervisors   and   38   data   collectors  participated  in  separate  focus  groups  for  the  same  reason  as  well  as  filled  out  surveys.  

 

Institutionalization  was  measured  by   two   indices  each  composed  of   four  components:     (1)  the  degree  of  training  and  experience;  and  (2)  the  degree  of  coordination  and  control  of  the  LQAS  process.    On  the  first  index,  five  districts  scored  75%  or  higher  and  on  the  second,  all  but  two  districts  scored  this  high.    The  second  index  is  the  more  critical  of  the  two  because  it  measures   how   much   the   results   of   LQAS   are   employed   in   planning   and   budgeting.   The  second   index  demonstrates   that   for   coordination  and  control  more   institutionalization  has  occurred   than  with   training   and   experience,   indicating   that   the   STAR-­‐E   LQAS   intervention  has  been  particularly  successful   in   integrating  monitoring  and  evaluation   into  planning  and  in   precipitating   intervention   strategies.     However,   a   contrast   between   Hoima   and  Kabermaido  indicates  that  it  is  preferable  to  support  at  minimum  two  rounds  of  LQAS  data  collection.     In   Uganda   13   districts   have   had   only   one   round   and   32   districts   have   not   yet  been  introduced  to  LQAS.    

 

To  answer  the  question  of  whether  LQAS  has  had  an  impact  on  the  effectiveness  of  service  delivery   the   two   indices   of   institutionalization   were   added   together   and   their   rank   order  correlated   with   the   rank   order   of   the   percent   of   mothers   having   deliveries   in   a   health  facility;   the   Spearman   Rho   is   .45.     To   substantiate   the   causal   connection,   a   number   of  examples  were   taken   from  the   interviews,  especially  with   the  DHO  and   the  Assistant  DHO  (MCH)  or  their  substitutes  (see  Table  A.2),  on  how  LQAS  was  used  to  improve  social  services.      

 

In  addition,  an   index  of  availability  of  health   facilities   (HF)  was  constructed  on  the  basis  of  the  number  of  HF  s  per  1000  population  located  in  a  100  square  kilometre  catchment  area.    The   Spearman   Rho   correlation   between   this  measure   and   delivery   of   births   in   HFs   is   .75.    

Page 7: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

However,  Bushenyi,  which  cored  100%  on  both   indices  or  high   institutionalization;  while   it  has  a  relative  low  number  of  HFs  it  does  have  a  very  high  percentage  of  births  in  HFs.    Thus,  it  represents  a  model  for  how  LQAS  can  be  used  to  improve  social  services.  

 

The   usual   challenges   are   the   problems   of   transportation,   lack   of   rain   gear,   insufficient  budgeting  for  translations,  and  allowances  for  over  night  stays  in  parishes  and  sub-­‐districts  that  are  difficult  to  reach  during  the  process  of  data  collection.    The  more  unusual  challenge  is  the  need  to  continually  train  new  appointees  to  the  positions  of  DHO  and  Assistant  DHO.    Those  with  no  exposure  have  little  interest  in  this  methodology.  

 

Learning  was  measured   in  multiple  ways.     LQAS  provides   a  major  mechanism   for   learning  about  sub-­‐districts  that  are  not  performing  well  and  adjusting  strategies  to  improve  service  delivery.    But   the  DHOs  and  the  Assistant  DHOS   (MCH)   reported   learning   in  other  ways  as  well,  especially  about  maternal  and  child  health.    Thus,  both   learning  mechanisms  account  for   the   improvement   in   services.    With   the   available   data   is   not   possible   to   estimate   the  relative  importance  of  either  mechanism.  

 

Considerable  support  exists  for  two  strategies  of  sustainability  from  the  health  management  team,   the   supervisors,   and   the   data   collectors.     The   first   strategy   is   to   integrate   the   data  collection   into   the  routine  work  of   the  supervisors  and  the  data  collectors.     In   the  process  some   of   the   challenges   in   data   collection   can   be   solved   including   allowing   more   time   to  collect   the   data   and   better   scheduling,   providing   better   transportation,   and   conducting  more  translations.    The  second  strategy   is   to  transfer  the  knowledge  about  LQAS  from  the  districts   that   scored   high   on   institutionalization   to   those   that   scored   low   or   have   no  exposure.     The  National   LQAS   Facilitators   (NLFs)   represent   an   elite   corps   for   this  mission.  Beyond  this,  the  support  for  LQAS  on  the  part  of  the  health  management  team  is  extremely  strong,  when  they  have  received  reports  and  have  been  involved  in  the  process.        

   

The  short-­‐term  recommendations  for  the  next  nine  months  are:  

1.    Attempt   to   include  members  of  other  ministries   in   the  Technical  Working  Group  being  formed  by  Patrick  Mutabwire,  Director  of  Local  Government  Administration.  

2.    Support  a  second  round  of  LQAS  data  collection   in  the  13  districts  that  so  far  have  had  only  one  round.  

   

The  long-­‐term  recommendations  for  the  period  afterwards  are:  

1.    Hold  planning  committees  among  districts  with  similar  sets  of  challenges  to  discuss  how  best   to   integrate   the   data   collection   process   in   the   on-­‐going  work   of   the   supervisors   and  data   collectors   in   those   districts   to   improve   quality   of   data   collection,   lower   stress   and  develop  creative  solutions  to  various  challenges.      

2.    Begin  to  train  new  data  collectors  and  supervisors  in  those  districts  that  as  yet  have  not  had  any  exposure  to  LQAS.      

3.       Every   several   years   train   the  newly  appointed  DHOs  and  Assistant  DHOs   (MCH)   in   the  LQAS  process  so  that  the  health  management  team  becomes  committed  to  this  process  and  uses  the  data  in  their  planning.    If  resources  permit  include  other  district  managers  to  build  support  for  LQAS.  

Page 8: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

8  

BACKGROUND  

USAID  has  been  supporting  Uganda  since  2009,  to  aid  districts  to  use  Lot  Quality  Assurance  Sampling  (LQAS)  to  manage  and  improve  their  health  programs.  This  work  is  undertaken  by  a   Management   Sciences   for   Health   (MSH)   managed   project   called   STAR-­‐E.   The   LQAS  component  of   the  STAR-­‐E  project,   STAR-­‐E   LQAS,   is   implemented   jointly  with   the   Liverpool  School  of  Tropical  Medicine  (LSTM).  Since  the  time  they  first  started  working  in  10  districts  during  2009,  now  close  to  80%  of  the  112  districts  have  used  LQAS  at  least  once.    Given  this  effort,  USAID  and  MSH  have  been  interested  in  assessing  what  has  been  accomplished,  and  the  extent  to  which  LQAS  has  been  accepted  in  Uganda.          To  make  this  evaluation,  we  asked  the  following  three  questions:    

(1) How  much  has  the  LQAS  methodology  been  institutionalized?  (2) What  challenges  remain  to  increase  the  degree  of  institutionalization?  (3) What   are   some   pragmatic   strategies   for   ensuring   sustainability   of   this  

methodology?    

The  answers  to  questions  2  and  3  can  provide  a  road  map  for  the  remaining  nine  months  of  the  current  project  extension,  and  also  advise  USAID  and  the  Government  of  Uganda  (GoU)  about  future  next  steps  in  supporting  LQAS  work.      

The  definition  of  institutionalization  that  guides  this  research  project  is:  

Institutions   are   clusters   of   norms   with   strong   but   variable   mechanisms   of  support   and   enforcement   that   regulate   and   sustain   an   important   area   of  social  life.  

Specifically,  norms  are  nothing  more   than  the  “rules  of   the  game”,  e.g.  how  often   is  LQAS  conducted   and   what   procedures   are   employed   for   the   presentation   of   results.   The   most  important   clusters   of   norms   are   jobs   or   who   does   what,   rules   about   supervision   and  enforcement  of  various  regulations  about  what  should  be  done.          In  addition  to  these  questions,  we  considered  another  three  queries  that  are  important  for  the   country   of   Uganda.     LQAS   is   a   means   to   an   end,   namely,   the   improvement   in   social  services   for   the   targeted   populations,  whether   they   be  men  with   HIV,   pregnant  mothers,  young   children   and   their   mothers,   or   orphaned   and   vulnerable   children   (OVC).     It   is  important   to   keep   a   clear   distinction   between   the   effectiveness   of   LQAS   data   collection  procedures  and  their  impact  on  the  quantity  and  quality  of  social  services.  We  can  examine  that  issue  by  observing  how  much  the  data  are  actually  used  in  planning  and  budgeting  for  social   services   to   help   the   people   of   Uganda.   This   distinction   is   an   important   one.     The  reason   why   is   that   one   wants   to   avoid   the   error   of   Dr.   Freud   who   thought   awareness   is  enough  and  individuals  will  change  their  behavior  once  they  understand  their  problems.    As  

Page 9: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

9  

is   well   known,   this   is   not   always   true   and   one   might   add   it   is   even   more   difficult   for  organizations,  versus   individuals,  to  change  their  programs,  policies,  and  business-­‐as-­‐usual.  Indicating   poor   performance   does   not   necessarily   produce   changes   in   organizational  strategies.    Part  of  the  reason  may  simply  be  because  a  root  cause  is  not  easily  changeable,  e.g.   the  absence  of  health   care   facilities.     In   these   circumstances,   LQAS   should  at   least  be  given  credit  for  the  recognition  of  the  problem.    The  opposite   is  also  the  case.    Sometimes  improvements  are  due  to  reasons  other  than  the  use  of  LQAS.    One  needs  to  avoid  assigning  credit  when  the  improvement  in  effectiveness  is  a  consequence  of  the  intervention  of  some  external  agent,  such  as  the  Baylor  Medical  School    (working  in  Uganda)  or  a  training  project  of  the  central  government.  Therefore,  identifying  when  LQAS  has  had  an  impact  and  when  it  has  not  is  an  important  component  of  this  evaluation.    Thus,  the  parallel  set  of  questions  for  the  benefit  of  the  Ugandan  people  are:    

(1) What   impact   has   LQAS   had   on   the   effectiveness   in   the   delivery   of   social  services?  

(2) What  challenges  remain  to  improve  the  impact  of  LQAS  on  service  delivery?  (3) What   are   some   pragmatic   strategies   for   increasing   the   impact   of   LQAS   on  

service  delivery?  

In   answering   these   questions,   it   is   important   to   understand   the   historical   experience   of    Uganda   with   LQAS.   Prior   to   STAR-­‐E,   MSH   has   had   a   long   history   for   supporting   health  services  in  Uganda  while  Prof.  Joe  Valadez  (currently  in  LSTM)  had  previously  introduced  this  methodology   (2003)   into   30   districts   through   the   Uganda   AIDS   Council   for   assessing  AIDS/HIV  prevention  and  control  through  the  World  Bank.  In  a  commendable  and  prescient  innovation  encouraged  by  USAID,   eight  major  USAID  projects   (STAR-­‐E,   STAR-­‐EC,   STAR-­‐SW,  STRIDES,  NU-­‐HITES,  SUNRISE,  CSF  and  SMP),  despite  their  quite  disparate  objectives  agreed  to  use  the  same  set  of  59  indicators  to  assess:  (1)  reproductive  health  and  family  planning;  (2)  child  health;   (3)  malaria;   (4)  sexually   transmitted  diseases   including  HIV  counseling  and  prevention;   (5)   tuberculosis;   (6)  water   and   sanitation;   (7)   nutrition;   (8)   orphans   and  other  vulnerable   children;   and   (9)   education.     The   seven   projects   involved   four   distinctive  implementing  partners:    MSH,  JSI  Research  and  Training,  Inc.,  Elizabeth  Glaser  Pediatric  AIDS  Foundation,  Plan   International,   Inc.   and   Johns  Hopkins  University.    MSH/LSTM  partnership  provided   technical   assistance   in   the   use   of   the   LQAS  methodology   to   these   partners.     In  addition   to   these  projects,  USAID  has   funded  the  SUNRISE  Project   to  help  OVC   left  behind  because   their   parents   have   died   of   AIDS,   and   Strengthening   Decentralization   for  Sustainability   (SDS)   Project   ,   another  HIV/AIDS  project   currently  working   in   35   districts.   In  addition,  many   international   non-­‐government   organizations   (INGOs)  work   in   tandem  with  local  non-­‐government  organizations  (NGOs);  these  organizations   include  AfriCare,  AidChild,  and  the  Program  for  Accessible  Health  Communications  and  Education  (PACE),  and  The  AIDS  Support  Organization  (TASO).    Further  LQAS  support  has  come  from  UNICEF  who  is  working  with   Child   Fund   International   and   LSTM   to   use   LQAS   together   with   other   management  methods  to  support  decision-­‐making  for   Integrated  Community  Case  Management  of  child  health   (21   Districts);   and   lastly,   UKAID   through   the   Department   for   International  

Page 10: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

10  

Development   (DfID)   support   use   of   LQAS   in   the   Acholi   and   Lango   Regions   to   assess   the  effectiveness  of  Results  Based  Financing.      

At   various   times   since   2009,   other   international   organizations   have   had   an   impact   on   the  effectiveness  of  the  delivery  of  services  such  as  the  Baylor  College  of  Medicine  with  support  of  Centers   for  Disease  Control   (CDC)   in   the  U.S.  and   its  program   for   training  village  health  team  members   (VHTs)   that  proved  highly  effective   in   the  districts   in  which   they  operated.    Another   important   international   consortium   is   the  Civil   Society   Fund   (CSF),  which   receives  funding   from  six   countries.     This   fund   is  primarily   engaged   in   various  projects  designed   to  make  local  institutions  stronger  relative  to  HIV/AIDS  prevention  and  helping  OVC.    This  Fund  asks   for  proposals   from  civil   society  organizations   (CSOs)  and  once  funded  one  of   the  nine  districts  selected  for  this  study.    Finally,  the  government  of  Uganda  also  has  introduced  new  policies   that   impacted  on   the  delivery   of   services.   Thus,   this   combination  of   interventions  funded   by   USAID,   other   international   agencies,   and   the   Uganda   government   is   quite  complex.    Due   to   the  complexities   concerning   the  number  of  actors   involved   in   this  work,  this   assessment   will   have   a   challenge   to   sort   out   the   various   intervening   factors.  Nevertheless,  as   the   focus  of   this  work   is   to  understand   the   institutionalization  of  LQAS   in  Uganda  (rather  than  in  attributing  institutionalization  to  any  particular  donor),  our  principal  responsibility  here  is  to  assessing  the  sustainability  of  LQAS  in  Uganda.  

 

The  report  has  four  sections.    The  first  describes  the  research  methodology  while  the  second  focuses   on   the   critical   problem   of   how   to   measure   institutionalization   and   sustainability,  hardly  easy  tasks.    Given  space  limitation,  additional  detail  is  provided  in  Appendices  A  and  B  respectively.     Section   three   contains   the   research   findings   about   how   much  institutionalization   has   occurred   and   strategies   for   sustainability   followed   by   the  recommendations  in  the  fourth  section.    Throughout,  this  assessment  has  been  designed  to  represent   a   model   for   what   USAID   can   accomplish   in   other   evaluations,   and   in   other  countries   by   supporting   other   countries   to   institutionalize   LQAS.     In   particular,   it  demonstrates  the  advantages  of  studying  institutionalization  and  sustainability  at  the  district  level.  

 

Page 11: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

11  

 

RESEARCH  METHODOLOGY  

 

The  research  strategy  for  answering  these  six  questions   is   to  focus  at  the  district   level  and  their  management  by  local  government  authorities  in  contrast  to  much  of  the  literature  that  discusses   institutionalization   and   sustainability   at   the   central   government   level   (Max   de  Xaxas  and  Vogel,  2007;  Stash  et  al.,  2012;  USAID,  2012).  The  projects  unfolded  at  the  district  level   and   therefore,   it   is   only   at   this   level   can   these   questions   be   adequately   answered.  Furthermore,   this   emphasis   is   consistent   with   the   GoU’s   devolution   of   authority   to   the  districts   in   the   1990s.     Given   the   concerns   about   ownership   and   social   capital   in   the  international  literature,  the  district  level  appears  to  us  to  be  the  more  appropriate  level  for  measuring  these  concepts.  

 

Another   important   reason   to  work   at   the  district   level   is   the   considerable   variation   in   the  number  of   times   that  LQAS  has  been  used   to  collect  data  at   this   level,  with  some  districts  having   experienced   this   process   five   or   six   times   (Bushenyi,   Kabale)   and   some   only   once  (Hoima  and  Kaberamaido)  (See  Table  A.  1,  for  a  complete  listing  of  all  districts  and  years  in  which   data   was   collected   before   2013).   Certainly   understanding   challenges   to  institutionalization   requires   interviewing   people   directly   about   their   experiences.   Some  districts  are  largely  rural  and  others  urban;  some  districts  have  fairly  adequate  coverage  with  health   facilities   (HF)   (e.g.   the   south-­‐western   region),   and   others   do   not   (e.g.,   northwest).    Some  districts  have  VHT  or  village  health  team  members  (hereafter  VHTs)  with  better  basic  education  than  do  others.  When  selected,  the  VHTs  need  to  have  some  customized  training  and  this  is  in  many  cases  donor  supported,  e.g.,  by  the  Baylor  College  of  Medicine  and  U.S.  CDC   among   others.   Both   the   availability   of   HFs   and   the   quality   of   training   of   VHTs   are  highlighted  because  of   their   impact   on   the   effectiveness   in   the  delivery  of   health   services  quite  independently  of  LQAS.  However,  the  most  important  reason  for  studying  the  district  level   is   to   determine   whether   various   strategies   for   sustainability   are   viable   and   what  challenges  have  to  be  overcome  to  sustain  this  process  of  monitoring  and  evaluation.    Again,  only  those  people  and  institutions  directly  involved  in  LQAS  can  make  these  assessments.    

 

To   capture   different   degrees   of   institutionalization   and   potentials   for   sustainability   at   the  district   level,   the   research   design   included   nine   districts   that   varied   in  mothers   accessing  maternal   health   care   services;   specifically,   we   assessed   three   districts   in   each   of   three  regions.    The  selection  of  districts  (see  Table  1)  is  designed  to  maximize  the  variation  within  region.  This  is  a  critical  point  because  the  conventional  wisdom  is  that  there  are  significant  differences   between   the   regions.     If   so,   then   the   question   becomes   how  well   do   districts  facing  approximately  the  same  conditions  as  reflected  in  regional  differences  utilize  LQAS  in  their  planning  and  choice  of  interventions.          We  operationalize  this  selection  by  using  one  indicator,  namely,  district  effectiveness  in  women  delivering  their  babies  in  a  health  facility;  

Page 12: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

12  

2012  was   used   as   the   index   year   as   all   districts   had   data   for   that   year.     These   data  were  obtained   directly   from   the   STAR-­‐E   LQAS   super-­‐dataset   that   includes   all   LQAS   data   from  2003-­‐2012.    

 Table  1  Percent  of  Mothers  Who  Delivered    Their  Child  in  a  Health  Facility  in  2012    District   Region   Percent  

2012  Other  LQAS  Surveys   Also  Measured    

In  2013  Bushenyi   Southwest   88.4   2011,  2010,  2006,  2003   Yes  Mbarara   Southwest   78.9   2011,  2003   No  Kabale   Southwest   70.5   2011,  2010,  2006,  2003   Yes  Kabarole   Western   80.5   2011,  2004   No  Hoima   Western   63.7   2004   No  Kamwenge   Western   56.8   2004   Yes    Mbale   Eastern   67.7   2011,  2010,  2006,  2003   Yes  Kaberamaido   Eastern   64.2   2004   No  Tororo   Eastern   50.9   2011,  2004   No  

 

The   nature   of   this   research   design,   a   comparative   case   study,   has   both   advantages   and  disadvantages.     It   is   perhaps   ironic   to   evaluate   a   method   such   as   LQAS   that   places   a  premium  on  random  selection  with  a  design  where  districts  have  been  chosen  purposively.  The  justification  for  this  procedure  is  the  need  not  only  to  answer  the  first  three  questions  indicated   above   specifically   related   to   LQAS,   which   could   be   examined   with   a   stratified  random  design  of  districts  classified  by  region  and  number  of  LQAS  visits.    The  justification  of  the  purposive  sample  was  to  connect  the  second  set  of  questions,  impact  on  health  services,  with   the   use   of   LQAS.   It   is   for   this   reason   that   this   comparative   case  method   design  was  utilized   and   in   the   jargon   of   social   research,   sampling   occurred   by   using   the   dependent  variable.     This   design   is   based   on   the   assumption   that  potentially   different   levels   of   LQAS  institutionalization  can  explain  variation  in  percent  of  births  in  hospitals  reported  in  Table  1.      

 

The   data   collection   for   this   study   consists   of   two   interviews   and   two   focus   groups   at   the  district   level.     A   basic   principle   was   to   attempt   to   obtain   information   from   two   different  sources   for   each   major   concept.   For   example,   the   DHO   was   interviewed   about   learning  relative   to   two   separate   indicators   of   safe  motherhood   (antenatal   visits   and   delivery   in   a  hospital)  while  the  Deputy  DHO  was  asked  primarily  about  learning  relative  to  two  indicators  of  child  health  (a  mother’s  treatment  seeking  behavior  for  her  sick  child  and  immunization).    In   some   districts,   the   DHO   or   the   Deputy   was   absent   and   in   others   they   were   too   busy  preparing  reports,  participating  in  meetings  or  absent  on  trips.    Wherever  possible  we  tried  to  conduct  telephone  interviews  with  the  managers  who  were  not  present  on  the  day  of  our  visit.     Also,   at   times  we   selected   knowledgeable   substitutes   (health   educators)   to   ask   the  same  battery  of  questions  (see  Table  A.2).  Seven  DHOs  and  six  Assistant  DHOs  participated  

Page 13: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

13  

in   the   study.   In   addition,   wherever   possible,   documentation   of   the   impact   of   LQAS   on  planning   was   secured   to   crosscheck   and   supplement   the   responses   received   in   the  interviews.    Separately  31  supervisors  and  38  data  collectors  (see  Table  A.2)  participated  in  focus   groups   about   various   challenges   and   strategies   for   sustainability.   Again,   both   were  asked  many  of   the   same  questions   to  obtain   a   crosscheck  on   the   information.   The  use  of  focus  groups  was  necessary  because  there  was  not  enough  time  to   interview  three  to   five  members  in  each  group  separately.    Individual  data  was  obtained  by  asking  each  supervisor  and  each  data  collector  to  fill  out  a  short  survey  about  their  position  within  health  districts,  their  responsibilities,  and  their  level  of  LQAS  training.    The  classifications  of  their  major  work  responsibilities   are   contained   in   Table   A.3.   As   can   be   observed,   the   bulk   of   the   data  collectors   and   the   supervisors   worked   either   in   the   health   sector   (43%)   or   in   community  development  (33%).  

 

An   extended   discussion   of   the   research   design   and   why   specific   regions   were   selected   is  contained   in   Appendix   A.     Initially   the   research   design   included   the   idea   of   contrasting  villages  by  the  extent  of  the  cooperation  among  community  members  (or,  in  other  words,  by  the  presence  of  social  capital)  since  this  is  a  hidden  resource  and  given  its  importance  in  the  international   development   literature   (Mas   de   Xaxas   and   Vogel,   2007).     But   as   the   focus  groups  unfolded  it  became  clear  that  there  was  not  much  variation  between  sub-­‐districts  or  parishes  in  the  case  of  Uganda.    But  since  this  study  is  concerned  with  a  general  framework,  it   is   important  to  consider  the   importance  of  sub-­‐unit  variation  even   if  not  relevant   in   this  country;  in  other  countries  we  have  found  social  capital  to  be  an  important  factor  (Valadez,  Vargas  and  Hage;  2005).    

 

Measurement  Section    

 

Projects  have  limited  time  periods.    In  the  case  of  Uganda,  most  of  the  USAID  projects  were  initially  funded  for  a  period  of  five  years,  and  presently  some  of  them  have  been  extended  for  one  year.    The  issue  then  arises  if  USAID  funding  ceases,  whether  data  could  continue  to  be   collected  and  used   in   the  normal  planning  process.   LQAS   is   a   complex   set  of   activities,  with   different   individuals   performing   distinctive   duties;   this   complexity   is   one   reason  why  both   interviews   and   focus   groups   were   used   to   collect   the   data   as   we   needed   to   assess  individual  functions  and  then  the  different  functions  working  together.      

 

Since   institutionalization  and   sustainability   are   so   critical   to  USAID  and  other   international  funding  agencies,  considerable  time  is  spent   in  this  section  on  how  to  operationalize  these  concepts  and  measure  them,  especially  at  the  district  level.     In  addition  these  concepts  are  complex  and  difficult   to  measure   (see  Hage,   1972   for   strategies  of   turning   constructs   into  operational   indicators  and   indices).     Institutionalization   is  measured   in   two  different  ways:    

Page 14: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

14  

(1)  training  and  experience  of  local  staff;  and  (2)  more  critically,  their  coordination  and  social  control   (or   regulation)   (see   Figure   B.1).     The   problem   of   challenges   is   explored   in   several  different  ways  and  at  the  level  of  data  collection  (see  Figure  B.2)  as  well  as  in  the  impact  of  LQAS   on   the   effectiveness   of   services   (Figure   B.3).   To   allow   for   other   reasons   for  improvement   in   the  effectiveness  of  health  services,  both   the  DHO  and  the  Assistant  DHO  were  asked  how   they   learned  about  new  strategies  of   interventions   relative   to  promoting  both  maternal   and   child   health   respectively   (see   Figure   B.4).     Finally,   three   strategies   for  sustainability  were  explored  in  both  the  interviews  and  the  focus  groups  (Figure  B.5).  

 

Measuring  Institutionalization  

   

The  concrete  questions  that  we  asked  about  the  level  of  training  and  experience  of  the  data  collectors  and  of  the  supervisors  are  indicated  in  Figure  B.1.    The  central  reasoning  is  that  as  the   number   of   times   the   individuals   within   a   specific   district   have   participated   in   LQAS  training  and  in  the  collection  of  data  increases,  the  more  likely  they  are  to  be  competent  and  capable  of  continuing   if  project   funding   is  no   longer  available.     In  addition,  the  supervisors  were  asked  to  evaluate  the  competence  of  their  data  collectors.  Following  this  question,  we  asked   the   supervisors   if   they   felt   that   the   data   collectors  were   capable   of   collecting   data  without  supervision.    For  the  supervisors,  we  asked  what  their  duties  are  as  a  check  on  their  knowledge   level.   In   addition   to   the   supervisors,   there   is   a   distinctive   category   called   the  National  LQAS  Facilitator  (NLF).    No  separate  interview  was  developed  for  this  category  but  several  were  involved  in  the  focus  groups  for  the  supervisors.  

 

Both   the   DHO   and   the   Assistant   DHO  were   also   asked   about   their   exposure   to   LQAS   and  whether  they  had  participated.  Some  reported  that  they  had  been  supervisors  at  one  point  in   time.   This   turned   out   to   be   far   more   important   for   the   impact   of   LQAS   on   the  effectiveness  of  social   services   than  the   training  and  experience  of   the  data  collectors  and  supervisors.    As  is  indicated  in  the  discussion  of  challenges,  when  the  DHOs  and  the  Assistant  DHOs  have  not  been  exposed  to  LQAS  or  instructed  about  it,  they  have  little  commitment  to  it.  This  is  an  important  point  as  it  indicates  that  the  comments  we  obtained  from  active  local  authorities  has  validity,  and  they  were  not  responding  to  some  assumed  preferred  response.    

 

At  the  level  of  the  district  two  indices  were  constructed  from  the  various  indicators  reported  in   Figure   B.1.   The   first   index,   the   degree   of   training   and   experience,  which   is   reported   in  Figure   One,   has   a   maximum   score   of   12   and   is   based   on   four   indicators   each   one   being  scored   0-­‐3.     For   each   year   that   the   DHO,   Assistant   DHO,   supervisors   and   staff   who  participated  in  data  collection  one  point  is  assigned  up  to  a  maximum  of  three.    For  the  two  levels  of  district  health  management,  that  is  the  DHO  and  Assistant  DHO,  the  key  issue  was  did   they   receive   reports   and   not   whether   they   had   been   trained   in   LQAS   data   collection  

Page 15: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

15  

procedures.     If   one   or   another   of   these   levels   of   management   or   their   substitutes   was  missing  or  if  they  were  new  and  had  not  received  any  reports,  a  zero  was  assigned  because  they  had  had  no  exposure  to  LQAS.    This  was  a  problem  in  several  districts  (see  Table  A.2).    At  the  level  of  the  supervisors  and  the  data  collectors  means  were  constructed  but  in  most  cases,   there   was   little   variation   because  most   of   the   supervisors   and   data   collectors   had  participated  in  all  rounds  of  data  collection  in  that  district  since  2009.      A  few  individuals  had  also  been  involved  in  the  earlier  rounds  mentioned  in  Table  1.  

 

Institutionalization   is   not   only   a   question   of   training   and   experience.     Perhaps   the   more  important   ingredient   is  whether  coordination  and  social   control  exist  over   the  activities  or  duties   of   the   data   collectors.     A   process   such   as   LQAS   must   not   only   be   coordinated,  different   people   doing   disparate   acts   at   specific   times,   but   the   actions   of   the   individuals  performing   the   disparate   duties   must   be   controlled,   providing   some   assurance   that   they  perform  their  responsibilities  competently.      

   

Since  coordination  and  control  can  be  accomplished  via  a  number  of  mechanisms  (see  Hage,  1974),  three  specific  ones  were  measured  and  at  multiple   levels.  Unlike  the  previous  index  that   counts   frequencies,   this   second   index   includes   a   set   of   indicators   concerning   the   last  time   LQAS   data   were   collected,   whether   this   was   in   2013   or   2012.   In   the   latter   instance  there   were   some   recall   problems   in   one   district,   especially   among   the   data   collectors.    Wherever  possible  the  answers  were  checked  against  reports  to  be  certain  that  an  accurate  assessment  was  being  made.    Again,  the  index  has  four  quantitative  components:    (1)  use  of  manuals;   (2)   quality   of   data   collection   control;   (3)   report   of   the   use   of   LQAS   to   change  strategies   by   the   data   collectors   and   by   the   supervisors;   and   (4)   the   report   of   the   use   of  LQAS  results  in  planning  and  budgeting  by  the  DHOs  and  the  Assistant  DHOs.          One  point  is  given  if  the  supervisors  report  the  presence  of  a  manual  for  supervision  and  one  point  if  the  data  collectors  also  report  they  had  a  manual.    Two  points  were  assigned  for  two  different  ways  of  checking  on   the  quality  of   the  data  by   the  supervisors.      Component  3  was  worth  four  points  while  component  4  was  worth  another  four  points.  

 

The  most  important  components  in  coordination  are  whether  LQAS  results  have  been  used  to  change  health  strategies,  and  in  planning  and  budgeting  them.  The  answers  to  questions  about  this  begin  to  lay  the  groundwork  for  understanding  whether  the  LQAS  monitoring  and  evaluation  system  is  leading  to  changes  in  the  effectiveness  of  the  delivery  of  social  services.  The  focus  groups  focused  on  the  third  component  while  direct  questions  were  asked  of  both  the  DHO  and  the  Assistant  DHO  but  relative  to  different  areas,   in  the  former   instance  safe  motherhood,  and  in  the  latter  child  health.    In  the  context  of  the  interviews,  DHOs  and  the  Assistant   DHOs   volunteered   other   areas   where   the   LQAS   results   were   incorporated   into  planning.    

Page 16: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

16  

Measuring  Challenges    

   

A   second   objective   of   this   research   study,   in   the   first   three   questions   listed   above,   is   the  measurement  of  challenges  to  the   institutionalization  of  LQAS.    Several  distinctive  kinds  of  challenges   were   explored   in   the   focus   groups   and   the   interviews   with   the   DHO   and   the  Assistant  DHO  (see  question  2  for  each  group  listed  in  Figure  B.2).    The  supervisors  and  data  collectors  were  asked  about  challenges  that  they  had  in  collecting  the  data.  In  contrast,  the  DHOs  and  the  Assistant  DHOs  were  asked  about  how  LQAS  could  be  made  more  effective.      

 

In  addition,  we  enquired  about  the  normal  work  of  the  data  collectors  and  the  supervisors  in  the   surveys   handed   out   after   the   focus   group   session   was   over   (see   Table   A.3).     This  information   is   critical   because   in   fact   most   of   them   are   employed   full   time.         One   can  imagine   that   there   can   be   a   conflict   between   their   normal   work   and   the   special   task   of  conducting  LQAS  community  surveys.  This  issue  is  also  relevant  for  evaluating  the  efficacy  of  one   of   the   strategies   for   continued   to   collect   LQAS   survey   data   after   donor   agencies  withdraw  their  support.  A  second  and  less  obvious  issue  is  whether  collecting  data  across  59  indicators   posed   any   special   challenges.     This   was   a   concern   that   we   explored   on   the  recommendation   of   Esther   Sempira,   the   LQAS   national   coordinator   in   STAR-­‐E   LQAS   (see  question  3  for  the  data  collectors  and  the  supervisors  in  Figure  B.3).        

   

Finally,   it   is   worth   determining   challenges   represented   by   the   use   of   the   LQAS   data   by  various  government  ministries.    Uganda  presents  a  certain  paradox.    Since  1996  Uganda  has  been  decentralizing  decision  making  to  districts  who  have  become  key  actors  in  most  areas  of  government  and  especially  in  social  services.    Some  projects  such  as  the  Civil  Society  Fund  (CSF),  as  we  have  seen,  are  attempting  to  further  this  objective.    But  as  decentralization  has  become  highly  institutionalized  now  after  almost  twenty  years,  it  has  made  it  more  difficult  for   the   two   levels   of   government   to   interface.   Therefore,   an   important   set   of   questions  explores  whether   the   LQAS  data   at   the  district   level   is   utilized  by   the   central   government  (see  questions  3  and  4  asked  of  the  DHOs  and  the  Assistant  DHOs  in  Figure  B.2)  To  measure  this  kind  of  challenge,   the  DHOs  and  the  Assistant  DHOs  were  asked   in  their  specific  areas  whether   or   not   their   data   had   been   transmitted   to   the   central   government   and   more  importantly  if  it  had  been  utilized.  

 

Measuring  Learning  

   

While  institutionalization  and  sustainability  dominate  the  discussion  of  all  donors  working  in  developing   countries,   surprisingly   little   has   been   said   about   learning.     Yet,   learning   is   the  essential   reason   for   why   LQAS   is   instituted.     The   assumption   is   made   that   once   poor  

Page 17: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

17  

performance   is   indicated,   individuals  will   change   their   strategies  and  adopt  more  effective  interventions.    This  requires   learning  and  therefore  learning  should  be  included  among  the  major   concepts   involved   in   monitoring   and   evaluation,   that   is   asking   how   much   has  occurred.    It  is  also  a  major  paradigm  within  the  organizational  literature,  where  many  useful  ideas  can  be  borrowed  (see  Argote  and  Miron-­‐Spector,  2011).    

 

Two  alternative  explanations  for  how  the  districts  learned  and  therefore  why  they  varied  in  the   effectiveness   of   having  mothers   deliver   their   babies   in   a   health   care   facilitating  were  explored.      The  first  hypothesis  is  that  it  is  LQAS  that  makes  the  district  aware  of  the  problem  and   because   of   this   information,   new   intervention   strategies   are   developed   and  implemented   by   the   implementing   partner   in   cooperation   with   the   local   district   (see  question  three  asked   in   the   focus  groups  with  the  data  collectors  and  the  supervisors  as  a  cross-­‐check,  Figure  B.3).        Both  the  DHOs  and  the  Assistant  DHOs  were  asked   if   they  used  LQAS  results  in  their  planning.  

 

The   second   hypothesis   is   that   independently   of   LQAS,   the   districts   are   learning   and  improving  their  strategies  and  tactics  of  intervention  (see  questions  2  and  4  for  the  DHO  and  the  Assistant  DHO  in  Figure  B.4).    Of  course,  it  can  be  a  combination  of  both  of  these.  

 

A  great  advantage  of  studying  the  institutionalization  process  of  LQAS  at  the  district  level  is  the  ability  to  examine  whether  there  is  a  direct  link  between  the  presentation  of  results,  the  making   of   recommendations   to   correct   particular   deficiencies-­‐-­‐the   red   flags-­‐-­‐and   their  adoption.  This  is  the  central  argument  as  to  why  LQAS  should  be  used.    Another  advantage  of  asking  these  questions  about  presentations,  recommendations  and  changes  in  strategies  in  various  areas  of  health  care  is  that  it  is  a  further  check  on  whether  the  data  collectors  and  especially   the   supervisors   were   following   their   instructions   on   how   to   use   LQAS.     The  answers   provide   a   cross-­‐validation   of   the   information   obtained   relative   to   the  institutionalization   of   LQAS   and   especially   for   planning.   From   these   questions   several  different  kinds  of  narrative  were  developed.      

 

One  of   the  most   important  ways   in  which   learning  was  encouraged   in   the  case  of  Uganda  was   the   presentation   of   what   are   labelled   “preliminary   reports”   to   all   the   important  stakeholders   in   the   district   including   political,   religious,   donor   agencies,   the   media   and  parish   chiefs.   At   this   moment,   many   of   these   stakeholders   discovered   problems,   the   red  flags,  that  they  had  not  realized  existed      As  can  be  seen  in  the  questions  posed  to  the  focus  groups   for   the   data   collectors   and   the   supervisors,   it   was   left   open   as   to   which  recommendations  in  any  of  the  nine  areas  were  made  (see  2  asked  in  the  focus  groups  with  the   data   collectors   and   the   supervisors,   Figure   B.3).     In   contrast,   and   consistent  with   the  choice  of  safe  motherhood  and  child  health  as  major  concerns,  the  interviews  with  the  DHO  and  the  Assistant  DHO,  the  questions  focused  entirely  on  these  two  areas  (see  Figure  B.4).  

Page 18: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

18  

Establishing   the   link   between   LQAS   and   changes   in   the   effectiveness   of   social   services   is  crucial  for  the  purposes  of  demonstrating  the  utility  of  monitoring  and  evaluation.  Obviously  this  link  cannot  be  studied  at  the  central  government  level;   it  requires  visiting  a  number  of  districts  and  asking  concrete  questions  at  multiple  levels.  

   

But  health  districts,  like  any  organization,  can  change  their  intervention  strategies  for  many  reasons  other  than  simply  monitoring  and  evaluation  (see  questions  2  and  4  of  the  schedule  for  DHOs  and  the  Assistant  DHOs  in  Figure  B.4).    It  is  important  that  any  evaluation  of  LQAS  be   open   to   this   alternative   perspective   in  measuring   institutionalization   and   sustainability  and   for  multiple   reasons.     Perhaps   the  most   important   one   is   that   if   a   district   shows   the  capacity  to  continue  learning  even  if  the  source  of  the   learning   is  not  from  LQAS,   it  means  that   the  district   is  more  open  to  change  and  adaptation  and  fundamentally,   this   is  what   is  most   important   for   Uganda.   It   means   that   the   district   or   at   minimum   its   leadership   is  engaged   in  a  problem  solving  mode  and  attempting   to  overcome  obstacles.  This   is  exactly  what  Uganda  needs  to  make  progress,  not  just  in  health  care  but  also  in  other  social  service  areas.    Selecting  DHOs  or  Assistant  DHOs  that  have  this  capacity  is  perhaps  the  best  strategy  for  diffusing  best  practices  from  one  district  to  another  including  the  use  LQAS  in  planning.    

 

When  health  management  is  problem  solving  to  improve  both  maternal  and  child  health,  it  implies  that  they  are  much  more  open  to  change.    As  can  be  seen  in  Figure  B.4,  to  measure  this   problem   solving,   management   was   asked   about   what   were   the   initial   intervention  strategies  for  two  indicators  of  safe  motherhood  and  two  indicators  of  child  health  and  then  how   these   strategies   changed   over   time.   These   four   questions-­‐-­‐when  we   could   interview  both  of  the  DHO  and  the  Assistant  DHO  or  their  substitutes-­‐-­‐provide  a  series  of  narratives  in  which   we   discovered   a   number   of   new   aspects   about   the   delivery   of   social   services   in  Uganda.    

 

What   emerged   in   several   narratives   was   the   causal   agent   of   change   were   training  interventions  by  either  the  Ministry  of  Health  or  a  specific  implementing  partner,  such  as  the  Civil  Society  Fund,  or  by  another  international  agency  than  USAID.  These  narratives  became  a   learning  device   for   this   interviewer  as  well   as  a  measure  of   the  amount  of   learning   that  had  occurred   in   these  disparate  districts  and  also  a  deeper  understanding  of   the  obstacles  that  management  faces  as  they  try  to  do  their   jobs.    But   implicitly  they  are  also  measuring  whether  management  approaches  their  work  as  problem  solving,  recognizing  that  obstacles  exist  that  prevent  effectiveness,  and  attempt  to  overcome  these  obstacles.      

   

Another   important   source   of   learning   and   one   that   reflects   a   potential   strategy   for  sustainability  is  the  sharing  of  experiences  across  district  levels  (see  questions  5  asked  of  the  DHOs  and  Assistant  DHOs  in  Figure  B.5  and  question  1  in  this  same  figures  used  in  the  focus  groups   with   the   supervisors   and   data   collectors).     Both   the   supervisors   and   the   data  

Page 19: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

19  

collectors  were  asked  if  they  ever  participated  in  meetings  with  their  counterparts  in  other  districts.    From  these  meetings  can  emerge  what  are  called  “best  practices”,  and  individuals  in  one  district  can  learn  how  to  solve  problems  that  they  face  with  the  experiences  of  those  in   other   districts   who   have   solved   the   same   problem   if   these   meetings   are   structured  properly.     Perhaps   even  more   noteworthy   is   that   this   practice   provides   a  mechanism   for  building   morale   and   making   the   LQAS   data   collection   process   more   meaningful   to   those  individuals   involved.    The  same  questions  were  asked  the  DHOs  and  the  Deputy  DHOs.     In  addition  to  asking  questions  about  participating  in  meetings  about  sharing,  we  interviewed  Alice  Nakagwa  about  whether  STAR-­‐E  LQAS    had  organized  meetings  for  this  purpose  and  we  received   reports   on   three   regional   meetings   in   the   Eastern   District,  Western   and   Central  areas.    Unfortunately,  very  few  of  the  respondents  that  we  interviewed  or  who  participated  in  the  focus  groups  had  attended  any  of  these  meetings.  

 

Measuring  Sustainability  and  Challenges  

 

The   problem   of   sustainability   has   been   a   major   issue   in   most   development   projects.    Although   a   set   of   skills  may   have   been   institutionalized,   the   issue   remains  whether   these  skills  will   continue   to  be  utilized.  To  answer   the   third  question  posed   in   the  mandate   that  was  given  by  the  Liverpool  School  of  Tropical  Medicine,  the  interviews  and  the  focus  groups,  contain  several  questions  about  alternative  strategies  that  would  ensure  that  the  LQAS  skills  that  had  been  acquired  would  continue  to  be  used  at  a  relatively  low  cost.    The  latter  is  an  important  constraint.    Governments  such  as  Uganda  have   limited  resources  and  therefore,  sustainability  becomes  the  search  for  strategies  that  are  of  low  cost.    

 

As  can  be  observed   in  Figure  B.5,   the  project  explored  three  strategies.  The  first  and  most  obvious   one   is:     Can   the   data   collection   and   analysis   be   integrated   into   the   regular  work  routines  of  the  data  collectors  and  supervisors?    In  the  beginning  of  the  project  STAR-­‐E  had  made   the   critical   decision   to   recruit   individuals   who   would   be   trained   from   district   level  staff.     As   a   consequence,   the   investment   in   human   capital   remains   until   these   individual  retire.  An  assessment  of  the  possibility  of  integrating  LQAS  into  the  routine  work  of  the  data  collectors  and  of  the  supervisors  was  asked  at  all  four  levels  of  the  district  assessment.  

 

Another   strategy   is   to   transfer   knowledge   from   expert   data   collectors   and   supervisors   to  individuals  in  other  districts  that  have  either  not  received  enough  experience  or  have  never  been  exposed  to  this  method  of  monitoring  and  evaluation.    Therefore,  in  the  focus  groups  both  the  data  collectors  and  the  supervisors  were  asked  if  they  could  train  others.    

 

Page 20: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

20  

Finally,   a   third   strategy   reflects   a   point   when   LQAS   is   so   successful   that   it   is   no   longer  necessary   to   collect   this   information.     Instead   the   data   analysis   can   rely   on   the   recurrent  Health   Management   Information   System   (HMIS).   The   reasoning   is   that   if   the   percentage  becomes   close   to   100   percent,   perhaps   it   is   no   longer   necessary   to   measure   a   specific  indicator  with  a  population-­‐based  survey.  Only  questions  were  asked  about  health  services  but  the  same  logic  could  be  applied  to  other  kinds  of  social  services,  e.g.  schools  measuring  attendance.    

 

These  three  strategies  only  concern  the  question  of  sustainability  at  the  district  level  and  not  at   the   level   of   the   central   government.     The  project   did  not  have  enough   time   to   explore  fully   this   latter   issue   but   did   conduct   several   interviews   about   efforts   at   the   central  government  that  indicate  some  promising  developments.  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Page 21: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

21  

RESEARCH  FINDINGS  

 

The  Degree  of  Institutionalization    

 

The  degree  of   institutionalization  of  LQAS   is  not  a  simple  presence  or  absence;   it  comes   in  degrees.     By   measuring   institutionalization   in   this   way,   it   becomes   much   easier   to  understand   how   it   might   be   further   increased.     Furthermore,   the   extent   of  institutionalization   varies   at   two   distinct   levels   of   analysis:   (1)   in   districts;   and   (2)   in   the  central   government.   The   data   in   this   report   primarily   concerns   the   degree   of  institutionalization  at  the  district  level  but  some  discussion  is  added  about  the  central  level,  which  is  more  than  simply  the  sum  of  the  112  districts  and  their  levels  of  institutionalization.    

 

As  is  indicated  in  Figure  One,  a  bar  chart,  five  of  the  districts  have  a  score  of  75%  or  better  on   the   first   index   of   institutionalization,   namely   training   and   experience.     Three   districts-­‐-­‐Bushenyi,   Kabale,   and   Kamwenge-­‐-­‐scored   100%   on   this   index.   Admittedly,   the   major  determinant   of   their   score   is   how  many   times   has   the   district   participated   in   LQAS   data  collection  but  this   is  also  the   logic  of  how  one  counts  experience.    Those  districts  that  had  four  or  more  LQAS  rounds  did  not  receive  extra  credit  since  it  would  appear  three  times  is  sufficient  to  say  the  skill  levels  involved  in  LQAS  have  been  institutionalized.  It  might  also  be  noted   that   consistently   these   individuals   who   had   multiple   rounds   of   experience   also  reported   that   they   received   refresher   training.     So   no   additional  weight  was   given   to   the  refresher  as  it  was  so  closely  associated  with  having  participated  in  multiple  LQAS  rounds.    

 

The   second   index   of   institutionalization   focuses   on   the   degree   of   coordination   and   social  control,  which   is  probably  of  most   concern   to   those   interested   in  whether   LQAS  has  been  institutionalized.     As   can   be   observed   in   Figure   Two,   all   but   two   districts-­‐-­‐Hoima   and  Kameramaido-­‐-­‐scored   above   75%   on   this   index.     In   other   words,   there   is   more  institutionalization   of   coordination   and   regulating   of   the   LQAS   process   (i.e.,   control)   than  there  is  training  and  experience.    This  is  an  important  finding  because  it  means  that  on  the  more   critical   index,   district   managers   are   incorporating   LQAS   results   into   the   planning  process;  these  nine  districts  at  least  have  higher  scores  of  institutionalization.    On  this  index,  two  districts-­‐-­‐Bushenyi  and  Kabale-­‐-­‐scored  100%.  

 

Some  examples  of  how  the  districts  used  the  findings  in  the  preliminary  reports  for  planning  and  budgeting  follow.    The  respondents  in  Kamwenge  stated  that  they  had  used  the  results  to  secure  additional  funding  and  mentioned  in  particular  attempts  to  increase  their  rates  of  immunization  and  male  circumcision.    Bushenyi  was  concerned  by  their  low  rate  for  having  four  or  more  antenatal  care  visits,  and  started  working  with  the  VHTs  to  get  mothers  to  visit  

Page 22: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

22  

the  clinics.    The  VHTs  were  instructed  to  visit  houses  to  emphasize  exclusive  breastfeeding.    Also  they  trained  the  VHTs  to  train  mothers  in  danger  signs  during  their  antenatal  visits  and  political   leaders  were   used   to   promote   immunization.     Finally,   health  management   teams  were  very  explicit   about  planning   to  build  a  new  health   facility  each  year  and   training   the  staff  for  it  on  the  basis  of  the  LQAS  results.    Mbale  also  used  the  LQAS  results  to  plan  infra-­‐structure.     Following  many   of   the   same  procedures   that   Bushenyi   did,   in  Mbale   the  VHTs  escorted  mothers  to  the  HF.  They  also  encouraged  fathers  to  transport  the  mothers.    In  sum,  the  DHO  in  Mbale  felt  that  LQAS  pushed  them  to  take  more  responsibilities  for  their  district.  

 

Figure  One  Degree  of  Institutionalization  Based  on  Training  and  Experience:  Percentages  Computed  on  a  Total  of  12  points  

 

 

Page 23: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

23  

Figure  Two  Degree  of  Institutionalization  Based  on  Coordination  and  Control:  Percentages  Computed  on  a  Total  of  12  

 

 

Although   Kaberamaido   scored   relatively   low   in   the   degree   of   institutionalization,   they   did  report   the   use   of   LQAS   data   to   indicate   that   there  was   a   basic   problem  with   the   level   of  latrines   in   some   of   the   sub-­‐districts   and   this   issue   was   addressed.   The   scores   on  breastfeeding  were  used  in  Tororo  to  inform  changes  in  their  interventions.    But  this  district  also   illustrates  a  challenge  when  the  LQAS  produces  too  much  demand  for  HF,  resulting   in  long  lines  at  the  HF.    The  absence  and/or  failure  of  electricity  at  the  HF  during  the  night  also  discouraged  many  of   the  women  who  had  come   for  delivery  of   their  babies  and   they   left.    Mbarara  used  the  LQAS  results  in  family  planning  to  improve  their  services  in  this  area.    Also,  in   monitoring   immunization   they   realized   that   they   needed   to   make   more   effort.     They  started  educating  the  mothers  and  encouraging  the  fathers  to  take  them  to  the  HF.    Kabale  had  their  VHTs  explain  that  health  services  were  free  and  make  house-­‐to-­‐house  visits.    They  also  integrated  their  immunization  efforts  with  other  health  services  to  improve  their  score  on  this  indicator.  

                         

 At   the   beginning   of   this   report,   we   started   with   the   assumption   that   there   was   a  relationship   between   the   degree   of   institutionalization   and   the   effectiveness   of   health  service   delivery,   at  minimum,   the   percent   of  women  who   give   birth   in   a   health   facility   of  some  kind.    This  relationship  can  now  be  tested  directly.    If  we  add  together  the  two  scores  on   institutionalization   in   Figures   One   and   Two   and   assign   ranks,   we   can   compute   a  Spearman’s  Rho  with  the  rank  ordering  of  the  districts  based  on  the  data  in  Table  1.    When  this  is  done,  the  correlation  is  0.45.    In  other  words,  the  combined  institutionalization  index  

Page 24: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

24  

scores  and  the  district  rank  are  related,  and  that  association  is  moderately  strong.    Given  the  above  data  on  the  number  of  examples  of  how  LQAS  was  used,  confidence  about  the  causal  connection  represented  by  the  correlation  is  established.    

 

Independent  of  quantitative  measures,  one  can  also  supply  some  qualitative   judgments  as  well   if  based  on  an  accurate  perception  of  the  focus  groups  and  interviews  across  multiple  levels   of   analysis.     On   this   basis,   Hoima   and   Kaberamaido   represent   interesting   contrasts.    The  data  collectors  and  supervisors  in  the  former  instance  were  unenthusiastic  about  LQAS,  with  one  person  suggesting  that  the  money  could  be  better  used  to  support  health  services  while   in   the   latter   instance   their   counterparts   in   Kaberamaido   were   quite   enthusiastic.    There   can   be   a   number   of   reasons   for   these   fundamentally   different   attitudes.     First,   the  budgets  for  the  data  collection  were  sorely  underestimated  in  Hoima  where  the  conditions  were  quite  challenging.  For  example,  that  area  has  particularly  difficult  access,  the  team  had  experienced  a  breakdown   in  transportation,  multiple   language  skills  were  needed,  and  the  area   lacked   overnight   lodgings.   These   deficiencies  may   have   led   to   resentment   upon   the  part   of   those   involved.   Although   the   budgets   for   data   collection   are   customized   for   each  district,   this   specific   LQAS   funding   came   from   SDS   and   probably   the   estimate   was   not  informed  by  experience,  especially  the  unusual  conditions  that  existed.      

 

Secondly,   Kaberamaido   even   without   a   second   round   of   data   scored   a   big   success   in  increasing  coverage  with   latrines   in  this  district  and  this  result  empowered  the  supervisors  and  data  collectors.    Also,  they  used  the  LQAS  results  to  obtain  additional  funding  from  CSF  for   HIV/AIDS   work.   This   is   one   of   the   critical   points   about   why   a   second   round   of   data  collection   can  be   so   crucial   in   the  process  of   institutionalizing   LQAS.    Although  Hoima  has  made   a   number   of   attempts   to   use   the   LQAS   data   on   the   basis   of   the   reports   that   are  available,   they  have  not  had  any  measurement  of  whether   their   changes   in   strategies  and  tactics   have   had   a   pay-­‐off   in   contrast   to   Kabermaido.    One   implication   of   this   tale   of   two  districts  is  the  importance  of  ensuring  that  there  are  always  at  least  two  rounds  of  LQAS  that  are   supported   so   that   changes   on   the   indicators   can   demonstrate   the   efficacy   of   this  methodology  and  provide  the  data  collectors  and  supervisors  with  a  sense  of  empowerment.      

 

Third,   a   subtle   factor   that  was   suggested  by  Charles  Nkolo   is   that  Hoima   is   an  old  district  whereas   Kabermaido   is   a   relatively   new   one.     There   is   a   large   literature   on   age   of  organizations   and   resistance   to   change   that  might   be   applicable   in   explaining   differences  between  these  district  organizations.    However,  this  issue  would  require  additional  research.      

 

Implied  in  the  discussion  of  these  two  districts  is  one  of  the  major  reasons  for  the  success  of  LQAS.    Not  only  does  it  highlight  with  its  system  of  red  flags  sub-­‐counties  and  districts  that  are   not   performing   well   but   also   when   changes   in   strategies   and   tactics   are   made,   the  resulting  improvement  in  the  effectiveness  in  the  delivery  of  services  reinforces  the  trust  in  

Page 25: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

25  

this  monitoring   and   evaluation   system   and   gives   health  management   a   sense   of   efficacy.    But   it   is   not   only   feedback   about   the   improvement   that   accounts   for   the   success   of   this  system.    There  are  in  our  opinions  and  based  on  the  information  received  in  the  interviews  three  important  ingredients  that  tend  to  reinforce  each  other.      

 

First,  when  the  preliminary  results  are  provided,  all  the  key  stakeholders  are  invited.  This  is  defined  not  only  by  the  key  political  and  technical   leaders   in  the  districts  such  as  the  Local  Council   V   Chairperson   and   the   Chief   Administrative   Officer,   the   Sub-­‐County/Parish   chiefs,  the  implementing  partners,  religious  leaders,  the  media  among  others.    Given  this  diversity,  the  district   feedback   session  mobilizes   support   for   changing   the   strategies  and   tactics   and  providing   better   services.     In   this   regard   it   is   instructive   that   the   terminology   used   in   this  setting   (e.g.,   stakeholders)   is  associated  with  business   schools   in   the  United  States.     It   is  a  concrete  example  of  how  to  create  ownership  in  a  developing  country  (USAID,  2013).  

 

Second,  the  villages  outside  the  urban  areas  appear  to  have  high  social  capital.  The  definition  is   the   extent   of   cooperation   within   the   village.     As   reported   in   Appendix   A,   it   became  apparently   in   the   focus   groups,   that   there  was   not  much   variation.    Most   sub-­‐districts   or  counties   had   a   high   degree   of   cooperation   as   illustrated   in   community   members   sharing  transportation,   helping   in   funerals,   and   aiding   in   planting.   In   the   original   research   design  visits   to   villages   had   been   planned   on   the   assumption   that   they   varied   on   this   dimension  (see   Appendix   A).   But   since   all   the   data   collectors   and   supervisors   reported   a   lack   of  variation,   this   part   of   the   research   design  was   eliminated.   This   high   level   of   social   capital  makes   it   much   easier   to   develop   effective   strategies   for   social   services.     Naturally   the  incorporation  of  the  sub-­‐county  /  parish  chiefs  in  the  dissemination  of  the  results  reinforces  the  extent  of  cooperation.  

 

Third,  and  described  below,  is  the  system  of  how  the  VHTs  are  selected  and  their  procedures  for  encouraging  women  to  have  antenatal  visits  and  to  have  their  children  immunized.    Since  they   are   elected,   they   have   the   trust   of   the   community.     They   also   rely   upon   the   parish  chiefs  to  help  them  in  their  campaigns  to  get  women  to  visit  the  HF  when  they  are  available.    And  since  the  number  of  households  for  which  each  member  of  the  VHT  is  responsible,  only  25,  they  know  when  a  woman  becomes  pregnant  and  can  rapidly  promote  early  antenatal  visits.  

 

The  measures  of  institutionalization  discussed  above  are  at  the  district  level.    Quite  separate  issues  emerged  when  one  moves  to  the  national  level.    At  this  level,  one  has  to  observe  that  some   20%   of   the   districts   in   the   country   have   had   no   exposure   to   LQAS.     Based   on   the  evidence   compiled   in   Figure  One,   one  observes   that   some  districts   need  more  experience  with   LQAS   before   one   might   describe   the   monitoring   and   evaluation   process   as  institutionalized  in  all  districts  where  it  has  been  introduced.    If  one  accepts  the  cut-­‐off  of  a  

Page 26: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

26  

score  of  nine  or  75  percent  on  the  first  index  of  institutionalization  as  suggested  above,  then  the  districts  of  Mbarara,  Kaberamaido,  Hoima,  and  Tororo  require  additional  training  inputs.  However,  what  should  be  exceptionally  encouraging  to  USAID  is  that  of  these  four  districts,  all  but  two,  Kaberamaido  and  Hoima,  scored  highly  on  the  utilization  of  the  data  in  planning  and   in   these  two   latter  cases,  other  data  obtained   from  Lydia,   the  NLF   in  Hoima  and  from  Esther   Sempira,   the   national   coordinator   of   STAR-­‐E   LQAS   indicate   that   Hoima   has   been  active   in   using   the   data   in   planning.     This   topic   leads   naturally   into   the   discussion   of  challenges.  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Page 27: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

27  

CHALLENGES  

 

The   first   challenge   is   to   extend   institutionalization   not   only   to   the   four   districts   indicated  above,   but   also   another   ten   districts   that,   like   Hoima   and   Kaberamaido,   did   not   receive  funding   in   2013.       From   our   list   as   annotated   by   Esther   Sempira,   STAR-­‐E   LQAS   national  coordinator,   the   other   districts   are:     Bundibugyo,   Masindi,   Buliisa,   Adjunami,   Kotido,  Koboko,  Nkapiripirit,  Kiboga,  Mubende,  Mukono,  and  Rakai.    Beyond  this  are  the  32  districts  that  have  never  had  any  exposure   to  LQAS.    But  as   indicated   in   the   recommendations   the  first  priority  should  be  these  13  that  is  the  list  of  11  plus  Hoima  and  Kaberamaido  to  solidify  the  previous  investment  in  LQAS  before  one  extends  this  monitoring  and  evaluation  process  to  another  32  districts.  

 

One  of   the  unexpected  challenges  that  emerged   in  the  data  collection  from  the  DHOs  and  the  Assistant  DHOs  was  the  problem  of  turnover  in  these  positions.    Recent  appointees  who  arrived   after   the   last   round   of   data   collection   did   not   know   about   LQAS   and   seemed  uninterested,   as   can   be   seen   in   Table   A.2.     This   poses   a   long-­‐term   threat   to   the  institutionalization  of  the  system  if  members  of  top  management  teams  are  uninterested  in  receiving   the   reports   and   they   are   unlikely   to   be   interested   if   they   had   not   received   any  exposure.      Thus   to  sustain   the  LQAS  system  one   important   issue   is   to  continue   to  expose  management  to  this  system  of  monitoring  and  evaluation.    To  ensure  that  future  DHOs  and  Assistant   DHOs   receive   some   training   in   the   advantages   of   LQAS   and   possibly   LQAS  principles,  necessitates  some  arrangement  between  the  Ministry  of  Local  Government  and  the   Ministry   of   Health   so   that   the   selected   individuals   for   promotion   or   those   newly  recruited  into  these  positions  can  receive  this  training  

 

The   data   collectors   and   supervisors   generally   agreed   that   the   major   challenges   to   data  collection   were   the   problems   of   collecting   data   from   quite   busy   people   and   the   need   to  return   to   the   households   selected   from   a   table   of   random   numbers.     These   issues   are  connected   with   the   need   to   sample   specific   age   categories   for   disparate   sections   of   the  LQAS   instrument.    Another   common  complaint  was   the  need   for  more   time   to   collect   the  data.    Some  data  collectors  reported  that  they  felt  stressed  by  the  time  pressures  and  having  to  work  sometimes  on  both  Saturday  and  Sunday  to  finish  their  data  collection,  sometimes  possibly  without  additional  facilitation.    Compounding  the  time  pressures  were  the  distances  involved   in   reaching   different   households   and   a   variation   on   this   theme   from   supervisors  was   the   difficulty   of   supervising   two   data   collectors  who  were  working   in   quite   disparate  parts  of  a  village  or  community.    Magnifying  the  time  pressures  and  distances  were  reports  that  the  transportation  provided  broke  down  or  was  not  readily  available.  Another  obstacle  mentioned  by   supervisors  was  a   lack  of   time   for  data  analysis.  One   complaint   in   a  district  with   only   one   round   of   LQAS   data   collection  was   a   problem   of   obtaining   data   during   the  rainy  season.    Other  districts  noted  that  this  problem  had  been  solved  in  their  second  round  of  data  collection.    As   indicated   in  the  recommendations  for  sustainability,  these  problems  

Page 28: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

28  

can  be  solved  easily  with  planning  that  responds  to  the  needs  of  the  districts.    Learning  was  also  demonstrated   in   a  district   that   reported  at   first   some   resistance   from  Muslims  about  certain  questions  regarding  sexual  behavior.    But  with  the  support  of  the  local  leaders,  these  issues   were   resolved.     One   of   the   anticipated   challenges,   namely,   problems   of   collecting  data  across  59  indicators  posed  no  problem.      

Two   challenges   for   the   future   of   LQAS   are   the   age   of   either   the   data   collectors   or   the  supervisors.    Four  individuals  were  50  years  of  age  or  older.    These  individuals  will  eventually  have  to  be  replaced  by,  perhaps,  younger  individuals.    This  Human  Resource  matter  ought  to  be  discussed  with  district  managers.    Another  problem   is   that   in   some  districts,   not  much  diversity  in  the  positions  occupied  can  create  difficulties  in  acceptance  (see  Table  A.3).  While  it   is  understandable  that  the  health  sector  would  supply  many  of  the  individuals  who  were  trained,   there   were   few   from   the   education   sector,   one   sector   that   the   Director,   Local  Government   Administration   wants   to   emphasize.     Ideally   one   would   like   to   have   data  collectors   and   supervisors   that   reflect   all   the   relevant   sectors   in   a   district   to   build   more  support   for   the  methodology.     In   Tororo,   six   of   the   data   collectors   and   supervisors   were  from   the   community   development   office,   one   was   from   planning,   which   is   akin   to   the  former,  and  two  were  health  assistants.    This  concentration  may  diminish  acceptance  in  the  future.      

 

Two   more   subtle   issues   emerged   had   not   expected.     The   first   is   the   expectations   of  individuals  interviewed  that  they  would  receive  something  for  participating.    Apparently  this  became  particularly  heartrending  when  interviewing  orphans  and  other  vulnerable  children.    Closely  connected  to  this  first  issue  is  a  second  one,  namely,  the  lack  of  dissemination  of  the  results  back  to  the  sub-­‐counties,  parishes,  villages  through  the  political  and  technical  leaders  and   also   the   VHTs.     Three   districts   mentioned   this   problem,   Bushenyi,   Kabale,   and  Kamwenge,   but   at   the   same   time   indicated   that   they   did   not   have   the   funds   to   ensure  dissemination.    This  would  appear   to  be  an  easy  problem  to  resolve  as   long  as  one  avoids  expensive  printing  procedures.  

 

Another   challenge,   particularly   in   the   eastern   region   was   the   problem   of   the   lack   of  availability   of   HF   as   well   as   an   unstable   supply   of   electricity   in   the   HF   that   affected   the  willingness  of  the  pregnant  women  to  give  births  there.    In  particular,  there  is  a  dearth  of  HF  in   the   national   park   located   in   the   eastern   region.   Also,   the   unevenness   of   the   supply   of  electricity  was   affecting   the   quality   of   the   vaccines   being  maintained   in   the   refrigerators.  Beyond  this,  many  of  the  heath  facilities  are  not  connected  to  the  electricity  grid  and  either  rely  upon  solar  or  paraffin/wax  candles.  These  problems  help  to  explain  to  a  certain  extent  the  negative  correlation  between  visits  to  HF  and  recognizing  danger  signs  in  children.    But  as  suggested  above,  the  challenge  may  be  the  inability  to  pursue  multiple  improvements  in  social  services  given  limited  budgets.    How  much  this  was  a  problem  in  the  districts  that  was  not  explored.    Nor  was  the  particular  emphasis  in  a  district  was  influenced  by  the  concerns  of  the  major  implementing  partner  investigated  either.  

Page 29: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

29  

The  research  did  examine  the   impact  of  the  availability  of  HF   in  a  specific  district  and  how  this  relates  to  the  indicator  reported  in  Table  1,  namely,  the  percent  of  women  having  their  delivery  in  a  health  facility.    The  availability  of  HFs  is  a  function  both  of  the  number  of  square  kilometers  and  the  number  of  people  who  live  in  a  specific  catchment  area.  The  number  of  facilities   for   each   district   per   1,000   people   living   in   100   square   kilometres   is   reported   in  Figure  Three.    The  construction  of   this   table  requires  two  assumptions,  neither  of  which   is  true.      

• First  assumption:  HFs  are  evenly  distributed  across  the  district.    As  indicated  above,   this   is   not   the   case   with   particular   areas   such   as   national   parks  lacking  these  facilities.      

• Second   assumption:   The   proportion   of   women   in   child-­‐bearing   ages   are  approximately   the   same   relative   to   the  population  across  all  nine  districts.    Again,  this  is  certainly  not  true  and  especially  in  the  districts  that  have  more  urban  areas.        

 STAR-­‐E   LQAS   analysis   revealed   that   mothers   having   their   births   in   a   health   facility,   is  correlated  (r  =  0.61)  with  the  number  of  antenatal  visits,  as  one  would  expect.      But  if  one  accepts  these  limitations,  Figure  3  still  does  reveal  interesting  findings.  As  can  be  observed   there   are   quite   strong   variations   among   the   districts   with   Hoima   in   particular  having  a  very  low  number,  one-­‐third  of  a  facility  per  1,000  persons  in  a  100  square  kilometre  catchment  area.      At  the  other  extreme  is  Kabale,  which  has  more  than  four  facilities  for  the  same  size  area  and  population  density.    If  one  ranks  the  availability  of  HFs  and  compares  this  to  the  rank  order  reported  in  Table  1,  the  Spearman  Rho  is  a  very  high  .79.    In  other  words,  there   is   a   strong   relationship   between   percent   of   women   who   deliver   their   babies   in   a  facility  and  the  availability  of  an  HF.    What   is  worth  observing   in  Figure  3   is   that  Bushenyi,  which   has   the   highest   percentage   of   women   giving   birth   in   HF,   has   the   second   lowest  availability.    This  district  represents  a  model  for  other  districts  and  illustrates  how  the  use  of  LQAS  can  allow  for  the  effectiveness  of  services  despite  a  considerable  resource  constraint.    It   also   suggests   one   of   the   best   ways   of   testing   for   the   effectiveness   of   LQAS,   namely  whether  it  can  overcome  a  major  resource  constraint.  

But  at  the  same  time,  if  one  analyses  all  the  districts  in  the  STAR-­‐E  LQAS  super-­‐dataset,  there  is  a  surprising  negative  correlation  between  the  use  of  HF  for  birth  delivery  and  taking  sick  children   to   one.     In   this   larger   analysis,   the   eastern   region   has   a   significant   negative  correlation.      In  an  interview  with  the  Assistant  DHO  in  one  of  the  districts,  she  reported  that  the  lack  of  electricity  meant  that  they  could  not  keep  vaccines  safe.  

                     

Page 30: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

30  

 Figure  Three  The  Number  of  Health  Centers  Relative  to  Population    and  Square  Kilometres  

 

 

Not   all   challenges   are   necessarily   negative   ones.     Several   positive   challenges   emerged,  positive   in   the   sense  of   how  various   respondents   felt   about   LQAS.    One  DHO  wished   that  LQAS   could   be   extended   to   additional   service   sectors   because   he   wanted   the   support   of  other   sectors   of   local   government.     Interestingly   enough   this   was   also   echoed   in   the  interview   with   Patrick   Mutabwire,   Director   of   Local   Government   Administration,   who  desires  to  extend  the  methodology  particularly  to  the  education  sector.    An  Assistant  DHO  would   like   to   have  more   individuals   in   the   local   government   trained   in   the  methodology.  This   idea  might  be  considered  when   the   transfer  of  knowledge  across  districts  occurs   (see  recommendation  below).    

 

On   the   relationship  between   the  district   government  and   the   central   government,   several  questions   were   included   in   the   interviews   with   the   DHO   and   the   deputy   DHO   on   the  assumption  that  there  might  be  problems  in  the  use  of  LQAS  data  by  the  central  government  when   it   is   making   its   five-­‐year   plans.     One   of   these   questions,   namely   the   amount   of  technical   support,  was  misconstrued   and   therefore  was   eliminated.     Instead   of   answering  about  the  support   from  the  central  government,  the  respondents  reported  on  the  support  received  by  STAR-­‐E.    The  two  questions  about  whether  the  data  had  ever  been  transmitted  and/or  used  by  one  or  another  ministry  in  planning  could  not  be  answered.    The  DHOs  and  the  Assistant  DHOs  were  unaware  of  how  the  data  were  transmitted  and  how  it  influenced  the  planning  processes  of   the  ministries  of   the   central   government.     This   then  becomes  a  major  challenge  for  ensuring  that  the  LQAS  process  becomes  institutionalized  at  the  national  level.     Strikingly,   one   of   the   DHOs   proudly   announced:   “the   districts   owned   the   data”.    

Page 31: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

31  

Indeed,  the  strategy  of  MSH  and  LSTM  was  to  encourage  this  attitude.    But  the  downside  is  that   this   emphasis   has   not   facilitated   the   movement   of   the   data   into   the   appropriate  planning  departments  at  the  central  government  level.      

It  might   be   noted   that   various  ministries   have   invested   in   large   surveys,   in   particular   the  Ministry  of  Health  as  it  expands  HMIS.    The  time  frames  and  concerns  are  not  the  same  and  this  poses  some  real  challenges  of  how  to  overcome  this  blockage.  

 

Learning  and  Challenges  

 

As  has  already  been  demonstrated,  LQAS  produced  a  considerable  amount  of  learning  in  the  nine  districts  involved  in  this  study.    The  three  regional  meetings  organized  by  MSH  reports  are   available   from   Alice   Nakagwa   of   MSH   (2013   Regional   Information   Sharing   Meeting:    Central   and   East   Central   Region  Districts     Kampala:    MSH,  Uganda   is   one   example).     Even  more  extensive  is  the  summary  report  prepared  by  Esther  Sempira,  the  national  coordinator  of   STAR-­‐E   LQAS,   that   includes   information   on   about   40   districts   (see   Appendix   C   for   one  year).  These  provide  additional  documentation  of  the  successes  of  LQAS  to  impact  on  social  service  effectiveness.    The  districts  recognized  that  they  were  not  performing  well  in  certain  areas  and  then  attempted  to  improve  their  scores.      

 

But   data   collected   by   LQAS   are   not   the   only   form   of   learning   that   occurred   in   the   nine  districts.   Two  major  external   sources  of   learning  were   reported   in   the   interviews  with   the  DHOs  and  Assistant  DHOs  (or  their  substitutes  in  several  cases,  see  Table  A.2).  and  in  more  than  one  district.    The  most  important  was  a  change  in  the  policy  of  the  Ministry  of  Health  to  encourage   the   development   and   training   of   VHTs.     A   number   of   districts   reported   on   the  important  change  of   introducing  VHTs  who  are  selected  by  the  members  of  the  village.     In  the   discussion   above   about   when   the   preliminary   reports   were   made,   frequently   the  decision  was  made  to  use  these  men  and  women  to  improve  social  services  and  in  particular  in  maternal  and  child  health.  Furthermore,  the  procedures  that  the  Ministry  of  Health  used  to  establish  this  key  component  of  the  health  care  system  needs  to  be  carefully  understood  to  appreciate  why  this  is  such  an  effective  policy.  Members  of  communities  are  involved  in  the   selection   of   VHTs   and   vote   for   them.     Therefore,   the   selected   men   and   women   are  trusted  and  their  recommendations  are  more  likely  to  be  taken.    One  member  as  has  already  been  stated   is   responsible   for  25  households  and   the  number  of   team  members   is  usually  between  four  to  six  since  most  villages  are  quite  small.    Indeed,  it  is  a  contention  that  part  of  the  success  of  LQAS  relies  upon  the  way  in  which  these  women  and  sometimes  men  could  be  utilized  given   the   recognition  of   some   failing   such  as   the   low   rate  of  antenatal   care  or   the  rates   of   immunization   or   tests   for   HIV/AIDS.     As   can   be   seen   above,   the   VHTs   sensitized  mothers   to   various   issues,   educated   them   in   the   importance   of   integrated   disease  management   (IDM),   the   recognition   of   danger   signs   in   small   children,   the   importance   of  antenatal  visits,  etc.    

Page 32: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

32  

 

A   second   example   of   learning   was   the   intervention   of   Baylor   College   of   Medicine   in  Kamwenge  and  Kabarole   to  provide  more   rigorous   training  of   the  VHTs.     In  addition,   they  paid   VHTs   25,000   shillings   (approximately   $10),   and   gave   them   umbrellas   and   bicycles   as  well  as  provided  districts  with  ambulances.  They  also  established  a  voucher  system  of  5,000  shillings   that   allowed   mothers   to   take   motorcycles   to   visit   HF   for   their   antenatal   visits.    Baylor  also  provided  pregnant  mothers  with  bed  sheets,  soap,  gloves,  and  baby  shorts.    The  consequence  was  a  very   large   increase   in  antenatal   visits  and  giving  births   in  HF.    But   the  intervention  was  for  only  one  year  and  since  then  percentages  have  started  to  drop  as  the  voucher   system   for   visits   no   longer   exists.     Thus,   this   system   cannot   be   maintained   but  perhaps  some  elements  might  be  sustainable.    This  is  a  significant  challenge.  

 

Relative  to  the  question  posed  at  the  beginning  of  this  report  of  whether  learning  that  has  improved   the   effectiveness   of   services   is   a   combination   of   both   LQAS   and   of   various  interventions  by  either  other  donor  agencies  and/or  ministries,  the  answer  is  that  indeed  it  is.    It  is  impossible  on  the  basis  of  the  data  available  to  estimate  what  proportion  of  changes  in  the  effectiveness  of  social  services  is  attributable  to  one  or  another  learning  process.    The  single  exception  was  the  striking  increase  in  the  use  of  HF  given  the  Baylor  intervention,  but  as  already  noted,  this  increase  was  unstable.  

 

Relative   to   other   forms   of   learning,   some   of   the   DHOs   did   report   participating   in   various  meetings  that  increased  their  learning.    A  national  conference  on  family  planning  organized  by  the  Ministry  of  Health  was  mentioned.    In  some  regions  there  are  regular  meetings  about  maternal   health.     And   of   course,   there   have   been   meetings   about   HIV/AIDS.     MSH   also  organized  three  regional  meetings  but  only  one  of  our  respondents  mentioned  it,  an  NLF.    

 

Sustainability  and  Challenges  

   

The   first   strategy  explored   in   the   interviews  with   the  DHOs,  Assistant  DHOs  and   the   focus  groups  was  whether   the  data  collection  could  be   integrated   into   the  on-­‐going  work  of   the  supervisors   and   the   data   collectors.     There  was   almost   total   consensus   that   this   could   be  done.    Quite  surprisingly   few  mentioned  the   loss  of   the  per  diem  that   they  received  when  collecting   data.     The   area   that   requires   some   expenditure   of   funds-­‐-­‐that   was   repeatedly  mentioned   as   indicated   above   in   the   discussion   of   challenge-­‐-­‐is   the   adequacy   of  transportation.    Therefore,  if  LQAS  is  to  be  sustained,  this  is  a  major  issue,  not  only  because  of  the  necessity  of  visiting  villages  that  are  remote  but  also  because  the  equipment  provided  is  inadequate.  From  the  perspective  of  the  DHOs  and  their  Deputies,  a  typical  response  was:  “Why  Not?”  And  some  indicated  that  there  were  active  discussions  about  how  this  might  be  accomplished.      

Page 33: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

33  

   

Although  there  was  consensus  that  data  collection  could  be  integrated,  one  word  of  caution  is  necessary.    As  can  be  observed  in  Table  A.3,  there  is  considerable  variety  in  the  positions  of   the   data   collectors   and   their   supervisors.   One  might   imagine   that   this   would   be  more  easily   achieved  with   some   positions,   e.g.   those  who  work   in   the   health   sector,   than  with  other   positions.     Another   word   of   caution   came   from   a   newly   appointed   DHO   who   was  familiar  with  the  process  because  of  a  previous  appointment.    He  thought  that  there  might  have  to  be  some  diminution  in  the  quality  of  the  data  so  as  to  reduce  costs.    Among  other  options  he  considered  were  a  reduction  in  the  number  of  indicators,  less  supervision  of  the  data  collection  and   fewer   tests  of   its  quality  because   the  double   interviewing  of   the   same  respondents   would   no   longer   be   conducted.     However,   other   members   of   the   health  management   team   stressed   the   importance   of   the   many   indicators   as   a   way   of   building  support  within  their  district  governments  for  the  process.  

   

Several  of  the  challenges  outlined  above  represent  opportunities  not  only  for  sustainability  but  also  for  improving  the  quality  of  the  collected  data.    Lengthening  the  amount  of  time  in  which  the  data  is  collected  leads  to  more  successful  integration  of  this  process  with  the  on-­‐going  work  of  the  supervisors  and  data  collectors.    More  critically,  it  would  reduce  the  stress  encountered  because  of  the  problems  of  certain  age  groups  not  being  available  at  particular  times  and  the  need  to  collect  all   the  data   in  one  week.    However,  much  the   integration  of  data   collection   into   the  work  of   the  data   collectors   and   supervisors  would  appear   to  be  a  solution  to  sustainability,  it  requires  carefully  planning;  see  the  recommendation  below.  

   

The  second  strategy,  namely,  the  transfer  of  knowledge,  to  those  districts  with  only  one  or  two  rounds  of  data  collection  and  even  more  to  those  districts  with  no  rounds,  also  received  considerable  support.    All   the  data  collectors  and  supervisors  believed  that   they  were  well  trained  enough  that  they  could  train  others.      

 

The  previous  two  strategies  were  strongly  endorsed  but  the  third  one  was  roundly  rejected.    In   this   strategy   we   posed   the   possibility   that   LQAS   could   be   phased   out   when   the  percentages   reached   a   certain   level   with   the   idea   that   the   health   care   system   would  systemically  collect  this  data.    In  particular,  we  used  the  examples  of  births  in  HFs.      We  had  not  considered  asking  questions  of  the  health  management  team  about  their  commitment  to  LQAS  because  of  the  conviction  we  would  only  obtain  “political  correct”  answers  since  the  whole  interview  was  about  this  methodology  but  in  fact  this  question  accomplished  this  very  objective.    All  the  DHOs  and  Assistant  DHOs,  who  had  been  exposed  to  LQAS  felt  that  even  if  the  percentage  reached  100%,   it  would  still  be  necessary  to  have  this   information  because  the  mothers  and  the  children  are  constantly  changing  and  there  was  no  guarantee  that  the  percentage   would   not   decline.     Several   mentioned   how   important   were   the   red   flags   to  know  which  particular  parishes  or  sub-­‐districts  needed  special  attention.  

Page 34: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

34  

 

As   indicated   in   the   discussion   of   the   research   strategy   (see   Appendix   A),   the   primary  objective  of  this  research  was  to  measure  institutionalization  and  sustainability  at  the  level  of  the  district  and  not  the  central  government.    Certainly  as  indicated  in  the  discussion  of  the  challenges,   the   connection   between   the   districts   and   the   central   government   remains   a  challenge   that  needs   to  be   considered.    However,  we  did   conduct  a   few   interviews  at   the  central   government   level,   in   particular   with   Patrick   Mutabwire,   Director   of   Local  Government   Administration.     He   chairs   a   technical   working   group   (TWG)   to   focus   on   the  sustainability   of   LQAS   and   even   its   expansion   to   include   more   indicators   in   sectors   of  concern.     He   stressed   the   importance   of   schooling   and   the   failure   of   the   public   sector   to  provide   adequate   education.     Also,   in   an   interview   with   Esther   Sempira,   the   national  coordinator   of   STAR-­‐E   LQAS,   she   reported   that   the  Ministry   of   Gender,   Labor   and   Social  Development  has  found  the  data  to  be  quite  useful.    

   

Although  a  number  of  efforts  have  been  made  by  MSH  to  sensitize  the  various  ministries  to  the   importance   of   sustaining   the   collection   of   data   by   the   LQAS   method,   the   question  remains  as   to  whether  a  new  strategy  can  be  devised.    One  possibility   is   to   identify   those  individuals   in  the  central  government  that  have  been  responsible  for  organizing  workshops  on  various  subjects,  not  just  in  the  Ministry  of  Health  but  in  other  ministries  as  well.    Implied  in   this   idea   is   that   those   individuals   who   have   done   this   are   the   ones  most   interested   in  solving   problems   and   therefore   most   open   to   examining   the   indicators   that   are   being  measured   as   the   source   of   new   kinds   of  workshops   that   represent   attempts   to   solve   the  problems   reflected   by   the   low   scores   on   specific   indicators.     These   individuals   might   be  recruited  to  serve  on  the  TWG  being  formed  in  the  Ministry  of  Local  Government.    Since  in  these  ministries   communications  with   these   individuals  must   go   through   the   head   of   the  ministry,  the  justification  for  their  serving  in  the  TWG  is  the  possibility  that  if  several  if  not  all  ministries   work   together   they   would   be   more   effective   in   raising   funds   for   LQAS   from  various  donor  agencies  in  the  world.      

 

 

 

 

 

 

 

 

 

Page 35: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

35  

RECOMMENDATIONS  

 

From  the  above  findings  several  recommendations  about  increasing  institutionalization  and  sustainability   while   reducing   challenges   emerge.     At   the   request   of   STAR   E-­‐LQAS,   these  recommendations   are   grouped   into   two   categories.     First   are   the   short-­‐term  recommendations  that  can  be  implemented  in  the  next  nine  months  while  funding  remains  depending   upon   the   amount   of   funding   still   available.   If   funding   is   not   available   for  recommendation  two,  then  this  would  move  into  the  category  of  recommendations  for  the  long-­‐term,   which   are   recommendations   requiring   additional   sources   of   funding.     But  although  they  might  require  additional  funding  the  recommendations  being  made  consider  a  number  of  ways  in  which  the  costs  can  be  reduced.  

 

Short-­‐Term  Recommendations  

 

1.    Attempt  to  include  members  of  other  ministries  in  the  Technical  Working  Group  being  formed  by  Patrick  Mutabwire,  Director  of  Local  Government  Administration.  

Considerable  efforts  have  been  made  by  STAR  E-­‐LQAS  to  find  individuals  in  other  ministries  that   are   strategically   placed   and   interested   in   sustaining   LQAS.     Certainly   the  Ministry   of  Gender  has  found  the  data  collected  to  be  useful  for  their  planning.  One  potential  source  of  champions  for  LQAS  in  each  of  the  ministries  relevant  to  the  sources  of  data  being  collected  across   the   59   indicators   are   those   individuals   who   have   organized   workshops   for   the  districts.  The  data   in  these   indicators  suggest  topics  about  how  social  services  can  become  more  effective  via  specific  kinds  of  interventions  that  they  might  be  willing  to  support.  Once  they  express  an  interest  in  this,  then  the  next  step  would  be  to  request  their  participation  in  the  TWG.  As   the  number  of  ministries   that  participate   in   the  TWG   increases,   then   sharing  the  costs  of  LQAS  becomes  possible,  which  in  turn  reduces  the  amount  that  any  one  ministry  would  contribute.  

 

2.    Support  a  second  round  of  LQAS  data  collection  in  the  13  districts  that  so  far  have  had  only  one  round.  

The   long  discussion  of   the  differences   in   the   reactions  of  Hoima  and  Kameramaido   above  indicates  the  importance  of  having  feedback  when  strategies  have  been  changed  so  that  the  district  becomes  more  committed  to  the  data  collection  process.  Since  one  round  has  been  accomplished  and  the  data  collectors  and  the  supervisors  in  at  least  the  two  districts  that  I  visited   think   that   they   have   the   necessary   skills,   it   would   be   unfortunate   to   lose   the  momentum  that  has  been  achieved.  This  would  also  reinforce  their   level  of  skill  as  well  as  provide  additional  information  for  planning  in  these  districts.    

Page 36: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

36  

Long-­‐term  Recommendations  

 

1.     Hold   planning   committees   among   districts   with   similar   sets   of   challenges   to   discuss  how  best  to  integrate  the  data  collection  process  in  the  on-­‐going  work  of  the  supervisors  and  data  collectors  in  those  districts.      

As  indicted  in  the  research  findings,  the  careful  integration  of  LQAS  data  collection  with  the  work  of  the  data  collectors  and  respondents  allows  one  to  reduce  stress  and  improve  quality  as  well   as   perhaps   address   some   other   issues.   For   example,   the   budgets   used   to   support  normal   health   service   delivery   could   support   most   of   the   transportation   costs,   which  precludes   having   a   separate   LQAS   budget.     These   meetings   should   involve   the   different  sectors  in  the  district  that  would  find  the  data  useful,   i.e.  more  than  just  the  health  sector,  so  that  there  is  commitment  built  in  the  district  for  this  exercise.      

 

2.    Begin  to  train  new  data  collectors  and  supervisors  in  those  districts  that  as  yet  have  not  had  any  exposure  to  LQAS.      

A  major   resource   is   not   only   those   supervisors   and   data   collectors   that   have   participated  four  times  but  also  the  NLFs  that  have  been  trained.    This  elite  group  should  be  involved  in  the   planning   and   transfer   of   knowledge   across   districts.     But  when  decisions   are  made   to  transfer   knowledge   to   new  districts,   it  would   be   important   to   include   in   the   planning   the  assignment  of  enough  funds  to  ensure  two  rounds  of  data  collection  even  if  they  should  be  two   years   apart.     As   indicated   in   the   previous   recommendation,   this   is   necessary   to   build  support  within  the  district  for  this  process.      

 

3.      Every  several  years  train  the  newly  appointed  DHOs  and  Assistant  DHOs  in  the  LQAS  process;  if  resources  permit,  retrain  all  district  managers  to  reinforce  LQAS  as  a  part  of  the  planning  process.  

This   might   be   easily   combined   with   the   second   recommendation   depending   upon   how  quickly  the  roll-­‐out  of  the  transfer  of  knowledge  takes  place.    In  any  case,  it  is  important  that  this  exposure  be  continued  in  the  districts  that  have  already  utilized  LQAS.      

 

In  summary  for  little  cost,  LQAS  can  be  sustained  and  extended  to  those  districts  that  have  not  yet  had  either  enough  experience  with  the  system  or  none.    

Page 37: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

37  

APPENDICES  

 

Appendix  A:  Discussion  of  the  Research  Design  

 

The  selection  criteria  for  both  regions  and  districts  within  them  were  as  follows.    The  central  region  was  excluded  because  of  the  high  rate  of  urbanization  and  development.    In  contrast,  the  northern  region  was  excluded  because  a  major  USAID  project,  NU-­‐HITES,  started  only  in  2012   and   will   continue   until   2017.     Therefore,   evaluating   institutionalization   and  sustainability   is  premature.  This   left  the  western  region,  the  south-­‐western  region,  and  the  eastern  region.    At  the  time  several  other  regions  such  as  the  Nile  or  northeast  regions  were  not   suggested.     The   choice  of  districts  was  designed   to  maximize   variation  within   regions.  Table  1   indicates   that  despite   the   interpretation  that   there  are  major  differences  between  regions,   there   is   also   considerable   variation  within   them   as  well.   Although   this   table   uses  data   collected   in   2012,   four  of   the   selected  districts   also   collected  data   in   2013.     The   five  exceptions  are  Kabarole,  Mbarara  and  Tororo,  whose   implementing  partner,  Sunrise-­‐-­‐OVC,  did  not  have  enough  money  to  fund  another  round  of  data  collection,  and  Hoima,  which  had  funding   for   only   one   year   from   the   CSF.     The   other   exception   is   Kaberamaido,  where   the  next  round  of  LQAS  is  scheduled  for  2014.    Indeed,  twenty  districts  out  of  the  72  reported  in  Table  A.1  did  not  have  funding  in  2013.    

 

Another  consideration  in  the  research  design  was  the  possibility  that  there  were  also  major  variations   across   the   sub-­‐districts.     The   LQAS   system  of   sampling   households   in   villages   is  designed  to  capture  this  distinctive  form  of  variation  and  signal  it  with  red  flags  to  indicate  where   specific  problems  exist.  Both   the  data  collectors  and   the   supervisors  were  asked   to  report  if  significant  differences  existed  within  parishes  or  sub-­‐counties  in  the  areas  in  which  they  worked.   The   reasoning   behind   this  was   that   perhaps   there  was  more   civil   society   in  some   local  communities   than  others,  which  could  be   important   for  overcoming  challenges  and  increasing  the  effectiveness  of  service  delivery.    A  few  significant  differences  emerged  in  the   context   of   questions   about   the   obstacles   to   collecting   LQAS   data.     But   the   more  important  issue  is  whether  across  these  small  areas  variations  in  the  amount  of  civil  society  or  willingness  to  cooperate  exists.  This  is  of  special  interest  given  the  SDS  fund  as  well  as  the  Community   Health   Alliance   Uganda   (CHAU).     In  most   cases,   both   the   data   collectors   and  supervisors   agreed   that   there   was   not   much   difference   in   the   extent   of   cooperation.    Neighbors  are  cooperative  and  help  each  other  with  funerals  and  transportation.  However,  the  few  exceptions  were  of  some  interest.  One  community  in  the  eastern  region  where  Plan  International  had  been  working  for  some  time  was  considered  very  different  from  the  other  communities.    This  suggests  that  in  future  research  the  number  of  red  flags  or  their  absence,  the  gold  stars,  in  small  areas  be  connected  to  specific  interventions  of  organizations  such  as  Plan,  International,  World  Vision,  UNICEF  or  a  very  important  one  that  emerged  in  the  data  collection,  Baylor  Medical  School  training  of  VHTs.    A  common  theme  in  the  four  cases  that  

Page 38: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

38  

should  be  explored  in  future  research  on  the  efficacy  of  the  LQAS  methodology  is  whether  their   interventions   build   an   even   stronger   civil   society   because   they  provide  programs   for  children.   As   a   consequence,   it   is   possible   that   people   in   the   villages   respond  more   to   the  various   initiatives   of   the   district   office   to   improve   their   health,   especially   the   support   of  VHTs.     In   other   words,   are   the   VHTs   more   effective   when   those   INGOs   or   international  projects  emphasize  children’s  well  being?      One  district,  Mbarara  mentioned  the  differences  between  urban  and  rural  areas  in  the  extent  of  cooperation,  which  is  to  be  expected.    Since  there  were  no  major  differences  between  sub-­‐districts,  this  data  is  not  reported.  However,  it  should  be  understood  that  the  degree  of  cooperation  becomes  a  hidden  resource  that  allows  the  collection  of  LQAS  data  as  well  as  health  district  interventions  such  as  sensitization  by  the  VHTs  to  be  more  effective  than  they  would  otherwise  be.  At  various  points  in  the  interviews  the   importance   of   being   able   to   obtain   the   cooperation   of   the   village   chiefs   was   cited.    Although   sub-­‐district   variation   does   not   explain   differential   effectiveness   of   LQAS,   it   is  important  for  international  donor  agencies  such  as  USAID  to  understand  that  the  success  of  LQAS   in   Uganda   may   not   be   easily   transferred   to   other   countries   where   cooperative  behavior  is  low.        

       

But   while   the   concentration   of   data   collection   was   on   the   district   level   of   services,   the  central  government  was  not  ignored.    Rather  than  ask  questions  in  the  central  government  looking  down  on  the  districts,  the  approach  was  to  ask  questions  in  the  districts  about  how  they   interfaced   with   the   central   government   in   the   use   of   LQAS   data.   This   would   be   in  alignment  with   the   decentralized  model   of   the   Ugandan   government.     Several   interviews  were  also  conducted  at  the  central  government  level  to  obtain  some  additional  information  about  the  challenges  of  sustainability.  

 

Table  A.1  

Percent  of  Women  in  Districts  Giving  Birth  in  a  Health  Facility    

By  Year  of  Data  Collection  and  Region  

 

District   Percent   Sample  size  2012  Western  Buhweju   57.3   96  Buliisa   56.8   95  Bundibugyo   61.1   95  Bushenyi   88.4   95  Hoima   63.7   113  Ibanda   62.8   113  Isingiro   64.2   95  Kabale   70.5   132  Kabarole   80.5   113  Kamwenge   56.8   95  

Page 39: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

39  

District   Percent   Sample  size  Kanungu   80.0   95  Kasese   71.6   95  Kibaale   59.5   131  Kiruhura   70.5   95  Kisoro   67.5   114  Kyenjojo   56.1   114  Masindi   57.9   114  Mbarara   78.9   114  Mitoma   83.2   95  Ntungamo   71.1   114  Rubirizi   54.2   96  Rukungiri   75.8   95  Sheema   80.0   95  Northern  Adjumani   72.6   95  Apac   71.9   114  Arua   49.5   95  Koboko   54.7   95  Kotido   63.2   95  Nakapiripirit   21.1   95  Nebbi   69.5   95  Eastern  Amuria   59.6   114  Budaka   60.0   95  Bududa   48.1   133  Bukedea   66.7   93  Bukwa   24.0   96  Bulambuli   44.2   95  Busia   64.2   95  Butaleja   63.2   114  Jinja   83.5   133  Kaberamaido   64.2   95  Kapchorwa   51.6   95  Katakwi   65.6   131  Kibuku   57.9   95  Kumi   84.2   114  Kween   10.5   95  Manafwa   42.1   114  Mbale   67.7   133  Pallisa   65.3   95  Sironko   56.8   95  Soroti   68.9   132  Tororo   50.9   114  Central  Kalangala   80.0   95  Kayunga   61.7   94  Kiboga   84.2   95  Luwero   90.4   114  Lwengo   61.9   113  Masaka   74.6   114  Mityana   82.1   95  Mpigi   81.1   95  Mubende   54.1   133  

Page 40: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

40  

District   Percent   Sample  size  Mukono   76.7   133  Nakasongola   76.8   95  Rakai   64.2   95  Ssembabule   72.6   95  2011  Western  Buhweju   41.7   96  Bushenyi   84.2   95  Ibanda   59.3   113  Isingiro   57.9   95  Kabale   59.8   132  Kabarole   71.9   114  Kamwenge   47.9   94  Kanungu   66.3   95  Kasese   71.6   95  Kiruhura   65.3   95  Kisoro   70.2   114  Kyenjojo   55.3   114  Mbarara   73.7   114  Mitoma   73.7   95  Ntungamo   71.9   114  Rubirizi   66.7   96  Rukungiri   75.8   95  Sheema   75.8   95  Northern  Arua   71.3   94  Nebbi   75.8   95  Eastern  Budaka   53.2   94  Bududa   43.6   133  Bukwa   21.9   96  Bulambuli   33.7   95  Busia   60.0   95  Butaleja   61.6   112  Jinja   85.6   132  Kapchorwa   46.2   93  Kibuku   60.6   94  Kumi   81.4   113  Kween   28.0   93  Manafwa   43.9   114  Mbale   57.6   132  Pallisa   71.6   95  Sironko   46.3   95  Tororo   49.1   114  Central  Kalangala   81.1   95  Kayunga   69.0   87  Luwero   83.3   114  Masaka   77.9   113  Mityana   73.4   94  Mpigi   85.1   94  Nakasongola   70.2   94  Ssembabule   64.9   94  

Page 41: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

41  

District   Percent   Sample  size  2010  Western  Buhweju   36.5   96  Bushenyi   76.8   95  Ibanda   66.4   113  Isingiro   58.9   95  Kabale   51.1   133  Kanungu   65.3   95  Kasese   58.9   95  Kiruhura   61.1   95  Kisoro   59.6   114  Mitoma   64.2   95  Ntungamo   69.3   114  Rubirizi   75.0   96  Rukungiri   68.4   95  Sheema   74.7   95  Eastern  Budaka   53.3   92  Bududa   32.3   130  Bukwa   16.7   96  Busia   49.5   91  Butaleja   57.8   109  Kapchorwa   30.1   93  Mbale   59.2   130  Pallisa   66.1   112  Sironko   42.6   94  2009  Eastern  Busia   49.0   96  Butaleja   51.8   114  Pallisa   63.2   114  Sironko   48.4   95  2006  Western  Bushenyi   53.2   94  Kabale   44.2   95  Masindi   47.4   95  Northern  Arua   34.6   133  Lira   34.5   113  Eastern  Kamuli   69.1   94  Mayuge   66.0   94  Mbale   52.1   94  Central  Kampala   89.4   94  Kayunga   52.6   95  Masaka   66.0   94  Mukono   63.2   95  2004  Western  Hoima   41.1   95  Kabarole   52.6   95  

Page 42: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

42  

District   Percent   Sample  size  Northern  Apac   35.8   95  Kitgum   31.6   95  Moyo   44.2   95  Nakapiripirit   21.1   95  Eastern  Jinja   82.1   95  Kaberamaido   25.3   95  Soroti   52.6   95  Tororo   34.7   95        Central  Kalangala   67.4   95  2003  Western  Bushenyi   58.8   114  Kabale   37.9   95  Kamwenge   15.8   95  Kyenjojo   31.0   113  Masindi   37.9   95  Mbarara   37.9   169  Northern  Arua   25.3   170  Lira   40.9   132  Eastern  Iganga   61.1   95  Kamuli   69.3   114  Mayuge   48.9   94  Mbale   43.2   95  Sironko   34.7   95  Central  Kampala   94.7   114  Kayunga   49.5   95  Masaka   58.8   114  Mukono   67.5   114  Rakai   43.6   94  Wakiso   87.2   94  

 

Page 43: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

43  

Figure  A.1  

The  Location  of  the  Selected  Nine  Districts  

 

 

 

 

Page 44: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

44  

Table  A.2  

Number  of  DHOs,  Assistant  DHOs  (MCH),  Supervisors    

and  Data  Collectors  Interviewed  in  Each  District  

 

 

 

District  

DHO   Assistant  DHO   Supervisors   Data  Collectors  

Mbale   Yes   Yes   5   4  

Kaberamaido   New**   Not  appointed   6   3  

Tororo   Yes   Yes   4   5  

Kaberole   DHE   No   1   5  

Kamwenge   Not  appointed   Yes   2   5  

Hoima   DHE   New   2   3  

Kabale   Yes   Yes*   4   5  

Bushenyi   Yes   Yes   4   4  

Mbarara   Yes*   Yes   3   4  

Total   7   6   31   38  

*     telephone  interview  

**   short  interview  about  strategies  regarding  LQAS  

 

 

 

Page 45: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

45  

 

Table  A.3  

The  Duties  of  Interviewed  Data  Collectors  and  Supervisors  in  Their  Districts  

 

Sectors  

District   Health   Community  

Development  

District   Other*   Total  

Mbale   4   4   0   1   9  

Kameramaido   2   5   2   0   9  

Tororo   2   6   0   1   9  

Kabarole   3   3   0   0   6  

Kamwenge   5   0   0   2   7  

Hoima   2   1   0   2   5  

Kabale   5   0   2   2   9  

Bushenyi   4   2   0   2   8  

Mbarara   3   2   0   2   7  

Total   30   23   4   12   69  

*   includes   parish   chief,   probation   officer,   several   education   officers,   a   secretary,   senior  information  scientist,  sports,  etc.  

Page 46: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

46  

Appendix  B:    DHO  and  Assistant  DHO   Interview  Questionnaires,  Focus  Group  Schedules,  and  Surveys  for  Supervisors  and  Data  Collectors  

Figure  B.1  

Measures  of  Institutionalization  of  LQAS:      

Training  and  Experience,  Coordination  and  Social  Control    

 

Training  and  Experience    

Data  Collectors  

1.    When  did  you  receive  your  basic  LQAS  training?         In  which  areas:         year             year     Reproductive  health       Tuberculosis     Child  health         Sexually  transmitted  diseases     Nutrition         Sanitation     Malaria         Other  (please  specify)    2.    Circle  each  year  in  which  you  participated  in  data  collection  with  LQAS:    2013       2012         2011     2010     2009        

3.    Have  you  had  any  LQAS  refresher  training?      When?  

Supervisors  

1.  What  are  your  duties  as  a  supervisor?  

2. When  did  you  receive  your  supervisory  training?           In  what  areas:       year             year       Reproductive  health       Tuberculosis     Child  health         Sexually  transmitted  diseases     Nutrition         Sanitation     Malaria         Other  (please  specify)    3.    Circle  each  year  in  which  you  participated  in  data  collection  with  LQAS:    2013       2012         2011     2010     2009      3.    Have  you  had  any  LQAS  refresher  training  in  supervision?    When   4.     In   your  opinion  how  competent   are   the   LQAS  data   collectors   that   you   supervised?       If  good,  Could  they  work  without  supervision  in  the  next  round  of  data  collection?  

Page 47: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

47  

   DHOs  and  Assistant  DHOs  (MCH):  

1.     Please   indicate   the   year   you  were   first   exposed   to   LQAS?       1a.What   was   the   level   of  exposure?    1b.Did  you  receive  any  training  in  LQAS?  

Coordination:    Manuals,  Quality  Control,  and  Planning  

Data  Collectors:  

1.    Are  there  manuals  available  that  you  can  refer  to  when  collecting  data?  

Supervisors:  

1.    Is  there  a  manual  with  clear  guidelines  for  the  supervision  of  LQAS?  

2.    What  kinds  of  checks  have  you  developed  to  verify  the  quality  of  the  data?  

DHOs  and  Assistant  DHOs:  

1.    Have  you  used  LQAS  results  for  making  decisions  about  program  improvement  after  any  of   these   reports   were   received?   If   yes,   in   what   year   and   how   were   they   used   how   and  when?    If  not  what  are  the  underlying  cases  that  deterred  you  from  not  using?  

 

 

 

 

 

Page 48: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

48  

Figure  B.2  

Measures  of  Challenges  to  Institutionalization  

Data  Collectors  

1.    It  is  my  understanding  that  you  have  multiple  responsibilities.  Please  indicate  what  your  duties  are  in  this  district.    How  much  time  do  you  spend  in  these  activities?  *    2.  Did  you  encounter  any  problems  in  collecting  the  data?        

2a.  If  yes,  what  were  these?            

2b.    Do  you  have  suggestions  for  how  these  problems  might  be  solved?3.  

3..    Does  collecting  the  data  across  59  indicators  present  any  special  problems?  

Supervisors  

1.   It   is  my  understanding  that  you  have  multiple  responsibilities.  Please   indicate  what  your  duties  are  in  this  district.      How  much  time  do  you  spend  in  these  activities?*      2.  What  are  the  obstacles  or  problems  that  you  face  in  your  supervision  work?        2a  If  yes,  what  are  these?      2b.    Do  you  have  any  suggestions  as  to  how  these  might  be  handled?    3.  Does  reporting  the  data  across  59  indicators  present  any  special  problems?    

DHOs  and  Assistant  DHOs:  

1.    How  good  has  been   the   technical   support  of   the  Ministry  of   Local  Government  and/or  Ministry  of  Health  in  carrying  out  LQAS  in  your  districts?**  

2.    Do  you  have  any  suggestions  on  how  LQAS  results  could  be  utilized  more  effectively   in  making  decisions?  

3.    In  the  last  three  years,  2012,  2011,  and  2010,  was  the  data  from  your  district  transmitted  to  the  Ministry  of  Health  or  the  Ministry  of  Local  Government?     If  yes,  ask  for  each  year   if  there  were  any  problems?  

4.    Are  you  aware  of  any  times  that  the  data  from  your  district  has  been  used  by  the  Ministry  of  Health  or   the  Ministry  of  Local  Government   for  planning  or  making  policy?     If   so,  when  and  for  what  policy?  

*                We  had  assumed  that  they  were  part-­‐time.    This  is  not  the  case,  so  this  question  really  became  unnecessary  and  confusing.  

**   Dropped   when   we   realized   that   the   respondents   were   talking   about   STAR-­‐E.

Page 49: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

49  

Figure  B.3  

Measures  of  LQAS’s  Impact  on  Services  and    Challenges  Affecting  the  Impact      Reports  of  Data  Collectors  in  Focus  Groups  

1.    Did  you  help  present  the  data  collected  in  2013  to  the  local  health  district  officer  or  any  other  member  of  the  district  health  management  team?          Which  ones?    Were  members  of  the  implementing  partner  present?  

2.    With   the  presentation  of   the  data,  did  you  help  make   recommendations  about  how  to  improve  the  strategies  for  intervention?    In  what  areas?  

What  recommendations?  

3.    Did  the  implementing  partner  indicate  that  they  would  accept  your  recommendations?  

Reports  of  Supervisors  in  Focus  Groups  

1.    Did  you  help  present  the  data  collected  in  2013  to  the  local  health  district  officer  or  any  other  member  of  the  district  health  management  team?          Which  ones?      Were  members  of  the  implementing  partner  present?  

2.    With   the  presentation  of   the  data,  did  you  help  make   recommendations  about  how  to  improve  the  strategies  for  intervention?  

In  what  areas?    What  recommendations?  

3.    Did  the  implementing  partner  indicate  that  they  would  accept  your  recommendations?  

Which  ones?  

Reports  of  DHO  

1.   In   what   years   did   you   receive   LQAS   reports   with   information   on   antenatal   visits   and  deliveries  in  Health  Facilities:  

[circle  answer]  

2013         2012         2011       2009  

2.    Have  you  used  LQAS  results   for  making  decisions  about  programme  improvement  after  any  of  these  reports  were  received?       If  yes,   in  what  year  and  how  were  they  used.     If  not  what  are  the  underlying  causes  that  deter  you  for  not  using?  

3.        Do  you  have  any  suggestions  on  how  LQAS  results  could  be  utilized  more  effectively  in  making  decisions?  

 

 

Page 50: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

50  

Reports  of  Assistant  DHOs:  

1.    In  which  years  did  you  receive  LQAS  reports  with  information  on  children  being  taken  to  a  hospital?       [circle  answer]  

2013         2012         2011       2009  

Were   recommendations   made   for   changes   in   district   level   strategies   and   tactics   made?    Which  of  these  were  adopted  and  when?  

2.     In   which   years   did   you   receive   LQAS   reports   with   information   on   children   being  vaccinated?  

 [circle  answer]  

2013         2012         2011       2009  

Were  recommendations  made  for  changes  in  strategies  and  tactics?      Which  of  these  were  adopted  and  when?    Did  you  find  the  implementing  partner  responsive  to  recommendations  from  LQAS?    If  recommendations  were  not  adopted,  why?  

 

Page 51: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

51  

Figure  B.4  

Organizational  Learning  and    

Challenges  To  Learning    

DHOs:    Safe  Motherhood  Learning  

1.    What  was  the  original  strategy  of  the  health  care  office  and  of  the  implementing  partner  to  encourage  women  to  have  pre-­‐natal  visits?      

2.  Were  there  any  changes  in  this  plan  for  pre-­‐natal  visits?    If  so,  what  were  these?  

For  each  change,  ask:  

a.    When?  Specify  month  and  year  

b.    Why?  

c.    What  was  the  actual  change?  

d.    Whom  or  what  was  the  source  of  the  idea?  

3.    What  was  the  original  strategy  of  the  health  care  office  and  of  the  implementing  partner  to  encourage  women  to  give  birth  in  hospitals  or  clinics?      

4.    Were  there  any  changes  in  this  plan  for  encouraging  mothers  to  give  birth  in  hospitals  or  clinics?    If  so,  what  were  these?  

For  each  change,  ask:  

a.    When?  Specify  month  and  year  

b.    Why?  

c.    What  was  the  actual  change?  

d.    Who  or  what  was  the  source  of  the  idea?  

5.    Over  the  course  of  the  last  three  years,  how  often  have  you  met  with  other  district  health  officers  to  discuss  strategies  for  encouraging  mothers  pre-­‐natal  visits  and  having  children  in  a  health  facility?    How  many  and  which  districts  were  involved?    What  did  you  learn  in  these  meetings?    

Assistant  DHOs:    Child  health  Learning  

1.  What  was  the  original  strategy  of  either  the  implementing  partner  or  of  the  local  health  district   to   encourage   women   to   take   their   children   when   sick   to   a   hospital   or   seek  assistance:      

2.  Were  there  any  changes  in  this  plan?    If  so,  what  were  these?  

Page 52: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

52  

For  each  change,  ask:  

a.    When?  Specify  month  and  year  

b.    Why?  

c.    What  was  the  actual  change?  

d.    Who  or  what  was  the  source  of  the  idea?  

3.    What  were  the  strategies  of  either  the  implementing  partner  or  of  the  health  district  to  encourage  women  to  have  their  children  receive  vaccines?    

4.  Were  there  any  changes  in  this  plan?    If  so,  what  were  these?  

For  each  change,  ask:  

a.    When?  Specify  month  and  year  

b.    Why?  

c.    What  was  the  actual  change?  

d.    Who  or  what  was  the  source  of  the  idea?  

5.    Over  the  course  of  the  last  three  years,  how  often  have  you  met  with  other  district  health  officers  to  discuss  strategies  for  encouraging  taking  to  children  to  hospitals  when  sick  or  to  increase  the  level  of  immunization?    How  many  and  which  districts  were  involved?  What  did  you  learn  in  these  meetings?    Supervisors  and  Data  Collectors:    Additional  Learning  About  LQAS  and  Services    1.    Over  the  course  of  the  last  three  years,  how  often  have  you  met  with  [supervisors]  [data  collectors]   from   other   districts   to   discuss   strategies   for   data   collection,   especially   about  reproductive   behavior   and   child   health   (pre-­‐natal   visits,   births   in   hospitals   and   clinics,  seeking  advice  for  sick  children,  immunization  rates)?    

If  yes,  ask:    how  many  times?  For  each  event,  when?          Which  districts  were  involved?    What  did  you  learn  in  these  meetings?  Were  any  of  these  events  at  the  national  level?  

 

Page 53: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

53  

Figure  B.5  

Measures  of  Sustainability  and  Challenges  to  Sustainability  

First  Strategy:    Integration  with  On-­‐going  Work  

Asked  of  Data  Collectors:  

1.    Could  you  slowly  integrate  your  LQAS  data  collection  into  your  routine  work?    If  Yes,  how  could  this  be  done?     If  not,  why?      What  are  the  obstacles?    How  might  these  obstacles  be  overcome?  

Asked  of  Supervisors:  

1.     Do   you   think   your   supervision   duties   could   be   gradually   integrated   with   your   other  duties?  If  Yes,  how  could  this  be  done?    If  not,  why?      What  are  the  obstacles?    How  might  these  obstacles  be  overcome?  

Asked  of  Assistant  DHOs:  

1.    Is  it  possible  to  integrate  the  LQAS  data  collection  process  in  routine  work  in  the  health  care  district  over  time?      How  would  you  suggest  this  be  done  in  your  district?  

Asked  of  DHOs:  

1.     Is   it  possible   to   integrate   the  LQAS  data  collection  process   in   routine  work  within  your  health  care  district  over  time?      How  would  you  suggest  this  be  done  in  your  district?  

Second   Strategy:     Diffusion   of   LQAS   from   Districts   with   a   Great   Deal   of   Experience   to  Districts  with  Little  or  No  Experience    

Asked  of  Data  Collectors:  

1.    Do  you  think  you  could  train  other  individuals  to  be  data  collectors?    How  do  you  suggest  this    be  done?  

Asked  of  Supervisors:  

1.    Do  you  think  you  could  train  other  individuals  to  be  supervisors?  How  do  you  suggest  this  be  done?  

Third  Strategy:    Gradually  Phasing  Out  of  LQAS  with  Data  Collected  at  Health  Facilities    

Asked  of  Assistant  DHOs:  

1.    At  what  percentage  of  pre-­‐natal  visits  for  women  in  your  district  do  you  think  it  would  no  longer  be  necessary  to  use  LQAS  to  determine  the  utilization  rate?    Could  this  be  effectively  recorded  by  the  hospitals  and  clinics  involved?  

2.    At  what  percentage  of  birth  deliveries  in  hospitals  or  clinics  in  your  district  do  you  think  it  would  no  longer  be  necessary  to  use     LQAS   to  determine   the  utilization   rate?     Could   this  be  effectively  recorded  by  the  hospitals  and  clinics  involved?    

Page 54: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

54  

Asked  of  DHOs:  

1.    At  what  percentage  of  pre-­‐natal  visits  for  women  in  your  district  do  you  think  it  would  no  longer  be  necessary  to  use  LQAS  to  determine  the  utilization  rate?    Could  this  be  effectively  recorded  by  the  hospitals  and  clinics  involved?  

2.    At  what  percentage  of  birth  deliveries  in  hospitals  or  clinics  in  your  district  do  you  think  it  would  no  longer  be  necessary  to  use  LQAS  to  determine  the  utilization  rate?    Could  this  be  effectively  recorded  by  the  hospitals  and  clinics  involved?    

Page 55: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

55  

Appendix  C  Monitoring  Report  on  LQAS/HFA  survey  data  use  by  districts      

Summary   table   showing   how  districts  are  utilizing  LQAS  and  HFA  survey  results  How  the  data  were  used    

Number  of  District   reporting  use  of  data    

Districts  reporting  use  of  data    

 1.   In   issuing   of   policy   by-­‐laws   and  ordinances      

12     Amuria,  Buhweju,  Bulambuli,  Bukwo,  Buyende,   Ibanda,   Isingiro,  Sembabule,  Kayunga,  Kumi,  Masaka,  Sheema    

 2.  During  planning      

35     Amuria,  Bududa,  Buhweju,  Bukedea,  Bulambuli,   Bushenyi,   Buyende,  Kibaale,   Bukwo,   Hoima,   Ibanda,  Isingiro,   Kabale,   Kaliro,   Kamuli,  Kamwenge,   Kanungu,   Kasese,  Katakwi,   Kayunga,   Kiruhura,   Kween,  Masaka,  Mbale,  Mubende,  Mitooma,  Mityana,   Nakasongola,   Namutumba,  Ntugamo,  Pallisa,  Rukungiri,  Sheema,  Sironko,  Sembabule    

 3.  During  budgeting      

31     Amuria,  Bududa,  Buhweju,  Bukedea,  Bukwo,   Bulambuli,   Buyende,   Hoima,  Ibanda,   Isingiro,   Kaliro,   Kamuli,  Kanungu,  Katakwi,  Kibaale,  Kiruhura,  Masaka,  Mbale,  Mityana,  Mubende,  Namutumba,   Ntungamo,  Nakasongola,   Kabale,   Kamwenge,  Kasese,   Kayunga,   Rukungiri,  Sembabule,  Sheema,  Sironko    

 4.  During  budget  allocation      

28     Amuria,   Bududa,   Buhweju,  Bulambuli,   Bukedea,   Bukwo,  Bushenyi,   Kibaale,   Hoima,   Ibanda,  Isingiro,   Kanungu,   Kaliro,   Kamuli,  Kamwenge,   Kanungu,   Katakwi,  Kasese,   Kayunga,   Kiruhura,   Masaka,  Mbale,   Mubende,   Mityana,  Namutumba,   Nakasongola,  Sembabule,  Sheema    

 5.   In   resource   mobilization,  distribution  or  allocation      

31     Amuria,   Bududa,   Buhweju,  Bulambuli,   Bukedea,   Bukwo,  Bushenyi,   Busia,   Buyende,  Sembabule,   Ibanda,   Isingiro,   Kabale,  Kaliro,  Kamuli,  Kamwenge,  Kanungu,  Kasese,   Katakwi,   Kayunga,   Kibaale,  Kiruhura,  Masaka,  Mbale,  Mubende,  Mityana,   Nakasongola,   Namutumba,  Ntungamo,  Rukungiri,  Sheema    

Page 56: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

56  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 6.  In  aligning  partners  to  red  flagged  areas      

26     Amuria,   Bududa,   Buhweju,  Bulambuli,   Bukedea,   Buyende,  Hoima,   Ibanda,   Isingiro,   Kaliro,  Kamwenge,   Kanungu,   Kamuli,  Kasese,   Katakwi,   Kayunga,   Kibaale,  Kiruhura,   Masaka,   Mubende,  Mityana,   Nakasongola,   Ntungamo,  Rukungiri,  Sembabule,  Sheema    

 7.   In   advocacy   and   community  mobilization      

24     Amuria,   Bududa,   Bulambuli,  Bukedea,   Buyende,   Hoima,   Ibanda,  Isingiro,   Kabale,   Kanungu,   Kaliro,  Kamuli,    

Page 57: Institutionalizing and Sustaining the Lot Quality ... 8-14.pdf · However,!Bushenyi,which!cored!100%!on!both!indicesor!high!institutionalization;!while!it! has!a!relative!low!number!of!HFs!it!does!have!a!very!highpercentage!of

57  

REFERENCES:  

Argote,  Linda  and  Miron-­‐Sepktor,  Ellen  2011    Organizational  Learning:    From  experience  to  knowledge    Organization  Science,  25  (5),  September-­‐October:    1123-­‐1137  

Hage,   Jerald   1972   Techniques   and   Problems   of   Theory   Construction   in   Sociology,   Wiley  Interscience,  New  York.  

Hage,  Jerald  1974  Communication  and  Organizational  Control:  A  Cybernetics  Perspective  in  a  Health  and  Welfare  Setting,  Wiley-­‐Interscience,  New  York.  

Mas   de   Xaxas,   Mercedes   and   Vogel,   Carolyn   Gobel   2007   Making   Country   Ownership   a  Reality:    An  NGO  Perspective  Population  Action  International,  2  (3)  July  

Stash,  Sharon  2012  Competing  Pressures  for  US  PEPFAR  in  Botswana:    Rising  Ambitions  and  Declining  Resources  CSIS  Report  Center  for  Strategic  and  International  Studies,  November.  

Valadez,  J.J.,  Jerald  Hage,  William  Vargas.  2005  “Understanding  the  Relationship  of  Maternal  Health  Behavior  Change  and   Intervention  Strategies   in  a  Nicaraguan  NGO  Network.”  Social  Science  and  Medicine,  61,:  1356-­‐1368.  

USAID  2013  Meeting  Report  Advancing  Country  Ownership:    Civil  Society’s  Role  in  Sustaining  Public  Health  USAID,  Washington,  D.C.  June