INTERVENTION GUIDEBOOK a

InterventionGuidebook

for implementing and monitoring activities to reduce Missed Opportunities for Vaccination

b

InterventionGuidebook

for implementing and monitoring activities to reduce Missed Opportunities for Vaccination (MOV)

Intervention guidebook for implementing and monitoring activities to reduce Missed Opportunities for Vaccination (MOV)

ISBN xxxxxxxx

© World Health Organization 2019

Some rights reserved. This work is available under the Creative Commons Attribution-NonCommercialShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo).

Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes, provided the work is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO endorses any specific organization, products or services. The use of the WHO logo is not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following disclaimer along with the suggested citation: “This translation was not created by the World Health Organization (WHO). WHO is not responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition”.

Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules of the World Intellectual Property Organization.

Suggested citation. Intervention guidebook for implementing and monitoring activities to reduce Missed Opportunities forVaccination (MOV). Geneva: World Health Organization; 2019. Licence: CC BY-NCSA 3.0 IGO.

Cataloguing-in-Publication (CIP) data. CIP data are available at http://apps.who.int/iris.

Sales, rights and licensing. To purchase WHO publications, see http://apps.who.int/bookorders.

To submit requests for commercial use and queries on rights and licensing, see http://www.who.int/about/licensing.

Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables, figures orimages, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third party owned component in the work rests solely with the user.

General disclaimers. The designations employed and the presentation of the material in this publication do not imply theexpression of any opinion whatsoever on the part of WHO concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not yet be full agreement.

The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters.

All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use.

Printed in Switzerland

INTERVENTION GUIDEBOOK i

www.who.int/immunization/ programmes_systems/policies_strategies/MOV/en/

For more information visit:

Table of ContentsPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xAcknowledgements xAcronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xAbout this document x

Introduction x

What is a missed opportunity for vaccination (MOV)? . . . . . . . . . . . . . . . . . . . . . . . . . x

Common reasons for MOV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x

Intended Use of this Intervention Guidebook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x

How do we ensure a smooth transition from action planning (Step 6) to implementation of interventions (Step 7)? x

Key resources for implementing MOV interventions . . . . . . . . . . . . . . . . . . . . . . . . . . x

Challenges and bottlenecks to implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . x

MOV-lite options instead of conducting a standard MOV assessment x

Conducting a Brainstorming Workshop to Reduce MOV . . . . . . . . . . . . . . . . . . . . . . . x

Small-scale health facility MOV assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x

STEP 7 Implement the interventions X

STEP 8 Provide supportive supervision and monitor progress X

STEP 9 Conduct rapid field evaluation of outcomes/impact of interventions X

STEP 10 Incorporate into long term immunization plans to ensure gains are sustainable X

Annexes

Annex G. Examples of facility-level practices that may result in MOV . . . . . . . . . . . . . . G-1

Annex H. Examples of promotional materials that can help to reduce MOV at the health facility level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H-1

INTERVENTION GUIDEBOOK iiiii

PrefaceMissed opportunities for vaccination (MOV) include any contact made with health services by a child (or adult) who is eligible for vaccination, but which does not result in the individual receiving all the vaccine doses for which he or she is eligible.

Today we are vaccinating more children than ever, yet millions of children still miss out on routine vaccinations. National immunization programmes continue to seek evidence-based strategies to understand the underlying reasons and to design tailored approaches to address them. Using a participatory mixed-methods approach, the MOV strategy provides step-by-step guidance on how to conduct a bottom-up root-cause analysis of bottle-necks in the immunization programme and to implement relevant interventions to address them. When applied appropriately, the steps outlined in the MOV guides have the potential to contribute to an increase in vaccination coverage and equity and an improvement in timeliness of vaccination.

This Intervention guidebook for implementing and monitoring activities to reduce Missed Opportunities for Vaccination is the third in a three-part series of the MOV strategy resource guides, and focuses on steps 7–10 of the 10-step MOV strategy – implementing, monitoring and evaluating actions to reduce MOV. This Intervention Guidebook can also be used together with the Planning Guide to Reduce Missed Opportunities for Vaccination and Methodology for the Assessment of Missed Opportunities for Vaccination or for situations where it might not be necessary to conduct a standard MOV assessment, this Intervention Guidebook can be used as a stand-alone guide.

The MOV strategy should not be viewed as a stand-alone or discrete “project”; rather as complementary to existing microplanning and programme improvement approaches such as RED (‘Reaching Every District’). The MOV strategy is conceived as a health system-wide service improvement effort targeted at improving vaccination as well as other health services within a given health facility.

To ensure sustainability, any interventions implemented to reduce MOV should be included in annual immunization plans, and the concepts behind the MOV strategy should ideally become part of the routine immunization strengthening dialogue.

For up-to-date information on the MOV strategy and the latest tools and materials, please visit: http://www.who.int/immunization/programmes_systems/policies_strategies/MOV/en/ and https://www.technet-21.org/en/topics/mov

AcknowledgmentsThis document was developed by the World Health Organization (WHO) Department of Immunization, Vaccines & Biologicals, WHO headquarters (WHO-HQ) and written by Stephanie Shendale, Laura Nic Lochlainn and Ikechukwu Udo Ogbuanu.

We are grateful to our colleagues who contributed towards this document through their unwavering support of the MOV strategy and by reviewing the early drafts and providing constructive comments, including the following (in alphabetical order): Blanche-Philomene Melanga ANYA (WHO-AFRO), Laura CONKLIN (US-CDC), Rebecca FIELDS (JSI), Tracey GOODMAN (WHO-HQ), Anyie LI (US-CDC), Lisa MENNING (WHO-HQ), Imran MIRZA (UNICEF), Laura NIC LOCHLAINN (WHO-HQ), Ikechukwu OGBUANU (WHO-HQ), Ana Maria Henao RESTREPO (WHO-HQ), Julia ROPER (CHAI), Stephanie SHENDALE (WHO-HQ), Lora SHIMP (MCSP/JSI), Aaron WALLACE (US-CDC), and Kirsten WARD (US-CDC).

Special thanks are due to the numerous other organizations and partners who contributed to the development of these documents through their membership on the MOV partner coordination platform (in alphabetical order): Agence de Médecine Préventive (AMP), the Bill and Melinda Gates Foundation (BMGF), United States Centers for Disease Control and Prevention (US-CDC), the Clinton Health Access Initiative (CHAI), Gavi, the Vaccine Alliance, John Snow, Inc. (JSI), Médecins Sans Frontières (MSF), the Pan-American Health Organization (PAHO), VillageReach, UNICEF and the ministries of health in over 20 countries across the globe.

During its April 2016 meeting, the Strategic Group of Experts on Immunization (SAGE) reviewed the initial results from the MOV pilot countries and provided valuable inputs to the methodology to make it more programmatically feasible and useful to countries at different levels of development. Similarly, two WHO advisory committees reviewed early drafts of the methodology and provided constructive feedback: the Immunizations and Vaccines related Implementation Research Advisory Committee (IVIR-AC) and the Immunization Practices Advisory Committee (IPAC).

Finally, we would like to specifically thank our colleagues at the ministries of health and WHO country offices in Chad and Malawi for allowing us to pilot the draft methodology in their respective countries in 2015.

INTERVENTION GUIDEBOOK viv

AcronymsCDC United States Centers for Disease Control and Prevention

cMYP Comprehensive multi-year plan

DTP Diphtheria-tetanus-pertussis vaccine

DHS Demographic and health survey

EPI Expanded Programme on Immunization

FGD Focus group discussion

HBR Home-based record

HMIS Health management information system

ICC Interagency coordinating committee

KAP Knowledge, attitude and practices

M&E Monitoring and evaluation

MICS Multiple indicator cluster survey

MOH Ministry of Health

MOV Missed opportunities for vaccination

NGO Non-governmental organization

RED Reaching every district (strategy)

UNICEF United Nations Children’s Fund

WHO World Health Organization

WUENIC WHO/UNICEF estimates of national immunization coverage

About this documentThe Intervention guidebook for implementing and monitoring activities to reduce Missed Opportunities for Vaccination provides advice on how to ensure a smoothtransition between Steps 6 and 7, and guidance to countries in implementing the final steps (Steps 7–10) of the 10-step MOV strategy, as outlined in the MOV Planning Guide1, and listed below. These steps will help putting the assessment findings into practice.

The Intervention Guidebook provides tips for planning actions to reduce or prevent MOV, even when a full MOV assessment has not been conducted. Alternative assessment options are also described, such as the MOV-lite option (e.g. integrating a MOV component into another planned assessment, conducting a MOV workshop informed by data already available in-country or a small-scale health facility MOV assessment). This guide also includes frequent reasons for MOV, potential interventions to reduce MOV, examples of job aids and other materials for use at the health facility level.

1 Planning Guide to Reduce Missed Opportunities for Vaccination. Geneva: World Health Organization; 2017. (https://apps.who.int/iris/bitstream/handle/10665/259202/9789241512947-eng.pdf)

PLAN AND PREPARE

STEP 1 Plan for a MOV assessment and intervention

STEP 2 Prepare for the assessment and secure commitment for follow-up interventions

IMPLEMENT AND MONITOR

STEP 7 Implement the interventions

STEP 8 Provide supportive supervision and monitor progress

STEP 9 Conduct rapid field evaluation of outcomes/impact of interventions

STEP 10 Incorporate into long term plans to ensure gains are sustainable

FIELD WORK

STEP 3 Conduct field work for the rapid assessment of MOV

STEP 4 Analyze preliminary data and identify key themes

STEP 5 Brainstorm on proposed interventions and develop an action plan for the interventions

STEP 6 Debrief with MOH leadership and immunization partners on proposed next steps

INTERVENTION GUIDEBOOK 1vi

This Intervention Guidebook is the third component of the MOV strategy resource guides that have been developed to reduce MOV:

1. Planning Guide to Reduce Missed Opportunities for Vaccination (“PlanningGuide”): Intended for use by decision-makers and programme managers at nationaland sub-national levels. This manual provides an overview of the MOV strategy, whichinvolves an assessment to demonstrate the magnitude and identify causes of MOV,followed by tailored health system interventions to reduce these MOV, leading to anincrease in vaccine coverage and timeliness of vaccinations.

2. Methodology for the Assessment of Missed Opportunities for Vaccination(“Methodology”)2: This manual provides the detailed instructions, standardizedmethodology, and tools for conducting MOV assessment field work (including generichealth facility exit interviews and health worker knowledge, attitudes, and practices(KAP) questionnaires). The manual also includes detailed guidance for conducting in-depth interviews and focus group discussions. Although it may be desirable in some countries to obtain an estimate of the proportion of MOV in health facilities, this should not be the focus of the assessment as the proposed simplified sampling methodology does not produce a statistically robust estimate. Therefore, the major outcome of the MOV assessment field work is rather to build a strong advocacy case for reducing MOV by convening in-country brainstorming sessions with core immunization partners to identify the underlying causes and address these problems. The brainstorming sessions following the field work are intended to achieve this outcome.

Note: In some situations, it may not be necessary to conduct a standard MOV assessment. Countries, districts or health facilities may have existing evidence of MOV as an issue and there may already be sufficient support for reducing MOV to improve coverage and equity. In such circumstances, programmes may choose to move directly to implementation of locally-tailored interventions to reduce MOV using guidance provided in this Intervention Guidebook.

3. Intervention guidebook for implementing and monitoring activities to reduce MissedOpportunities for Vaccination (the present document): This guidebook providespractical information about translating the findings of the MOV assessments intoactionable work plans. It includes: a description of common reasons for MOV, anoverview of potential interventions to reduce MOV, examples of job aids and othermaterials for use at the health facility level, and guidance for activities and processes

2 Methodology for the Assessment of Missed Opportunities for Vaccination. Geneva: World Health Organization; 2017. (https://apps.who.int/iris/bitstream/handle/10665/259201/9789241512954-eng.pdf)

Introduction What is a missed opportunity for vaccination (MOV)?

Missed opportunities for vaccination (MOV) include any contact with health services by a child (or adult) who is eligible for vaccination (unvaccinated, partially vaccinated or not up-to-date, and free of contraindications to vaccination), but which does not result in the individual receiving all the vaccine doses for which he or she is eligible.

Reducing MOV is a strategy to increase immunization coverage simply by making better use of existing vaccination sites and services (at health centres, hospitals, outreach/mobile services etc.). Efforts to reduce MOV can also contribute to improving timeliness of vaccination, enhance health service delivery in general, and promote synergy between treatment services and preventive programmes at the health facility level.

Planning Guideto Reduce

Missed Opportunities for Vaccination

Methodologyfor the Assessment of

Missed Opportunities for Vaccination

InterventionGuidebook

for Reducing Missed Opportunities for Vaccination

to explore, design, monitor and evaluate locally tailored solutions to reduce MOV. This Intervention Guidebook can also be used as a stand-alone guide to plan actions toreduce MOV, even when a full MOV assessment has not been conducted.

A TechNet MOV topics page https://www.technet-21.org/en/topics/mov has been created in parallel to the Intervention Guidebook. The MOV topics page contains resources for reducing MOV and potential interventions to reduce MOV by addressing health worker knowledge, attitudes and practice; health systems issues; and vaccination demand. As additional interventions become available, they will be added to the TechNet MOV topics page.

All MOV documents and http://www.who.int/immunization/

supporting tools can be programmes_systems/policies_strategies/MOV/en/

accessed at:

FIGURE 1. Key questions addressed by the MOV strategy

1. How many opportunities forvaccination are missed at existingvaccination sites?

2. Why are opportunities for vaccinationbeing missed at the differentvaccination sites?

3. What can be adjusted or donedifferently (e.g. policies, behaviours,structural or organizational changes)so that we do not continue to missany opportunity to vaccinate?

The MOV strategy answers three important questions:

HOW MANYopportunities are missed?

WHYare these

opportunities being missed?

WHATcan be adjusted

or done differently?

Exit interviews with mothers/caregivers

In-depth interviews

Health worker KAP

questionnaires

Focus group discussions (mothers/caregivers and

health workers)

Brainstorming sessions

INTERVENTION GUIDEBOOK 32

Common reasons for MOV

Previous MOV assessments have found that reasons for MOV generally fall into three broad categories: 1) factors relating to health care workers knowledge, attitudes and practices; 2) factors due to healthsystem issues or constraints, including national policies; and 3) demand-related issues (caregiver/community behaviours). Figure 2 illustrates this. Additional examples of facility-level practices that mayresult in MOV can be found in Annex A.

Intended Use of this Intervention Guidebook

The most critical elements of the new 10-step MOV strategy are the steps for implementing proposed interventions to reduce MOV ( Steps 7-10 ). Recall that in Step 6 of the MOV strategy (Debrief withMOH leadership and immunization partners on proposed next steps; see Planning Guide), the Ministry of Health (MOH), in collaboration with the core immunization partners, should have come up with an endorsed action plan, at the national or subnational level, to reduce MOV.

Building on the experiences from countries that have implemented the MOV strategy, this Intervention Guidebook provides additional guidance to the national or subnational MOV Strategy Team in translating these action plans into activities that can be implemented and monitored at the health facility and higher levels of the health system.

Given the substantial leadership and involvement of MOH senior staff, and core immunization partners, the MOV strategy is most useful as an advocacy tool to highlight MOV as an important barrier to fully vaccinating children. In countries where MOV are suspected to be a problem, the process of the field work (see Methodology) is intended to identify where and why these bottlenecks are occurring and create a commitment from the MOH to address the identified issues. The bottom-up qualitative

approach of interviewing health workers and caregivers further adds to the value of the MOV process by identifying challenges and solutions that may have been ignored when other programme reviews were performed.

In other countries, previous programme reviews and desk analyses may already reveal MOV as an important issue and the MOH may already be committed to addressing the problems. In such situations, a full MOV assessment may not be needed. The MOH may choose to directly introduce interventions to reduce MOV where feasible. In such situations, this Intervention Guidebook can be used as a stand-alone guide to directly implement actions to reduce or prevent MOV in health facilities.

FIGURE 2. Summary of common reasons for MOV

Identifying “MOV champions” and forming a “MOV Strategy Team” early in the planning phase is one of the critical steps for success. The team should have a lead (the Assessment Coordinator if the country conducts as assessment) and ideally include a representative from each of the key immunization partners in the country. This team is responsible for advocating for the MOV strategy to the MOH and should maintain their role as the MOV implementation focal points during implementation of activities to reduce MOV.

RECALLMOV Strategy Team

CAREGIVERS

HEALTH CARE

WORKERS

» Limited hours

» Shortage and stock-outsof vaccines and/or HBRs

» Lack of integration

» Poorly-designed records

» Other adverse policies

» Vaccination cardavailability

» Restrictive policies onage-limits

» Failure to screen

» False contra-indications

» Concern re catch-upschedules and eligibility

» Low HBR retention/notbringing HBR to clinic

» Lack of awareness ofschedule

» Vaccine hesitancy

HEALTH SYSTEMS

1. In countries that have completed the first six steps of the 10-step MOV strategy, thisguidebook provides practical steps to implement the action plans endorsed in Step 6;

2. In countries that do not need to perform an assessment of MOV (due to sufficientevidence, desk reviews, or recent programme evaluations showing that MOV are acontributor to suboptimal coverage and equity), this guidebook could assist them toidentify commonly encountered causes of and interventions to reduce MOV in othersettings (further details available in the chapter MOV-lite options instead of conducting astandard MOV assessment).

This Intervention Guidebook is designed to be used in two possible ways:

INTERVENTION GUIDEBOOK 54

How do we ensure a smooth transition from action planning (Step 6) to implementation of interventions (Step 7)?In this chapter, we highlight some key actions and resources needed to transition from debrief to implementation, as well as some of the challenges and opportunities that the MOV Strategy Team needs to keep in mind.

The planning and field work steps of the MOV strategy are intended to result in country-led interventions to reduce MOV and improve vaccine coverage, timeliness and equity. As detailed in the Planning Guide1, Step 6 is critical to ensure that the MOH leadership, at the highest level, is aware of the findings of the MOV assessment, including the results of the brainstorming sessions and the resultant MOV action plan.

One of the expected outputs of Step 6, is a detailed action plan for reducing MOV (see Table 1), with clear implementation timelines and assigned responsible parties, as endorsed by all immunization partners. The MOV Strategy Team should take additional notes during the debrief session and continue to work together to finalize the action plan in the weeks following the debrief meeting.

The main expected output from Step 6 of the MOV strategy is an endorsed detailed action plan, including the following minimum details. It is critical that each proposed intervention is assigned to specific persons/organizations, tied to a timeline for implementation and, where needed, funding and sustainability plans identified.

TABLE 1 Components of the MOV Action Plan

MAIN ISSUES IDENTIFIED

PROPOSED IN-TERVENTIONS

IMMEDIATE NEXT STEPS

RESPONSIBLE PERSON/ORGANIZATION TIMELINE

REMARKS ON SUSTAINABILITY/ FUNDING PLANS

1.

2.

3.

As is often the case after brainstorming sessions, the list of potential interventions to include in the action plan may be long and aspirational. In order to improve the likelihood and feasibility of actually implementing the action plan, we recommend prioritizing a maximum of 3-5 of the proposed interventions. A useful tool for prioritizing among a long list of proposed interventions is the Impact vs. Feasibility Matrix shown in Figure 3.

To reduce delays in moving to the intervention phase and ensure the highest impact on reducing MOV, countries are encouraged to prioritize high impact interventions that have the highest probability for implementation. These may include interventions that are relatively easy and low cost to implement (“quick fixes”); interventions that have been planned for a long time but with no funding behind it; or interventions that one of the local partners may have a vested interest in (e.g. developing an electronic register or a stock monitoring system may be high-cost and expensive, but may be the a priority activity for one of the partners, and could therefore be prioritized and leveraged to reduce MOV).

1. Endorsement of a detailed action plan for reducing MOV, with clear implementationtimelines and responsible parties.

2. Identification/commitment of catalytic funding and/or plans for integration withexisting programmes

3. Plans for social mobilization and development of communication materials.

RECALL The expected outputs of Step 6 include:

INTERVENTION GUIDEBOOK 76

When determining which quadrant a proposed intervention would fall within, consider the following:

1. Feasibility

• What is the timeline for implementation?

• What commitment (time, technical capacity) is necessary?

2. Funding

• Think about existing funding streams and budget lines

• Is the intervention likely to be cost-effective?

3. Partner collaboration

• Explore synergies with existing work plans and current partner priorities and interests

• How does this fit within national immunization plans?

• How acceptable and appealing is the intervention to key stakeholders?

4. Potential impact

• What is the possible impact on reducing or preventing MOV in the short, medium, long term?

FIGURE 3. Impact vs. Feasibility Matrix for ranking potential interventions.

Key resources for implementing MOV interventions

In order for interventions to be successfully implemented in the country/district/health facilities, key resources need to be in place:

1. Leadership and management

2. Technical resources

3. Financial resources and sustainability plan

Leadership and managementFor purposes of sustainability and to ensure the MOV activities are approved at the highest level, the MOV Strategy Team should ensure that the detailed action plan of proposed interventions is integrated into the annual EPI work plan, national comprehensive multi-year plan for immunization (cMYP), or other similar country plans. These should not exist as standalone documents. The MOV action plan should also form the basis for more in-depth discussions, at the highest MOH level possible, on how to tackle the underlying challenges within the health system, EPI and, importantly, other implicated programmes to improve system efficiencies. Implementation of the MOV action plan and integration of MOV prevention strategies into ongoing programme planning should be discussed regularly at technical working group meetings. The leadership of the MOH and the EPI team is crucial for getting from assessment results to improved coverage and equity. This should not be viewed as a partner-driven activity.

Technical guidance and resourcesThere are many existing examples of technical resources, adaptable tools, and useful publications that can be drawn upon to address MOV issues and challenges. In this guidebook, under Step 7, we have summarized several resources that may be adapted by the MOV Strategy Team when designing sustainable interventions to reduce MOV.

Financial resources and sustainability planFindings from previously conducted MOV assessments have shown that many interventions will not necessarily require additional funds. In many countries, re-thinking health facility work flows, updating and disseminating existing vaccination policies, and relocating places where immunization services are conducted constitute some examples of prioritized low- or no-cost activities that may have a huge impact on access and uptake of vaccination services. Creativity on the part of the MOH in building synergies and tailoring existing work streams have been most effective in implementing successful and sustainable interventions.

High impactRequires planning and/or persuasion

Long Term Investment

Low impactDifficult or expensive

Resistance

High impactQuick fixesEasy wins

(“low hanging fruit”)

Low impactQuick Fixes

Nice to haves

B A

FEASIBILITY

IMPA

CT

HARD

LOW

HIG

H

EASY

D C

INTERVENTION GUIDEBOOK 98

On the other hand, some interventions may require modest or significant financial investments in order to be implemented. Examples include increasing the frequency and quality of supportive supervision, printing and disseminating job aids for health workers, improving stock management practices to reduce stock outs of vaccines and supplies, refresher trainings for both immunization and non-immunization staff on key issues identified in the assessments (false contraindications, catch up vaccination, etc.).

In such circumstances, the MOV Strategy Team will benefit from exploring in-country resources that could be applied directly or re-programmed for the MOV-specific tasks. Examples of such resources may include, unspent Gavi health system and immunization strengthening (HSS) funds; other partner funds earmarked for health system improvements or for enhancing primary health care; and resources from local and international non-governmental organizations (NGOs) (including in-kind resources).

When new or additional funding is required, several countries have had great success in including results from their MOV assessment and MOV action plans as activities in Gavi HSS applications. Others have successfully used the annual Targeted Country Assistance (TCA) process during Gavi joint appraisals (JAs) to earmark partner funds for improving coverage and equity using the MOV strategy. In general, these MOV activities have been very well received by the Gavi Independent Review Committee.

Other partners may have ongoing activities in defined areas of the country that may be synergistic to the MOV action plan. For example, there may be an ongoing project in which an implementation partner is assisting the MOH with redesign, printing and distribution of home-based records (HBR). The MOV Strategy Team could capitalize on such opportunities to ensure that the layout of the HBR is user-friendly and will facilitate identification of previously missed vaccines, include guidance on catch-up vaccination, does not include messages that may deter vaccination (e.g. inappropriate age restrictions) and also designed to include a reminder system/defaulter tracking system, etc. Identifying and capitalizing on such opportunities can help minimize the resources required to implement actions to reduce MOV.

Challenges and bottlenecks to implementation

The MOV Strategy Team should be aware of possible challenges and bottlenecks that may arise when implementing the action plan, and plan accordingly. Failure to do so has the potential to seriously undermine efforts, therefore additional planning should be undertaken to ensure the following:

• Coordination among partners during the planning and field work steps. Partners are more likely to take ownership and support the implementation if they were part of the planning and execution of the field work, including prioritization of the interventions. A mapping of all key implementation partners involved in immunization activities should be included as part of the assessment planning. Involving these partners early on is critical to identifying opportunities for cooperation.

• Buy-in of key opinion leaders and decision-makers. In seeking someone to champion efforts to reduce MOV, the MOV Strategy Team should aim as high as possible within the MOH. In many countries, participation by the Minister of Health in the debrief session resulted in additional confidence in implementing the proposed interventions. Managing this high-level involvement can be time-consuming and challenging, but it is well worth the effort. One of the best ways to facilitate this is to remind the EPI team and other partners to share MOV plans with the Minister, or their alternate, as early in the planning steps as possible (for instance, during the ethical review process or when the concept note is first shared with the technical working group or Inter-Agency Coordinating Committee (ICC)).

• Clarity on roles and responsibilities for each activity in the action plan. The MOV Strategy Team should ensure that each activity is assigned to a responsible party, depending on relative partner strengths and comparative advantages. Each activity should also have a time frame for implementation, agreed to by all the partners responsible for execution, in order to increase accountability.

• Resource allocation/availability. Along with responsible actors and timelines, each activity in the action plan should be linked to an appropriate source of resources – both human and financial – in order to ensure implementation.

• Prioritization of time and resources. Including MOV interventions in the annual EPI work plan, cMYP) or other similar country plans can help ensure that the activities are prioritized. In addition, applying a holistic ‘MOV reduction’ mind-set to the implementation of other EPI activities can also indirectly help address MOV – for example, in the redesign or update of HBRs and other recording and reporting tools for a new vaccine introduction, vaccine stock management improvement activities, etc.

INTERVENTION GUIDEBOOK 1110

MOV-lite options instead of conducting a standard MOV assessmentThe standard MOV assessment and the generation of country-specific data on MOV can be very useful for advocacy and raising awareness of the problem, but it is not always necessary. Many programmes and health workers are already well aware of the existence and causes of MOV in their setting. In such circumstances, programmes may choose to move directly to implementation of locally-tailored interventions to reduce MOV or to further explore issues.

Programmes can conduct a desk review making use of existing data from e.g. expanded programme on immunization (EPI) reviews, demographic and health surveys (DHS), multiple indicator surveys (MICS), health management information systems (HMIS) data, WHO and UNICEF Estimates of National Immunization Coverage (WUENIC) or recent immunization coverage surveys. Countries may also choose to incorporate MOV specific questions when conducting an EPI review3 or a small-scale health facility MOV assessment, as described below.

Conducting a Brainstorming Workshop to Reduce MOV

Following the desk review, EPI review or small-scale health facility MOV assessment, convening focus group discussions or brainstorming workshops with EPI staff from all levels, and core immunization partners are useful steps for designing interventions for rapid implementation of activities.

WHO and partner resources such as the new EPI Review guide3, Reaching Every District (RED)4, 2YL guidance5 and the guide for sustaining maternal and neonatal tetanus6 highlight opportunities to include MOV reduction strategies into different aspects of immunization work.

3 A guide for conducting an Expanded Programme on Immunization (EPI) Review. Geneva: World Health Organization; 2018 (http://apps.who.int/iris/bitstream/10665/259960/1/WHO-IVB-17.17-eng.pdf)

4 Reaching Every District (RED) – a guide to increasing coverage and equity in all communities in the African Region. Brazzaville: World Health Organization; 2017 (http://www.afro.who.int/sites/default/files/2018-02/Feb%202018_Reaching%20Every%20District%20%28RED%29%20English%20F%20web%20v3.pdf)

5 Establishing and strengthening immunization in the second year of life: practices for immunization beyond infancy. Geneva: World Health Organization; 2018 (http://apps.who.int/iris/bitstream/handle/10665/260556/9789241513678-eng.pdf)

6 Protecting All Against Tetanus: Guide to sustaining maternal and neonatal tetanus elimination (MNTE) and broadening tetanus protection for all populations. Geneva: World Health Organization; 2019. (https://www.who.int/immunization/diseases/tetanus/Protecting_All_Against_Tetanus_final_draftV4_23Jan_web.pdf)

FIGURE 4. Are MOV contributing to suboptimal performance in your country? Where/how to get these data?

7 8

7 National Institute of Statistics, Directorate General for Health, and ICF International, 2015. Cambodia Demographic and Health Survey 2014. Phnom Penh, Cambodia, and Rockville, Maryland, USA: National Institute of Statistics, Directorate General for Health, and ICF International (https://dhsprogram.com/pubs/pdf/FR312/FR312.pdf)

8 Wallace AS et al. Assessment of vaccine wastage rates, missed opportunities, and related knowledge, attitudes and practices during introduction of a second dose of measles-containing vaccine into Cambodia’s national immunization program. Vaccine. 2018 Jul 16;36(30):4517-4524. doi: 10.1016/j.vaccine.2018.06.009.

MOV-LITE OPTIONS

GO DIRECTLY TO BRAINSTORMING (STEP 6) WITH THIS INFORMATION (a MOV Strategy Team is still needed to drive this work forward)

Integrate a MOV component into another planned

assessment

Convene a MOV workshop using available data

(mainly quantitative) or supplementing with qualitative

aspect (e.g. focus group discussions)

Carry out a small-scale

health facility MOV assessment

STAND-ALONE ASSESSMENT

OR ORSTEPS

3-6

Despite strong progress with immunization, the 2014 Cambodian DHS found that only 73% of children aged 12-23 months were fully immunized.6 In October 2017, a comprehensive national EPI review was conducted in 10 of its 24 provinces. The EPI review comprised of a broad range of existing questions related to MOV, such as availability of immunization services; recording and reporting of vaccination data; screening of child’s vaccination status; vaccine availability, including stock outs, health worker knowledge of contraindications to vaccination and caregivers knowledge, attitude and practices.

A three-day workshop on MOV was held in November 2017. The workshop included EPI staff from all levels, as well as core immunization partners. The workshop comprised of plenary and group work sessions to develop strategies and create an action plan to address MOV. A general introduction to the concept of MOV, followed by a review of current evidence of MOV in Cambodia, which included: 1) findings from the EPI desk review; 2) recent EPI review data related to MOV; 3) findings from the 2014 measles second dose post-introduction evaluation; and 4) the 2013 vaccine wastage study.7

COUNTRY EXAMPLE Using the MOV-lite model in Cambodia

INTERVENTION GUIDEBOOK 1312

Participants were encouraged to actively engage in the brainstorming and contribute towards discussions on additional reasons for MOV in Cambodia. In addition, working groups were tasked to identify barriers and challenges to reducing MOV in Cambodia. The participants were split up into groups to discuss potential strategies to reduce MOV at the health centre level, among health workers and caregivers and at the national-level. Cause and effect diagram analyses (Figure 5) were used to identify the major cause categories, possible causes and problem statements. Each team then presented their findings to the plenary. An action plan to reduce MOV was formulated for implementation at the national level and subnational levels.

FIGURE 5. Group work using cause and effect diagram analyses to identify causes of MOV during a brainstorming workshop to reduce MOV in Cambodia

This was the first WHO supported MOV-lite implementation. The process was seen to be helpful for MOH staff and immunization partners to identify causes for MOV and brainstorm about activities to reduce MOV. Pre-requisites for success in future countries will include existing data sources, high-level support for reducing MOV and strong commitment to follow-up on agreed activities. Countries that have recent data may be able to apply a similar method.

COUNTRY EXAMPLE: CONTINUED Using the MOV-lite model in Cambodia

Small-scale health facility MOV assessment

If a country does not have the capacity and/or funding to conduct a full-scale MOV assessment, as outlined in the Planning Guide1 and Methodology Guide2, but would like to explore if MOV occurs, a small-scale health facility MOV assessment can be conducted in a given county/state or district, given that permission by the relevant authorities has been obtained in advance. Once approved, the teams should follow Step 3 of the MOV strategy on how to Conduct field work for the rapid assessment of MOV, and adapt the methodology, as deemed necessary.

A sample of health facilities can be visited on immunization days to capture information related to MOV from caregivers, health workers and senior health facility staff (i.e. those in-charge) through health facility exit interviews, health worker KAP questionnaires, in-depth interviews in senior health facility staff and focus group discussions with caregivers and health workers. Similar to guidance in the methodology guide, the health facilities should be a mix of public or private in urban and rural settings. Questionnaires in the annex of the Methodology Guide2 and on the MOV webpage9 should be adapted to the country context.

Field teams conducting the interviews should be aware of the core principles when collecting data2 and ensure that all participants in the small-scale health facility MOV assessment have given informed consent.

Once the interviews have been completed, the team should convene to consolidate their findings and discuss any themes that emerged during interviewing of participants. If practices that may result in MOV emerged, the team should first brainstorm together on which interventions could reduce MOV. The team should then bring their findings and interventions to reduce MOV to the attention of the relevant authorities.

Following this discussion, a decision may be taken at a higher level to implement interventions to reduce MOV or to further explore MOV in other parts of the county/state or district or country. Therefore, reviewing the Planning Guide1 and Methodology Guide2 will be required. However, if a decision is made to implement the interventions, Steps 7-10 of the MOV strategy will be helpful in implementing the final steps of the 10-step MOV strategy.

9 WHO website. Missed Opportunities for Vaccination (MOV) Strategy (https://www.who.int/immunization/programmes_systems/policies_strategies/MOV/en/)

INTERVENTION GUIDEBOOK 1514

1

2

3

5

7

9

4

6

8

10

1

2

3

5

9

4

8

10

6

7

1

2

3

5

7

9

4

6

8

10

STEPSTEP

WHO District staff, Health facility staff, MOH and immunization partners, MOV Strategy Team

WHEN Should commence within 6-12 months following the MOV assessment

TASK 7.1

Based on the findings of the MOV assessment, implement interventions to address specific findings

TASK 7.2

Provide additional policy guidance, directives, job aids and other communication materials from the national level

Implement the interventions7

7

Based on the findings of the MOV assessment, implement interventions to address specific findings

It is important that the proposed interventions to reduce MOV actually target the problems that were identified during the desk review, assessment and/or brainstorming workshop. These problems may differ by district or by type of health facility (e.g. urban/rural or public/private). The overall intent is to promote supportive policies, capable service providers and managers, strong logistics, and to stimulate broad acceptance of immunization by health workers and in communities.

Interventions to reduce MOV may be implemented at the national level, or in specific districts or health facilities/communities.

Table 2 below outlines common causes of MOV and potential interventions to consider. Additional examples and adaptable tools are available in Annex B and online at www.technet-21.org/en/topics/mov.

Provide additional policy guidance, directives, job aids and other communication materials from the national or subnational level

The MOV Strategy Team should work with the MOH and the ICC, or equivalent, to develop or revise policies and other types of guidance to address specific issues, e.g. revision of age restrictions for vaccination if applicable, implementation of the multi-dose open vial policy, clarification of true and false contraindications. The process of updating and disseminating immunization policies is usually initiated by the National Immunization Technical Advisory Group (NITAG) or a sub-group of the ICC.

Measures should be taken to ensure that new or updated policies are systematically introduced and adequately communicated to health workers and their immediate supervisors, reinforced, and optimally implemented. The MOH in Kenya created an active WhatsApp group with all immunization managers at the county and sub-county levels. Such platforms may be adaptable to facilitate rapid, low-cost and efficient dissemination of information from authorized persons. They can also provide remote support and answer questions from health workers or those putting policies into practice. Depending on the types of policies, caregivers and community members should be empowered to demand the right types of services. In Kenya, for instance, since 2014, there has been a national policy on conducting daily vaccination services, as well as on opening lyophilized multi-dose vials for even one child, but these are not uniformly implemented across all health facilities. If caregivers became aware of these policies and would demand their rights, the caregivers themselves could become catalysts for change.

TASK

7.1

TASK

7.2

INTERVENTION GUIDEBOOK 1716

1

2

3

5

9

4

8

10

1

2

3

5

9

4

8

10

6 6

7 7

Additional examples available online at: www.technet-21.org/en/topics/mov and at www.who.int/immunization/programmes_systems/policies_strategies/MOV/en/

POTENTIAL CAUSES INTERVENTIONS TO CONSIDER COUNTRY EXAMPLES AND/OR RESOURCES

RELATIVE COST

HEALTH WORKER KNOWLEDGE, ATTITUDES AND PRACTICE

FACTOR LEADING TO MOV:

Failure to screen vaccination status during routine visitsHealth workers are busy and forget, are not trained, or are not in the habit of checking vaccine history at every visit

» Provider reminders: stickers or designson cover of HBR, fridge magnets,posters in health facilities, table topdisplay/sign, computer screensaver,caregiver-owned prompt cards.

» Use new vaccine introduction asopportunity to train health careworkers to check HBRs.

• The community Guide Vaccination programs:Provider remindersbit.ly/2XWJ2FH

• Agency for Healthcare Research and QualityReminder systems for immunizations andpreventive services bit.ly/2ZwShwr

• Balas et al., Improving preventive care byprompting physicians . Arch Intern Med.2000;160(3):301-308.

• 4 Pillars™ The 4 Pillars PracticeTransformation Program for Immunizationbit.ly/2XqZAIJ

• Lin et al., Using the 4 pillars™ practicetransformation program to increase adultinfluenza vaccination and reduce missedopportunities in a RCT. BMC InfectiousDiseases 201616:623.

$

The HBR is not available

» During supervision, remind healthworkers to use all available meansto find out the vaccination status(checking health facility registers,contacting regular health centre if thisis not their usual clinic, or they haverelocated, etc.). Lack of documentationis not a valid reason for not vaccinatingeligible children.

» When in doubt, vaccinate and issuea new or temporary card; remind thecaregiver to keep the HBR safe butavoid criticism or humiliation as thatmay deter the caregiver from returningfor future doses.

• Practical Guide for the Design, Use andPromotion of Home-Based Records inImmunization. Geneva: World HealthOrganization; 2015. bit.ly/2L26JsC

• WHO recommendations on home-basedrecords for maternal, newborn and childhealth. Geneva: World Health Organization;2018. bit.ly/2XXdKyg

• JSI Coordination and Implementationof Child Health Record Redesigns (Home-Based Records) Resources.bit.ly/2RoobIA

$

The HBR is in a different format, or language, to what the health worker is used to

HBR is poorly designed/easily damaged

» Immediate term: conduct inquiry withhealth workers to identify specific areasof confusion in existing card so as toprovide clarification through supportivesupervision or training.

» Medium term: Revise and improveHBR.

$-$$

POTENTIAL CAUSES INTERVENTIONS TO CONSIDER COUNTRY EXAMPLES AND/OR RESOURCES

RELATIVE COST

FACTOR LEADING TO MOV:

Failure to screen vaccination status during visits for curative care or other servicesLack of integration between curative and preventive services. Non-immunization staff are not trained or in the habit of checking vaccination history

» Reinforce implementation of IntegratedManagement of Childhood Illnesses(IMCI)-based screening algorithmto use in sick child visits (includesvaccination check for all children)

• IMCI chart booklet. Geneva: World HealthOrganization; 2018. bit.ly/2WVXDQs

• Integrated MCH flow chart. UNICEF, Kenya.www.technet-21.org/en/topics/mov

• Working together, an integration resourceguide for immunization services throughoutthe life course. Geneva: World HealthOrganization; 2018. bit.ly/31HwDHV

$

» Provide whole-site supportivesupervision on immunization thatengages staff from services other thanimmunization so that they are orientedon, and commit to, steps that they cantake to reduce missed opportunities.

$$

Lack of practice of screening vaccination status of children that accompany mother for other services (e.g. ANC check-up, medical care for a sibling, etc.)

» Introduce the practice of having healthworkers from other health servicesencourage caregivers to bring theHBRs for their children to every visit tothe health facility or outreach site.

$

TABLE 2 Common causes of MOV and potential interventions to consider

INTERVENTION GUIDEBOOK 1918

1

2

3

5

9

4

8

10

1

2

3

5

9

4

8

10

6 6

7 7

POTENTIAL CAUSES INTERVENTIONS TO CONSIDER COUNTRY EXAMPLES AND/OR RESOURCES

RELATIVE COST

FACTOR LEADING TO MOV:

Confusion regarding eligibility (including false contraindications) and catch-up schedulesMisconception that children cannot be vaccinated when they have a mild fever, cough, diarrhoea, or other mild illness

» Develop and disseminate a policy orguideline that specifically highlightsthis issue and is signed by a high-levelMOH authority.

» Job aids explaining true and falsecontraindications for vaccination.

• Vaccine safety and false contraindications tovaccination training manual. Copenhagen:WHO Regional Office for Europe; 2017.bit.ly/2KV79km

• WHO Addressing Vaccine Hesitancy.bit.ly/2Fj2c1a

• *Immunization action coalition. ScreeningChecklist for Contraindications to Vaccinesfor Children and Teens. bit.ly/2WO58sB

• Annex 4: Illustrative example of job aid onscreening for vaccine eligibility(2YL Guidance)

$

Perception that children over 12 months are no longer eligible for missed vaccinations

» Job aid and/or policy circular from thenational level, defining ages of vaccineeligibility

• Annex 4: Illustrative example of job aid onscreening for vaccine eligibility(2YL Guidance)

$

Health worker confusion over eligibility for missed vaccination/catch-up schedules

» Decision support tools/job aids(graphs, checklists, algorithm, mobileapp) for screening and catch-upvaccination, including an acceleratedschedule for children who present lateand are missing vaccinations

» Electronic health registers that provideautomatic alerts for vaccines that are overdue

• Annex 4: Illustrative example of job aid onscreening for vaccine eligibility(2YL Guidance)

• Smartphone or desk-top application (e.g.STIKO App)

• Diskette from Thailand indicating eligibilityand windows for catch-up.www.technet-21.org/en/topics/mov

• Poster from Thailand of vaccination calendar,including accelerated schedule. www.technet-21.org/en/topics/mov

• Child Care Vaccination Calendar fromZimbabwe. www.technet-21.org/en/topics/mov

• Module 5: Managing an immunizationsession. Chapter 3.1, assessing eligibilityfor immunization. Immunization in Practice.Geneva: World Health Organization; 2015.bit.ly/2XZmlAC

$-$$

Reluctance to co-administer simultaneous (injectable) vaccines to children who are out-of-date for multiple vaccines

» Job aid for health workers onacceptability of multiple injections

» Develop a policy or guideline thatspecifically highlights this point and issigned by a high-level MOH authority.

• WHO resources on Safety and acceptability ofmultiple vaccine injections. bit.ly/2Rn4OQt

• The Ugandan Immunization In Practicemanual provides a clear summary ofwhere to give each injection.www.technet-21.org/en/topics/mov

$

POTENTIAL CAUSES INTERVENTIONS TO CONSIDER COUNTRY EXAMPLES AND/OR RESOURCES

RELATIVE COST

FACTOR LEADING TO MOV:

Concerns about vaccine wastageHealth workers are hesitant to open a multi-dose vial for only one or a few children

» Reinforce (through supportivesupervision) application of the WHOpolicy and national policies on useof open multi-dose vials for selectedvaccines.

• Policy Statement: Multi-dose Vial Policy(MDVP). Geneva: World Health Organization;2014. bit.ly/2KtJ6cT

$

» Policy circular from national level (andbacked up by supportive supervision)to open measles and BCG vials, evenfor one child. District health teamsand clinic managers are key targetgroups for such a circular. Add this tosupportive supervision checklists.

• Some countries use a “utilization rate” (i.e.the inverse of vaccine “wastage rate”) tomonitor vaccine consumption. This avoids thenegative connotation attached to the term“wastage” and encourages use of vaccine forevery child.

$

» The goal of the vaccine wastage ratescalculator is to estimate more accuratewastage rates automatically accordingto country contexts and for all WHOprequalified vaccines. The overallobjective of this new tool is to provideguidance for more accurate vaccineforecasting and setting benchmarksfor monitoring vaccine utilization andwastage. Extended benefits of thetool include enhanced service deliveryplanning and optimal vaccine vialselection.

• WHO Vaccine Wastage Rates Calculator.bit.ly/2WSVVE7

• Wallace et al., Assessment of vaccine wastagerates, missed opportunities, and relatedKAPs during introduction of a second dose ofmeasles-containing vaccine into Cambodia’snational immunization program. Vaccine.Volume 36, Issue 30, 16 July 2018, Pages4517-4524.

Health workers schedule vaccinations only on certain days of the week/month

» Increase the number of days on whichvaccination with all vaccines offered(ideally every day)

$$

» If daily is not possible, coordinate withcommunity leaders and committees toagree on the most suitable days/timesfor vaccination. Ensure the schedulesare well publicized and posted.

$

INTERVENTION GUIDEBOOK 2120

1

2

3

5

9

4

8

10

1

2

3

5

9

4

8

10

6 6

7 7

POTENTIAL CAUSES INTERVENTIONS TO CONSIDER COUNTRY EXAMPLES AND/OR RESOURCES

RELATIVE COST

HEALTH SYSTEMS ISSUES

FACTOR LEADING TO MOV:

National policy barriersRestrictive policies indicating upper age limit for vaccination (e.g. one year)

~ or ~

Lack of clear national policy on vaccination of children over the scheduled age

» Revise (or reinforce) national policy to provide clear guidance to health workers on ages that children can be vaccinated. Actively and systematically introduce this guidance in training, supportive supervision, feedback.

» Disseminate updates to policies via WhatsApp or other means to ensure it reaches all health workers.

• WHO recommendations for routine immunization - summary tables. bit.ly/2L0lXyc

• Cameroon changed their rotavirus policy by lifting age restrictions. Rotavirus vaccine could then be given alongside Pentavalent, and allow for catch-up of missed doses.

• Annex 4: Illustrative example of job aid on screening for vaccine eligibility (2YL Guidance)

$

FACTOR LEADING TO MOV:

Lack of integration between preventative and curative servicesNon-immunization staff are not trained or able to screen and provide vaccinations

» Update policies or standing orders to allow additional cadres of health staff to provide vaccinations

• The Community Guide. Vaccination Programs: Standing Orders. bit.ly/2XXVwNe

• Working together, an integration resource guide for immunization services throughout the life course. Geneva: World Health Organization; 2018. bit.ly/31HwDHV

$$

» Integrate basic immunization principles and schedule into in-service and pre-service training for non-immunization staff

» Posters and/or job aids for non-immunization staff with the EPI schedule and reminding them to check HBRs for missing vaccines

» Coordination at the facility level of approaches to discussing vaccination with caregivers

$$

» Cross-referral system – coupons are printed for curative service staff to give to parents of children missing vaccinations

• Ogbuanu et al., describe the use of a coupon referral system in Chad to screen the vaccination status of children who come to health facilities for curative and preventive services PLoS One. 2019 Jan 24;14(1):e0210648. DRC also described a coupon referral system they are piloting at the Global Immunization Meeting in 2018. bit.ly/2WSVPYh

$

POTENTIAL CAUSES INTERVENTIONS TO CONSIDER COUNTRY EXAMPLES AND/OR RESOURCES

RELATIVE COST

Vaccination area is separate from the clinical/treatment area, with minimal interactions between staff from each area

» Improve the visibility, accessibility, and/or location of the vaccination clinic (e.g. move the vaccination area closer to the outpatient waiting room or registration area)

• Li et al., describe how in Timor Leste, vaccination areas were moved to the front of the health facility to have more visibility. Vaccine. Volume 37, Issue 30, July 2019, Pages xx-xx.

• Elia et al., Providing opportunistic immunisations for at-risk inpatients in a tertiary paediatric hospital. J Spec Pediatr Nurs. 2017 Jan;22(1).

$

» If logistically impossible to relocate, improve the visibility, using brightly coloured posters or paint, bold arrows or lines

$

» Screening and/or vaccination in outpatient waiting area (also use this time to educate caregivers about immunization and disease prevention)

$

» Vaccination defaulters list shared with curative service staff

$

Vaccination clinic hours are not the same as curative care services

FACTOR LEADING TO MOV:

Delays/long queuesStaffing shortages or poor workflow organization may mean caregivers have to wait a long time, some may leave the health centre before they receive the needed vaccines.

» Document waiting times at some health facilities to advocate for increased staff numbers.

» Use the time caregivers are waiting to offer health promotion about vaccination and other health interventions.

» Take advantage of the long wait times in out-patient (curative) departments by offering vaccination services to patients waiting in the curative areas.

» Identify bottlenecks and reduce to the extent possible and within the context of health facility resources; e.g., by arranging for community health workers to assume certain functions such as triage, recordkeeping, health talks, etc.

• Immunization flipchart from Ghana designed for health education sessions to be given in waiting areas of health facilities. www.technet-21.org/en/topics/mov

$$

INTERVENTION GUIDEBOOK 2322

1

2

3

5

9

4

8

10

1

2

3

5

9

4

8

10

6 6

7 7

POTENTIAL CAUSES INTERVENTIONS TO CONSIDER COUNTRY EXAMPLES AND/OR RESOURCES

RELATIVE COST

FACTOR LEADING TO MOV:

Stock outs of vaccine and/or vaccine supplies, (including HBRs)Inadequate vaccine forecasting or stock management practices can occur at any level of the health system

» Identify root causes of weak practices and take actions to address them. This could include job aids or refresher training on vaccine forecasting.

• WHO Vaccine Management Handbook. bit.ly/2WRHsmT

• WHO EVM assessment tools and user guides. bit.ly/2Ksgq3N

• Mobile phones and digital technology to boost vaccine delivery in Uganda. bit.ly/2KpCRH7

$$

» Improve stock monitoring systems and reporting at the lower levels of the supply chain.

» Data quality improvement plans and reviews of denominator to ensure more accurate target population for vaccine forecasting .

• A denominator guide will be available soon from UNICEF.

• WHO Data quality review (DQR) toolkit. bit.ly/2Kv0HAX

$$

Lack of funding for delivery of vaccines and/or supplies

» Track stock outs and their practical consequences in order to advocate for additional funding.

• WHO Vaccine Wastage Rates Calculator. bit.ly/2WSVVE7

$$

Lack of funding for printing of home-based records

$$

POTENTIAL CAUSES INTERVENTIONS TO CONSIDER COUNTRY EXAMPLES AND/OR RESOURCES

RELATIVE COST

FACTOR LEADING TO MOV:

Limited scheduling/availability of vaccination services (in general or for specific vaccinations)Lack of policy regarding delivery of daily immunization services

» Revise (or reinforce) national policy that vaccination should be offered daily (in all facilities with a functional cold chain).

$

Hours that immunization services offered not compatible with caregivers, particularly in urban settings where caregivers are employed in full-time economic activities.

» Extend services hours to evenings and weekends based on caregiver feedback to reduce probability of MOV occurring.

• Urban immunization toolkit developed by the Urban Immunization Working Group bit.ly/2WThUWO

Concern about wastage/lack of clear national policy on opening measles and BCG vaccine vials

» Revise (or reinforce) national policy endorsing provision of all vaccinations (including measles and BCG vaccines) anytime an eligible person has a healthcare visit.

» Develop job aid and behaviour change strategy directed toward health workers and their supervisors to address reluctance to open vaccine vials and promote improved practices.

» Session size

• Some countries use a “utilization rate” (i.e. the inverse of vaccine “wastage rate”) to monitor vaccine consumption. This avoids the negative connotation attached to the term “wastage” and encourages use of vaccine for every child.

• Wallace et al., Assessment of vaccine wastage rates, missed opportunities, and related KAPs during introduction of a second dose of measles-containing vaccine into Cambodia’s national immunization program. Vaccine. Volume 36, Issue 30, 16 July 2018, Pages 4517-4524.

$

FACTOR LEADING TO MOV:

Caregiver expected to pay for some/all vaccines or vaccine supplies (including HBRs)Lack of enforcement of national policy to ensure that all vaccines provided as part of the national immunization programme are free

» Development/enforcement of national policy to ensure that all vaccines are provided for free

INTERVENTION GUIDEBOOK 2524

1

2

3

5

9

4

8

10

1

2

3

5

9

4

8

10

6 6

7 7

POTENTIAL CAUSES INTERVENTIONS TO CONSIDER COUNTRY EXAMPLES AND/OR RESOURCES

RELATIVE COST

DEMAND-SIDE (CAREGIVER BEHAVIOURS)

FACTOR LEADING TO MOV:

Failure of caregiver to bring HBR, or low HBR retention Caregiver is not aware that HBR should be brought to every visit

~ 0r ~

Caregiver is not aware of the importance/

value of the HBR

~ 0r ~

Lack of vaccine card availability

» Posters in health centres remindingcaregivers (and reminding health careworkers to remind caregivers) to bringcard to every visit and to keep it safe.

» Job aid/counselling card for healthworkers to promote appropriatepractices (not denying vaccinationbecause of lack of card; issuing newcard; communicating with caregiver).

HBR is poorly designed/easily damaged

» Immediate term: conduct inquiry withhealth workers to identify specific areasof confusion in existing card so as toprovide clarification through supportivesupervision or training.

» Medium term: Revise and improveHBR.

• Practical Guide for the Design, Use andPromotion of Home-Based Records inImmunization. Geneva: World HealthOrganization; 2015. bit.ly/2L26JsC

• WHO recommendations on home-basedrecords for maternal, newborn and child health. Geneva: World Health Organization; 2018. bit.ly/2XXdKyg

• JSI Coordination and Implementation of Child Health Record Redesigns (Home-Based Records) Resources. bit.ly/2RoobIA

POTENTIAL CAUSES INTERVENTIONS TO CONSIDER COUNTRY EXAMPLES AND/OR RESOURCES

RELATIVE COST

FACTOR LEADING TO MOV:

Caregivers do not ask to have child screened for vaccination status during non-vaccination visitsLack of awareness on the part of caregivers that they can request to have their child vaccinated during non-vaccination visits

» Prompt-to-screen cards for caregiversto show to providers at any health visit.Orient personnel at registration deskfor facility on the need to screen eachcaregiver and child for vaccinationeligibility the day of the visit.

Caregivers do not know that more than one dose is needed for some vaccines

» Communication by health workers tocaregivers about each vaccine, therecommended age/s at which thevaccine should be given, how manydoses are needed for protection, andthe interval between doses. This shouldbe supplemented with community-based communication and health talksin waiting rooms.

• Immunization sticker from Lao PDR.www.technet-21.org/en/topics/mov

FACTOR LEADING TO MOV:

Vaccine hesitancyConfusion regarding false contra-indications.

Caregivers may not want their child to be vaccinated while they are receiving treatment services

» Table-top aid/poster listing true andfalse contraindications, that healthworker can use to address caregiverconcerns.

» Wall poster with MOH policy signed byhigh-level authority on true and false contraindications.

» Provider education to respond with keymessages about vaccination eligibilityand safety

» Materials tailored for caregivers oncontraindications

• WHO training modules for health workers oncontraindications. bit.ly/2Ku6TJu

Concerns over the child receiving multiple injections at the same visit

» Educate caregivers on how to respondwith key messages when caregiversexpress hesitancy about vaccination

» Educate caregivers about benefits ofvaccination to ensure they providestrong recommendation that allvaccines should be administered, asscheduled.

» Introduce protocols for managing painat the time of injections

• WHO resources on safety and acceptabilityof multiple vaccine injections and trainingresources on reducing pain during injections.bit.ly/2Rn4OQt

• Post vaccination care informationto parents in Australia.www.technet-21.org/en/topics/mov

Concerns over potential AEFI

» Educate caregivers on how to respondwith key messages when caregiversexpress hesitancy about vaccination

• Vaccine safety events: managing thecommunications response. Copenhagen:WHO Regional Office for Europe; 2013.bit.ly/2WWhONP

INTERVENTION GUIDEBOOK 2726

1

2

3

5

7

9

4

6

8

10

1

2

3

5

7

9

4

10

6

8

1

2

3

5

7

9

4

6

8

10

STEP

Establish a clear monitoring and supervision plan

It is important to develop a monitoring and evaluation (M&E) plan at the same time as the intervention is being designed, and implementation is being planned. However, it must be noted that developing a M&E plan is distinct from the rapid field evaluation of outcomes/impact of interventions, described in Step 9. Evaluation is an episodic assessment of the feasibility, implementation or change resulting from the programme, project or intervention, while monitoring is a systematic and ongoing process of collecting, analysing and using information about progress over time, to help guide implementation and inform ways to improve.

Monitoring measures progress toward results by collecting information on inputs, activities, outputs, and sometimes short-term outcomes. Analysis of the collected information allows for monitoring progress over time to help guide implementation and inform ways to improve.10,11,12

Monitoring should be seen as part of the MOV intervention, not performed as a separate activity and should focus on:

a) What was done (e.g. number of trainings held)?

b) How well it was done (e.g. strengths and challenges)?

c) What are the immediate results (outputs) from the effort (e.g. number of people trained), with emphasis on the latter (results-focussed monitoring)?

Key to monitoring, and evaluation, is the notion that purpose informs deign (e.g. define what you want to know, then determine how to get the desired information).

Answering the following questions in Box 1 will help guide decisions about what and how to monitor and on how to develop a monitoring plan13,14:

10 BetterEvaluation. C4D Hub: Definition of key terms.(https://www.betterevaluation.org/en/node/6005)11 Knowledge for health. Chapter 8 Managing Information: Monitoring and Evaluation (https://www.k4health.org/sites/default/files/

MSH%20eHandbook%20ch08.pdf)12 Compass. How to develop a monitoring and evaluation plan. (https://www.thecompassforsbc.org/how-to-guides/how-develop-

monitoring-and-evaluation-plan)13 Adapted from Summary of the U.S Centers for Disease Control and Prevention. Framework for program evaluation in public

health. (https://www.cdc.gov/eval/materials/frameworksummary.pdf)14 Community Toolbox. Section 5. Developing an Evaluation Plan. (https://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/

evaluation-plan/main)

STEP

WHOMOH and the MOV Strategy team, with support from core local immunization partners

WHEN To commence within 6-12 months immediately following the MOV assessment, and in conjunction with step 7

TASK 8.1 Establish a clear monitoring and supervision plan

TASK 8.2

Provide funds for supportive supervision, data collection and corrective actions

TASK 8.3

Provide monitoring charts and ensure adoption of corrective actions with visible display of monthly coverage estimates

Provide supportive supervision and monitor progress8

8

TASK

8.1

INTERVENTION GUIDEBOOK 2928

1

2

3

5

7

9

4

10

1

2

3

5

7

9

4

10

6 6

8 8

15

15 CDC Program Evaluation Self Study Guide. Introduction - Step 4, Gather credible evidence. (https://www.cdc.gov/eval/guide/step4/index.htm)

How is “success” defined by each stakeholder affected by the intervention and what key questions do stakeholders want answered?

• It is important to understand all points of view, especially those who will be the primary end users of the monitoring data.

• A shared view of the components of the intervention should be reached, though it is not always possible to reach consensus on what defines success of the intervention. Therefore it’s important to capture various stakeholders’ perspectives of what success is.

Example: implementation alone might be success to some; others might be interested in change in outcomes. Key questions may include: how is the intervention being implemented? What do the health care workers think of the new process? How many sites has the intervention been implemented?

What will be the focus of monitoring?• Should be based on what stakeholders define as success and questions they would

like to answer.

• It may not be possible or practical to answer every monitoring question. Selection of which questions to answer will be affected by numerous factors, including stakeholder priority and feasibility and some may be more relevant for evaluation or research.

• Consider monitoring not only the intervention of interest, but also the context in which it is being implemented (e.g. change in health policy, changes in financing, staff turnover)

Example: processes only, knowledge-attitude and practices of caregivers, change in presence of a resource or process (i.e. monitoring chart, checking vaccination cards)

What will be monitored (to answer the key questions)? • An indicator is often used as a measurable marker of change over time for an activity

(e.g. status of the programme, implementation or service delivery). In many cases, indicators provide information about the status of programme implementation (process indicators) or outcomes.

• Indicators should be designed with careful consideration of a number of factors. See guidance for developing indicators and the checklist for selection of high performing indicators which should link to the purpose of monitoring and key questions the process will answer.

• An indicator requires an operational definition and methodologically sound data collection strategy to gather information about it. Not all information needs to be quantifiable; indicators can also use qualitative data. Not everything needs to be measured.

Example: change in health care workers knowledge to checking vaccination history, change in number of health care facilities with up-to-date immunization monitoring charts, number of health care workers trained.

Box 1 How and by whom will this evidence be collected, analysed and reported?• Collection, analysis, interpretation and use of data are time consuming.

• It is not possible to answer all questions, or use existing data to answer all of them.

• Select appropriate methodology to answer the key questions – avoid starting with the methodology, think about the questions and let that guide the methodology.

• Consider:

i. feasibility to collect available or new data and conduct analysis for reporting

ii. what is “nice to know” vs what we “need to know”.

iii. measuring confounders and context (helps with interpretation of the evidence collected).

• Develop a process for routine monitoring of the intervention. This will include knowing:

i. what data to collect/analyse/report;

ii. who will collect/analyse/report it;

iii. when it will be collected/analysed/reported;

iv. how it will be collected/analysed/reported and used.

Example: Data collection (e.g. interviews, surveys, routine reporting of administrative data, supportive supervision), should involve immunization staff at all relevant levels of the health system. Analysis should ideally be done by those with technical expertise to do so, though simple collation of the data may be done by any suitably trained person, including frontline health care workers and vaccinators.

How will the evidence from monitoring be used?• This should be determined from the beginning and included in the monitoring plan.

Also include the mechanisms through which the data will be available (e.g. monthly EPI meeting, annual joint appraisal and/or final report).

• Monitoring data should not be used solely by the end-users. As with immunization coverage (as an example of programme monitoring), it should be used by everyone implementing the programme, project or intervention.

Example: monitoring data will be reviewed and discussed by district staff at each visit to the health facility, monitoring data will be used to determine the extent of implementation and an update presented at monthly EPI meetings.

INTERVENTION GUIDEBOOK 3130

1

2

3

5

7

9

4

10

1

2

3

5

7

9

4

10

6 6

8 816

16 Ogbuanu et al. Can vaccination coverage be improved by reducing missed opportunities for vaccination? Findings from assessments in Chad and Malawi using the new WHO methodology. PLoS One. 2019 Jan 24;14(1):e0210648.

Provide funds for supportive supervision and corrective actions

In principle, the supervision and monitoring of the implementation of MOV activities should take advantage of existing supervisory mechanisms. Although routine supportive supervision should already be funded by the MOH, in many countries, the funds are not available when and where needed. The ICC or similar decision-making body should take measures to ensure availability of personnel and funds for regular supportive supervisory visits, which could involve linking with other departments at the national level or with sub-national authorities, such as local or district governments. In some instances, additional funding for intensified supervisory visits with a specific focus (e.g. to monitor MOV interventions) may be required, and these should be budgeted for, as appropriate.

Similarly, if the supervisory visits reveal that additional corrective actions are required (e.g. more training, additional job aids, etc.), funds for implementing these should be included in the budget.

The MOV Strategy Team should regularly (at a minimum, quarterly), analyse the data from monitoring of the MOV interventions, including data from supervisory checklists. The results should be shared widely with all stakeholders including the front line health workers. Such data should also be used to adapt the MOV interventions as needed, and to determine the need for further evaluations.

Supervision, a key tool to implement routine monitoring A key tool for monitoring the implementation of any intervention is supervision. During the first 6-12 months of the intervention, emphasis should be placed on supportive supervision of routine service delivery as well as any interventions. Institutionalize supervision where it does not exist, ensure that it takes place as scheduled, and/or provide supervisors the tools and funding needed to facilitate regular supervision and monitoring activities.

In 2015, a MOV assessment in Chad found 51% of children had a MOV. As one of the innovative approaches implemented to reduce MOV, “vaccination tokens” were distributed to all curative services at health centers in the implementation districts. Curative health workers were trained to screen the vaccination status of children and record any missing doses on the tokens. If doses were missing, the curative health workers referred the children to the vaccination area, where the vaccinators retrieved the tokens and administered any missing doses. Preliminary data showed a token retrieval rate of 84% and a 12% increase in the number of vaccine doses administered in 2017, compared with the same months in 2016.14 A further evaluation of the impact of the tokens is planned.

COUNTRY EXAMPLE Monitoring of referrals for catch up vaccination in Chad

TASK

8.2

Focused – on specific needs, especially those of the staff being supervised; incorporates on the job training for particular knowledge and skills; and monitors progress towards specified goals;

Friendly – delivered by staff with technical competence and strong interpersonal skills; not fault-finding; encourages open, two-way communication with active listening and powerful questions; incorporates mentoring, facilitates problem solving;

Feedback – provides immediate and ongoing positive and corrective feedback about implementation and progress; highlight what went well, what could be improved and suggestions for how to improve; use data for decision-making;

Follow up –regular re-visits to supervise sites to support implementation, accountability and monitor progress.

Supportive supervision visits should be about helping to make things work, rather than checking to see what is not working.

Supportive supervision should be:

INTERVENTION GUIDEBOOK 3332

1

2

3

5

7

9

4

10

1

2

3

5

7

9

4

10

6 6

8 8

Most countries already have a supervisory checklist.17 The MOV Strategy Team should review the existing supervisory checklists and forms to ensure they include items to allow for monitoring activities to reduce MOV. When necessary and feasible, these should be adapted to capture the key elements of the MOV action plan and the proposed interventions (Box 3).18

The focus of the supervisory visits should be on ensuring that the interventions to reduce MOV are consistently implemented. Any identified problems or deviations should be corrected. To avoid duplication of efforts, the monitoring and supervision plan should strengthen existing supervisory systems, whenever possible. Supportive supervision needs to be systematized and regular.15 Templates for routinely collecting and using data at the level at which it’s collected, and reporting to higher levels should be provided. Data should be regularly collated for onward reporting and to facilitate use.

To ensure ongoing adoption of the MOV interventions as detailed in the work plan, supportive supervision should occur on a regular basis in line with the frequency that monitoring is needed (monthly, bimonthly or quarterly, as appropriate). At the beginning of the implementation of the interventions, monthly to bimonthly supervision and monitoring visits to each health facility may be needed to enhance accountability. A system for publishing monitoring results and for feedback to participants (e.g. health facility staff, district staff) should be created, strengthened or supported as part of the monitoring plan. Follow-up feedback should then be reviewed and followed up on at the next supervisory visit.

17 WHO Regional Office for Africa . Reaching Every District (RED) - A guide to increasing coverage and equity in all communities in the African Region 2017 Edition. (http://www.afro.who.int/publications/reaching-every-district-red-guide-increasing-coverage-and-equity-all-communities)

18 Ward, K et.al. Building workforce capacity through the Strengthening Technical Assistance for Routine Immunization Training (START) approach in Uganda. https://doi.org/10.1016/j.vaccine.2019.04.015

Provide coverage monitoring charts and encourage health workers to update them monthly with coverage estimates

Monitoring MOV interventions should, where possible, also try to strengthen routine monitoring of the immunization programme. This requires an investment in regular monitoring of vaccination coverage, as this is one potential indicator that will be affected by the MOV intervention. In the medium-term, it is expected that reducing MOV will lead to administration of more vaccines which should result in faster depletion of vaccine stocks and an increase in vaccine coverage.

All health facilities should track monthly vaccination coverage using standard monitoring charts. Large, poster-like wall monitoring charts provide constant reminders of targets and help health workers to visualize progress toward achieving them. Each country should have standard immunization monitoring charts which are printed centrally and distributed to all health facilities. The charts should provide a blank space for personalization, such as facility/village name, date, etc. These monitoring charts should be large enough to be displayed and visible to all users of the health facility, as well as for review during community meetings. Proficiency in their use should be included in the training and supervision of health workers. Emphasis should be placed on numerator tracking for different antigens, rather than on estimating coverage per se. Coverage estimation at the health facility level is plagued with inaccurate denominators. Numerator tracking is sufficient to monitor progress from month to month, or to compare with similar months from previous years. See example in Figure 6.

For more on how to make a monitoring chart tracking doses administered and dropouts, please refer to Immunization in Practice, Module 6: Monitoring and surveillance. WHO, 2015.19

19 WHO. Immunization in Practice, Module 6: Monitoring and surveillance. WHO, 2015. (http://www.who.int/immunization/documents/IIP2015_Module6.pdf)

TASK

8.3

INTERVENTION GUIDEBOOK 3534

1

2

3

5

7

9

4

6

8

10

1

2

3

5

7

4

8

10

6

9

1

2

3

5

7

9

4

6

8

10

STEP

Following at least 12-18 months of implementation of interventions and supportive supervision conduct an evaluation of effectiveness of the interventions in selected health facilities or districts

What is an evaluation and what does it involve?Most programmes assess the value and impact of their work when they ask questions, consult partners, make assessments, and obtain feedback. They then use the information collected to improve the programme, which can be defined as “any set of organized activities supported by a set of resources to achieve a specific and intended result.” 20

There are several types of programme evaluation:21

• Formative evaluation ensures that a programme or programme activity is feasible,appropriate, and acceptable before it is fully implemented. It is usually conducted when anew programme or activity is being developed or when an existing one is being adapted ormodified.

• Process/implementation evaluation determines whether programme activities have beenimplemented as intended.

• Outcome/effectiveness evaluation measures programme effects in the target populationby assessing the progress in the outcomes or outcome objectives that the programme is toachieve.

• Impact evaluation assesses programme effectiveness in achieving its ultimate goals.

As with monitoring, evaluation is also different from research. Key differences are summarised in the U.S Centers for Disease Control and Prevention Introduction to Programme Evaluationfor Public Health Programmes: A Self-Study Guide.18 This self-study guide also provides valuable information about the Programme Evaluation Framework22, which outlines the steps to conducting a programme evaluation and the standards to which these should adhere. A condensed version of the framework is available23 and summarized in the next section.

20 CDC Program Evaluation Self Study Guide. Introduction - Distinguishing Principles of Research and Evaluation. (https://www.cdc.gov/eval/guide/introduction/index.htm#Distinguishing)

21 CDC. National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention. Division of STD Prevention. Types of Evaluation (https://www.cdc.gov/std/program/pupestd/types%20of%20evaluation.pdf)

22 CDC. Framework for program evaluation in public health. MMWR 1999;48 (No.RR-11):1-42.23 CDC Evaluation Working Group. Summary of the Framework for Program Evaluation. Revision: August 14, 1999

(https://www.cdc.gov/eval/materials/frameworksummary.pdf)

STEP

WHO MOH and MOV Strategy Team, with support from local core immunization partners

WHEN 12-18 months following the implementation of interventions

TASK 9.1

Following at least 12-18 months of implementation of interventions and supportive supervision, conduct an evaluation of effectiveness of the interventions in selected health facilities or districts

Conduct rapid field evaluation of outcomes/ impact of interventions (12-18 months later)9 TASK

9.1

9

INTERVENTION GUIDEBOOK 3736

1

2

3

5

7

4

8

10

1

2

3

5

7

4

8

10

6 6

9 9

Framework for designing and implementing a programme evaluation Based on the CDC’s Programme Evaluation Framework20, the following summarises the key steps in designing and implementing an evaluation. Examples of how these steps apply to evaluating interventions to address MOV are included.

ENGAGE STAKEHOLDERS

DESCRIBE THE PROGRAM

FOCUS THE EVALUATION

DESIGN

GATHER CREDIBLE EVIDENCE

JUSTIFY CONCLUSIONS

ENSURE USE AND SHARE

LESSONS LEARNED

STEPS

STANDARDSUtility

FeasibilityProprietyAccuracy

DESCRIBE THE PROGRAMME (INTERVENTION)25

A description of the key components and intended outputs, outcomes and impact from the intervention is critical for an evaluation. A description of the intervention/s is commonly presented diagrammatically through logic models26,27 or theories of change23 or through narrative description. Aspects to include in a programme description are:

Need: What is the big public health problem you aim to address with your programme? Often incorporated in aim of the intervention/s. This can be taken directly from the results of the MOV assessment and/or brainstorming session.

Resources/Inputs: What is needed, locally or from the larger environment, for the intervention/s to be implemented successfully? Includes the individuals, resources and organisations that need to take action.

Activities: What will the intervention/s do? Who, how, when, where and why?

Outputs: What tangible capacities or products will be produced as a result of the interventions?

Outcomes – short or long term: What are the long term goals of the evaluation?

Impact: What population-level effect will this intervention contribute to? This should be linked with the need (e.g. reduce vaccine preventable diseases)

Relationship of activities and outcomes: Which activities are being implemented to produce progress on which outcomes?

FOCUSING THE EVALUATION DESIGN28

The broad objective of evaluating specific MOV intervention/s is to understand if any changes have occurred as a result of implementing the specific interventions. Determining the correct evaluation focus is a case-by-case decision. Once there is a clear understanding and consensus on the programme, project or intervention/s, there is a need to focus the evaluation which includes:

a) determining the most important evaluation questions;

b) determining the appropriate design for answering these questions (e.g. deciding when and where to conduct the evaluation);

c) selecting appropriate data collection methodologies (e.g. interviews, surveys, focus group discussions etc.).

25 CDC. Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide. Step 2: Describe the program. (https://www.cdc.gov/eval/guide/step2/index.htm)

26 C4D Hub. Develop program theory or logic model. (https://www.betterevaluation.org/en/C4D-Hub/Define/Develop-program-theory-or-logic-model)

27 Knowlton, L. and Phillips, C. 2013 The Logic Model Guidebook. Better Strategies for Great Results. Second Edition. Sage Publications (http://www.sagepub.com/upm-data/23938_Chapter_3___Creating_Program_Logic_Models.pdf)

28 CDC. Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide. Step 3: Focussing the evaluation design. (https://www.cdc.gov/eval/guide/step3/index.htm)

FIGURE 7. Framework for Programme Evaluation

ENGAGE KEY STAKEHOLDERS24

Almost all programme work involves partnerships. Therefore, any programme evaluation requires considering stakeholders’ values and key questions about the interventions (see Step 8). Engaging stakeholders in the evaluation will help to ensure that their perspectives are understood, thus reduce the possibility that the evaluation findings might be ignored, criticized, or resisted because they do not address the stakeholders’ questions or values. Evaluation of any MOV intervention should be considered as an in-depth exercise that requires engagement from the MOH and MOV Strategy Team, with support from core immunization partners.

24 CDC. Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide. Step 1: Engage Stakeholders. (https://www.cdc.gov/eval/guide/step1/index.htm)

INTERVENTION GUIDEBOOK 3938

1

2

3

5

7

4

8

10

1

2

3

5

7

4

8

10

6 6

9 9

If monitoring data are available and provide sufficient information about processes (i.e. to what extent the intervention has been implemented as intended) the evaluation should focus on intermediate and/or short term outcomes. Examples of outcomes include changes in, service delivery practices; knowledge, attitudes and practices of the health workers; or variations in themes raised by the qualitative assessment components (i.e. HBR being checked at each visit, reduction in caregivers being turned away from vaccination sites due to unavailability of health care worker or vaccine).

If monitoring data are not available or provide insufficient information about processes and outcomes, a process evaluation should also be conducted. This can be done along with outcome/effectiveness evaluation. Although evaluation of impact is the ultimate goal, this is resource intensive, challenging and sometimes not feasible or rational to undertake, especially for programmes, projects and interventions that have only been implemented for a short time but for which change is expected to take a longer time (e.g. behaviour change or impact on disease outcomes).

Regardless of the focus, all evaluations should adhere to the Program Evaluation Standards29 (Box 4).

GATHERING CREDIBLE EVIDENCE30

Gathering evaluation data resembles gathering monitoring data (see Step 8) and having credible evidence strengthens the recommendations that follow from them. All types of data have limitations; therefore, an evaluation’s overall credibility can be improved by using multiple procedures for collecting, analysing, and interpreting data. It is also important that key stakeholders are involved.

As key evaluation questions are often broad and expressed in global or abstract terms (e.g. improve coverage), having indicator/s helps define exactly what is meant and what data are needed (e.g. number of children aged 12 – 23 months receiving second dose of measles containing vaccine in current vs previous year). More information and key references for developing indicators is presented Step 8 (Box 1).

29 CDC (2015). Evaluation Standards, adapted from Yarbrough et.al (2011). Program Evaluation Standards (3rd Edition). (https://www.cdc.gov/eval/standards/index.htm)

30 CDC. Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide. Step 4: Gathering credible evidence. (https://www.cdc.gov/eval/guide/step4/index.htm)

Some approaches to collection of primary data for research, monitoring and evaluation data include desk reviews, surveys, interviews, observations and focus group discussions. Some lesser known approaches include: case studies31, most significant change (MSC)32, reconstructing base line data33

ANALYSE, SYNTHESIZE AND INTERPRET EVIDENCE34

When planning the evaluation, persons responsible for data management, analysis and synthesis should be identified. The focus and approach to analysis of the data collected will be guided by the evaluation questions and indicators. Data management and analysis best practices should be followed at all times, including documenting analytic methods to allow for reproducibility.

Similarly, the approach to synthesis of the evidence collected should be agreed upon by all stakeholders when planning the evaluation. This includes the level of sub-analysis (e.g. by health facility, district, heath worker role etc.) and the format in which synthesized data should be presented (e.g. report, presentation, dashboard, manuscript). Presentation of results should be tailored to the end-users, including their needs, but also their level of understanding of the topic. Box 5 provides some tips to remember when interpreting evaluation findings.

31 Yin RK. Case Study Research. Design and Methods. Thousand Oaks, California: SAGE; 2009.32 Davies, R. and Dart, J. (2005) The ‘Most Significant Change’ Technique - A Guide to Its Use. (http://www.mande.co.uk/wp-

content/uploads/2005/MSCGuide.pdf)33 Bramberger M (2010). Reconstructing Baseline Data for Impact Evaluation and Results Measurement.

(http://siteresources.worldbank.org/INTPOVERTY/Resources/335642-1276521901256/premnoteME4.pdf)34 CDC. Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide. Step 5: Justify conclusions.

(https://www.cdc.gov/eval/guide/step5/index.htm)

The four standards will help inform the selection of indicators and data collection methods.

Utility - serve the information needs of intended users.

Feasibility - be realistic, prudent, diplomatic, and frugal.

Propriety - behave legally, ethically, and with regard for the welfare of those involved and those affected.

Accuracy - reveal and convey technically accurate information.

Hints for conducting strong evaluations available at: www.cdc.gov/eval/strongevaluations

The four standards for high quality program evaluations:

» Interpret evaluation results with the goals ofyour program in mind.

» Keep the audience in mind when preparingthe results. What do they need and want toknow?

» Consider the limitations of the evaluation:biases, validity, reliability

» Are there alternative explanations for theresults?

» How do the results compare with those ofsimilar programmes?

» Do different data sources about the sameissue show similar results?

» Are the results consistent with existingtheories about the change to be observed,usually found through results from existingresearch?

» Are the results similar to what you expected?If not, why do you think they may be different?

Source: US Department of Health and Human Services. Introduction to program evaluation for comprehensive tobacco control programs. Atlanta, GA: stacks.cdc.gov/view/cdc/23472

Tips to remember when interpreting evaluation findings

INTERVENTION GUIDEBOOK 4140

1

2

3

5

7

9

4

6

8

10

1

2

3

5

7

9

4

6

8

10

STEP

1

2

3

5

7

4

8

10

6

9

Evaluation findings should highlight areas that need strengthening and recommend the type of further support needed. If the evaluations show that there are continued weaknesses in some areas or interventions are not suited for purpose, then changing the MOV interventions or strategy is strongly recommended. Nonetheless, the long-term objective is to ensure that reducing MOV becomes a regular part of normal clinical practice in all health facilities and a routine part of sub-national and national immunization programmes.

SHARE LESSONS LEARNED AND ENSURE USE OF EVALUATION FINDINGS35

The MOV Strategy Team has the primary responsibility to ensure that the evaluation findings are disseminated appropriately and should help to facilitate use of the evaluation findings for future work planning by stakeholders.

There are numerous opportunities to share lessons learned across all levels of the health system and beyond - from community health worker meetings, health facility staff meetings, district management meetings, national-level technical working group meetings (i.e. Key decision making meetings for EPI) and other forums, such as EPI managers meetings, ICC and Technical Advisory Group (TAG) meetings, national and regional immunization meetings and conferences. It is important to remember that findings need to be shared with the participants who implemented the interventions (e.g. health care workers, communities), as well as those that participated in the evaluation. Ideally, these participants will continue to implement and benefit from the MOV interventions.

Additional and deliberate effort is required to ensure that lessons learned in the course of an evaluation will translate into informed decision-making and appropriate action, for instance to adapt the interventions based on the evaluation findings. Preparing for use involves strategic thinking and continued vigilance, both of which begin in the earliest stages of stakeholder engagement and continue throughout the evaluation process. Key opportunities should be identified for using evaluation findings to inform strategic priorities (e.g. to justify coverage results, to inform application for funding).

In addition, what has worked in many countries to ensure use of evaluation findings is securing the buy-in of the decision makers at the highest level possible, and ensuring they are involved in or very aware of activities and results from the MOV strategy, particularly the planning, assessment, brainstorming and M&E steps.

Approaches to advocacy may include constant sharing of results at high-level meetings, as well as the successes and challenges encountered. Whenever possible, the results of the evaluation should be presented to other (non-EPI) departments and ministries, to illustrate the successes and highlight continued challenges that are mutually beneficial for these non-EPI stakeholders.

35 CDC. Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide. Step 6: Ensure use of evaluation findings and share lessons learned. (https://www.cdc.gov/eval/guide/step6/index.htm)

Incorporate into long term immunization plans to ensure gains are sustainable10

WHO MOH, with support from MOV Strategy Team and core immunization partners

WHEN Ongoing

TASK 10.1

To ensure sustainability, include interventions to reduce MOV in long-term immunization plans (e.g. cMYP and annual EPI plans)

10

42 INTERVENTION GUIDEBOOK

3

5

7

9

4

8

10

6

G-1

2

11

2

3

5

7

9

4

8

6

10

The MOV strategy should not be conceived as a one-time activity or a project to increase vaccine coverage. Instead, it should be considered as a health system-wide service coordination effort to improve vaccination, as well as other health services. As such, from the outset, the MOV Strategy Teamshould ensure that MOV activities and processes are included as part of country plans such as the cMYP and the annual EPI work plans. The intervention activities should be routinized and sustained, by ensuring the availability of political will and sufficient funding. In general, periodic supportive supervision and monitoring of MOV should continue on a monthly or quarterly basis, as part of the monitoring and supervision plan for health services.

Collaborations with non-EPI stakeholders (e.g. antenatal care, family planning, nutrition, curative services, etc.) built during the brainstorming, implementation and evaluation steps should be sustained. Similarly, EPI staff should ensure that the work plans to reduce MOV do not place undue burdens on staff from these ancillary services. Instead, whenever possible, staffing needs and additional funding for implementation of interventions should initially be supported by the EPI.

If the MOV strategy or interventions were implemented only in selected areas, decisions about scaling up should be based on the findings from the monitoring and evaluation, stakeholder interest and available resources. However, too often interventions that work in small-scale pilot studies fail to be expanded, and if they are, may not have the same success. Similarly interventions that appear successful in one context may fail to transfer to a different context (e.g. from one country to another) largely as a result of contextual differences.

The WHO developed a practical guide to facilitate systematic planning for scaling up health interventions. It is intended for use by programme managers, researchers and technical support agencies who are seeking to scale-up health service innovations that have been tested in pilot projects or other field tests and proven successful.36

Implementation science is a relatively new field, commonly defined as the study of methods and strategies to promote the uptake of interventions that have proven effective into routine practice, with the aim of improving population health. It includes the study of influences on healthcare professional and organisational behaviour37. Including implementation science as part of broader implementation of successful MOV interventions can help to strengthen the potential for adoption and sustainability.38

36 World Health Organisation (2010). Nine steps for developing a scaling-up strategy. Geneva, Switzerland. (http://www.who.int/reproductivehealth/publications/strategic_approach/9789241500319/en/)

37 Eccles MP and Mittman BS. Welcome to Implementation Science. Implementation Science 20061:1 https://doi.org/10.1186/1748-5908-1-1

38 Peters DH et al., Implementation research in health: a practical guide. Alliance for Health Policy and Systems Research, World Health Organization. Geneva (http://www.who.int/alliance-hpsr/resources/implementationresearchguide/en/ )

Annex G: Examples of facility-level practices that may result in MOV

Some of the facility-level practices below may be identified during a MOV assessment, or other programme reviews. Important that the proposed interventions target the specific problems identified.

CATEGORY EXAMPLES OF FACILITY-LEVEL PRACTICES THAT MAY RESULT IN MOV

Caregivers Caregivers don’t bring the child’s HBR when they come for other services. Difficult for the health worker to know what vaccines the child needs.

Caregivers/Health workers

Caregivers bring children to health centres for a specific clinical service (treatment) for the child. Neither they nor the health workers consider this an opportunity to ask about the vaccination status of the children and administer any missing vaccinations.

Caregivers/Health workers

Caregivers, and sometimes health workers, mistakenly consider that if the child is suffering from a cold, fever, diarrhoea or any other mild illness, the child should not be vaccinated that day (“false contraindications”).

Caregivers/Health workers

Health workers and caregivers may mistakenly believe that once children are over a certain age, they are no longer eligible or it is no longer important (or may be dangerous) for them to be vaccinated with missing doses.

Caregivers/Health workers

Sometimes, both pregnant women and some health workers are unaware of the need to provide vaccination against tetanus as part of antenatal care.

Health workersCaregivers bring along their children to the health facility when attending an antenatal visit or other clinical service. As this visit was not specifically for the child, health workers do not ask about the vaccination status of the child.

Health workers Vaccinators do not open a vial of lyophilized vaccine (measles or BCG) because they believe too few children have shown up to justify doing so.

Health workers

Lack of clear information communicated by health workers to caregivers about each vaccine, the recommended age at which the vaccine should be administered and the minimum interval between doses. Often the perception is that the child is fully vaccinated after receiving the first dose, when in reality three doses may be needed (e.g. OPV, PCV or DTP).

Health workers/ Health system

As most health facilities are busy, the health workers are focused on specific tasks (taking clinical histories, examining patients, handling emergencies, etc.) and forget/are unable to ask about what vaccines the child has received. Caregivers seldom request, if the health workers don’t ask.

Health system

The vaccination area is separate from the clinical care/treatment area, and the respective health workers have minimal interactions with one another. As such, children that come for treatment of illnesses do not get the opportunity to interact with or be seen by vaccination staff.

Health systemCaregivers bring their child for vaccination, only to find that on that day no vaccines are being offered because the vaccination staff is not on duty, there are no vaccines available or it is not a scheduled vaccination day.

Health system Caregivers may have to wait a long time before the child receives a vaccine. Because of the delay, some leave the health centre before they receive the needed vaccines.

Health systemSome communities are not adequately informed about where and when to go for vaccination services. Similarly, false information about contraindications confuses the community and may lead to vaccine hesitancy.

INTERVENTION GUIDEBOOK

2

3

5

7

9

4

8

10

3

5

7

9

4

8

10

6 6

H-2H-1

2

11

Annex H: Examples of promotional materials that can help to reduce MOV at the health facility level39

Listed below are suggested materials that can be designed and printed to assist in reducing MOV. The correct design and use of these materials will be critical in optimizing health worker and public awareness. As with any new tools, job aids, or training materials developed, a structured process for systematically introducing and disseminating these is required.

1. Annotated/graphical and simplified vaccination schedule (large poster): This should include all the vaccines and number of doses currently recommended by the MOH. It should be visible to both the health personnel and users of the health services. It should be located inside or near the vaccination service area (as well as near the waiting areas), where the maximum number of people will see it. Finally, it should include a clear message or recommendations for vaccination of children with interrupted or delayed routine immunization.

2. A small copy of the vaccination schedule (job aid): This should be designed like a table-top calendar for use in outpatient services that have large flows of children eligible for vaccination (including the well-baby section, curative services, growth monitoring, nutritional assessments, antenatal clinics, etc.). It should include simplified information on the times and availability of vaccination services.

3. A “Here” poster (bright, attractive, large poster): The objective is to inform the community about the times and availability of vaccination services. It should be visible to all users of the health facility. Suggested location is the registration/waiting area where service users usually congregate. It should also include a space to write down the local vaccination times/clinic schedules as agreed with the community.

4. A “Stop” poster: The aim is to remind service users of the importance of taking advantage of each health visit to vaccinate their children. The best place to locate this poster is just before the health facility’s exit door (gate). (e.g. “STOP! Now that you’re here, make sure your child is fully vaccinated! This way your child will grow up healthy and happy.”)

5. “I support immunization” badge: Following the workshop/training of health facility staff on strategies to reduce MOV, all staff would make a commitment to support vaccination activities at the health facility. This badge is designed to make this commitment apparent. It should be made available to the security staff (watchman), the registration clerk, doctors, nurses, auxiliary nurses, etc. They should be encouraged to wear the badges every day. Other options are armbands, T-shirts or caps.

39 This list of suggestions has been adapted from the a training manual on reducing missed opportunities originally developed by the Colombian Ministry of Health with support from the Pan American Health Organization.

6. Alert stickers or stamps for health records (including vaccination cards, health passports, inpatient files, etc.): The registration staff should pre-screen and place a colourful sticker/flag on the child’s home-based record during initial intake, as a reminder that they are missing some vaccines (“Immunization incomplete”). Ideally they should be placed on the front cover of the child’s health card/home-based record, so they are visible to all health staff.

The stickers should remain attached to the patient record until the child is up-to-date on all recommended vaccines. The stickers will serve as a reminder to any health worker to verify the vaccination status of the child and rule out any contraindications.

7. Ongoing community involvement: Community engagement in immunization is vital to its success. It is important to maintain a spirit of encouragement, good relations and co-operation to achieve the proposed objectives. Vaccination can be attractive to the community because preventative services normally do not incur any costs to the community and have the intrinsic benefit of avoiding many deaths and diseases in children. Having promotional materials for immunization at gathering places such as community centres, churches, markets, etc. can remind and encourage a dialogue about the importance of vaccination.

8. Encouraging the spontaneous demand for vaccination by caregivers requires that members of health facilities fully integrate with them. Health facility staff should endeavour to attend the community meetings, educate caregivers about the importance of immunization and efforts to make services more user-friendly, encourage them to come for full immunization, and respond to any rumours or concerns they may have about immunization.

9. Mass communication: If there is a local bulletin or news media (or other locally effective means of mass communication, such as village meetings and town criers), efforts should be made to capitalize on its reach to educate the community about, and encourage, childhood vaccination.

I support immunization!

H-3

2

21

345678910

INTERVENTION GUIDEBOOK H-4

1

2

3

5

7

9

4

6

8

10

H-5

This document was published by the Expanded Programme on Immunization (EPI) of the Department of Immunization, Vaccines and Biologicals.

This publication is available on the Internet at: www.who.int/immunization/documents

Copies of this document as well as additional materials on immunization, vaccines and biologicals may be requested from:

World Health Organization Department of Immunization, Vaccines and Biologicals CH-1211 Geneva 27 Switzerland

Fax: + 41 22 791 4227 Email: vaccines@who.int Web: www.who.int/immunization/en

For more information visit: http://www.who.int/immunization/

programmes_systems/policies_strategies/MOV/en/