The National Evaluation Platform (NEP):

An introduction for Tanzania stakeholders

Most current evaluations of large-scale programs aim to use designs like this

Impact

Coverage

Program

No impact

No coverage

No program

But reality is much more complex

General socioeconomic and other contextual factors

Impact

Coverage

Routine health services

Interventions in other sectors

Other healthprograms

Program

Other healthprograms

Gov’t Ministries

NGOs

DPs

Many Stakeholders

This presents important challenges for evaluation and accountability

Simultaneous implementation of multiple programs Separate, uncoordinated, inefficient evaluations, if

any Inability to compare different programs due to

differences in methodological approaches and indicators

The NEP aims to help Governments and their partners meet these challenges

5

The NEP approach

Supports answers to “what works” questions in accelerating intervention coverage and reductions in mortality and undernutrition

Addresses integration by assessing multiple programs (i.e. iCCM, nutrition, immunizations, WASH) simultaneously over time

Promotes country ownership of data, builds capacity for evidence-based policies and programs

Facilitates global and national accountability reporting (e.g. MDG, CoIA, SUN, etc)

Common Evaluation Framework

Evaluation data systemwith districts as the units

• District-level databases covering the entire country combining data from multiple sources

• Containing standard information on: Inputs (partners, programs, budget allocations, infrastructure)Processes/outputs (DHMT plans, ongoing training, supervision,

campaigns, community participation, financing schemes such as conditional cash transfers)

Outcomes (availability of commodities, quality of care measures, human resources, coverage)

Impact (mortality, nutritional status)Contextual factors (demographics, poverty, migration)

Permits national-level evaluations of multiple simultaneous programs

A single, integrated data base with districts as the rows

District … …

District 1

District 2

….

Core Data Points from Health and Nutrition Sectors

Core Data Points from Other Sectors

HMIS

DHS

National Health Accounts

Nutrition Surveillance System

Climate Factors

Women’s education

Quality Checking & Feedback to Source

8

Collect new data only as required to fill gaps

• Partner mapping• Strength of program implementation • Quality of care• District-level contextual factors

Types of NEP Evaluation Questions

• Interim (formative) analyses• Are programs being deployed where need is greatest?• Is implementation strong enough to have an impact?• How to best increase coverage?• How can programs be improved?

• Summative analyses• Did programs increase coverage?• Was coverage associated with impact?• Did programs have an impact on mortality and nutritional status?• Are there alternative explanations for the findings?• How equitable are the programs?• How much did programs cost?

Types of comparisons supported by the NEP approach

• Districts with or without a given program – Traditional before-and-after analysis with a

comparison group• Dose response analyses

– Regression analyses of outcome variables according to dose of implementation

• Stepped wedge analyses– In case program is implemented sequentially

12

NEP Project Objectives

With support from Canada DFATD:

1. Develop NEPs in 4 countries, and build local institutional capacity to use and maintain them

2. Demonstrate that NEPs can provide high-quality, timely data on results and implementation strength for use in guiding decisions

3. Use the experience to develop guidelines and tools for use in additional countries

13

NEP Project Overview

• 39 months starting August 2013; 3 months’ start-up and three years’ full implementation

• In each country:– Scope: At least MNCH & nutrition; could broaden if called

for in a particular country– Advisory Committee: Need high-level buy-in from

Government and major partners; can be existing country group

– In-country institutional partner: Credible, stable, opportunities to build sustainable institutional capacity

– Project resident advisor: Can be national or international

14

Major Activities

• Entry – Conduct situation assessment– Meet with stakeholders – Select in-country home institution(s)– Identify Advisory Committee

• Building and populating the data system– Existing data (HMIS, household surveys, service assessments,

budgets)– New data as required (partner mapping, implementation strength

assessments, quality of care assessments, cost-effectiveness)

15

Major Activities - 2

• Establishing processes – Data quality assessments– Feedback to data source for quality improvement– Analysis and response to Government/stakeholder questions– Continuous improvement of NEP functionality

• Using the NEP to strengthen programs – Use of NEP evidence to generate questions and support evidence-based

decision making – Improved reporting on key indicators for CoIA, SUN, etc. – Regular feedback to programs on implementation strength, quality of

care, etc.– Small grants program in program-relevant gender analyses

16

Capacity Building • Mentoring and apprenticeship for home institution(s)

– Technical work for NEP data system development, maintenance and analysis

– “Learning by doing” approach, with continuous TA that evolves over time

• Formal skills-building workshops with policy makers and program planners

– Bringing together NEP home institution and relevant Government counterparts (e.g. ministry

planning /M&E units, TWG members, etc)

– Focus on formulation of policy-relevant evaluation questions and interpretation/application of NEP

findings

– Tailored to country-specific needs

• Cross-country teaching and learning encouraged

Proposed strategies for NEP capacity building

Policy and Program Planners: Workshop 1 (Q2, 2014)

How to define core NEP questionsIdentifying data needs and availabilityData quality assessment procedures

Workshop 2 (Q4, 2014)

Measuring implementation strengthBuilding and maintaining data systemsDose-response analysis

Workshop 3 (Q2, 2015)

Equity analysis techniquesMaps, graphs and tablesReporting on CoIA indicators

Ongoing mentorship and technical assistance to build capacity in NEP home institution (s)

HOW CAN THE NEP PROJECT BEST CONTRIBUTE IN TANZANIA?

Discussion

Asanteni SanaThank You

What are priority questions for the NEP?

What answers do Government and partners need answers to in order to deliver stronger programs for women and children?e.g.,• Do districts with coordinated child survival and

nutrition programs achieve better outcomes?• Are districts using CHW strategies achieving better

results?• Are RMNCH, Nutrition & HIV/AIDS services truly

reaching the poorest and most in need?

What should be the scope of the NEP in Tanzania?

• Nutrition is an essential element of women’s and children’s health.

• How can the Scaling Up Nutrition work being led under PMO-RALG and TFNC be fully integrated into the NEP approach?

Source: MOHSW (Sep 2013) MIDTERM ANALYTICAL REVIEW OF PERFORMANCE OF THE HEALTH SECTOR STRATEGIC PLAN III 2009–2015

NEP Interventions Maternal and newborn Child Household

Contraceptive prevalence# Unmet need for family

planning (FP) Demand for FP satisfied ǂ Antenatal care ( 4+ visits)#ǂ Neonatal TT # IPT for malaria IFA supplementation Skilled attendant at birth #ǂ Facility delivery # C-section rate Postnatal visit for mother #ǂ Postnatal visit for newborn ǂ Early initiation of BF HIV-care (PMTCT / ARV

treatment) #ǂ

All 0-5y Diarrhea care: ORS,

zinc, feeding ITN use # Malaria treatment Pneumonia care

seeking & antibiotic treatment ǂ

High-dose Vit. A supplementation #

Deworming Acute malnutrition

screen & treat

0-5 month Hib3 DPT3 #ǂ Exclusive

breastfeeding ǂ PMTCT+

6-23 month Measles

immunization # IYCF indicators

o Dietary Diversity

o BR duration

Improved drinking water sources

Improved sanitation facilities

Food Security Iodized salt (national

policy; HH utilization) Other food fortification

(by type and scale

# TZ HSSP III indicatorǂ CoIA indicator

EQUITY ANALYSISDisaggregation by gender, wealth, urban/rural, region for core coverage indicators measured through household surveys

NEP Impact Indicators

Maternal mortality ratio #ǂ

Under-5 mortality rate #ǂ

Neonatal mortality rate % Stunting #ǂ

% Wasting #

Cost per life saved, by age

Proposed nutrition-focused % anemia women 15-49 y % anemia in 6-59 month

olds % Acute Malnutrition U5

# TZ HSSP III indicatorǂ CoIA indicator

Key Contextual FactorsCategories Indicators

ENVIRONMENTAL, DEMOGRAPHIC AND SOCIOECONOMICRainfall patterns Average annual rainfall; seasonal rain patterns Altitude Height above sea levelEpidemics QualitativeHumanitarian crises Qualitative

Socio-economic factors Women’s education & literacy; household assets; ethnicity, religion and occupation of head of household

Demographic Population; population density; urbanization; total fertility rate; family size

HEALTH SYSTEMS AND PROGRAMSUser fees Changes in user fees for IMCI drugs

Other Programs The presence of other programs or partners working in MNCH, Nutrition and HIV/AIDS

How might the NEP best be organized in Tanzania?

• Are there groups that can contribute to developing and sustaining the NEP? Who?

• Is there an existing advisory group(s) that would be suitable for the NEP?

• If not, what would the recommended composition of an NEP-specific advisory committee?

How can the NEP contribute to existing Government efforts to strengthen evidence and accountability for

women’s and children’s health?

• Examples of current efforts - Tanzania’s leadership in the Commission on

Information and Accountability for Women’s and Children’s Health

- Ongoing improvement of the HMIS- Scaling Up Nutrition (SUN) monitoring activities- Global initiatives like Countdown to 2015, A

Promise Renewed, Open Data Initiative etc.