WHAT WORKS?

Credit: Gideon Mendel

INDEPENDENT EVALUATION OF THE PILOT CMAM SURGE MODELCONCERN WORLDWIDEMARCH 2015

P E T E R H A I L E Y

C E N T R E F O R H U M A N I TA R I A N C H A N G E

THE SURGE MODEL An innovation that enables the

health system to predict and cope with surges in cases of acute malnutrition through the setting of caseload thresholds and a set of phased actions to respond flexibly to a threshold being met.

a. Planning and Preparedness

b. Response

SURGE MODEL PILOT

SURGE MODEL PILOTMay 2012 – October 2014 (29 months)14 Health FacilitiesMoyale, Sololo and Chalbi.

EVALUATION AIMSExamine if the model works in the way that it had been conceived.Share lessons learnt as others implement the model.

BACKGROUND AND

EVALUATION AIMS

SURGE MODEL PILOT

i. To determine whether the model is effective in setting realistic threshold levels and whether the interventions proposed take place and are appropriate when thresholds are reached,

ii. To determine whether the model positively or negatively influences other health systems activities (facility and district level),

iii. To determine the acceptability of the model to the various stakeholders,

iv. To determine whether the model is more cost effective than previous standard practice of external non-integrated support,

v. To determine the sustainability of the model,

vi. To share lessons learned with involved stakeholders.

EVALUATION OBJECTIVES

SURGE MODEL PILOT

i. How can the Health Facility and SCHMT surge model be improved?

ii. How should the governance and leadership role of the SCHMT and the CHMT for the Surge Model be developed?

iii. How should the Surge Model ensure more community based health system inclusion in the surge model approach?

iv. How can the Surge Model better link to on-going Health and Nutrition Strengthening programming?

v. How can the Surge Model link to and inform the early warning and response systems for Northern Kenya?

SYSTEMS PERSPECTIVE

PRINCIPAL EVALUATION QUESTION

Can the IMAM Surge Model strengthen the health system to manage increased caseloads of acute malnutrition during predictable emergencies without undermining ongoing health systems strengthening efforts?

SURGE MODEL PILOT

Can the IMAM Surge Model strengthen the health system to manage increased caseloads of acute malnutrition during predictable emergencies without undermining ongoing health systems strengthening efforts?

Within the constraints of a pilot programme the Surge Model was found to have:

1. Strengthened the Health Systems ability to manage increased caseloads of acute malnutrition.

2. There was no evidence of negative impacts on ongoing health systems strengthening efforts and to the contrary evidence was found that the surge model approach contributes to improved coverage and improved use of data and communication between the Health Facility and SCHMT.

OVERALL FINDINGS

OVERALL EVALUATION

FINDING.

Surge Model Pilot rated as VERY GOOD- SATISFACTORY.

Therefore, the evaluation recommends further scale up within the pilot sub-counties and at a wider scale in Kenya and elsewhere.

SURGE MODEL PILOT

The present evaluation is very context specific. Both geographically and partnership context are influential.

i. Ensure robust monitoring and evaluation plan is in place at start of scale up. Context will have a considerable influence of the success of the next phase.

ii. The model has not been tested during a extra-ordinary emergency A plan for real time evaluation of its performance during such a situation is required.

The Surge Model pilot focused on the Health Facility /SCHMT pairing.

iii. Ensure a systems approach is taken to scale up. Including development of leadership and governance roles of CHMT and SCHMT. In Kenya the role and links to the NDMA should be included.

iv. Ensure all health system building blocks are included in the systems approach. With particular attention to human resources, financing and supplies.

The pilot was situated in a Government Health System was present and HSS activities were on-going.

v. Embedding the scale up of the surge model in a HSS approach has many advantages. However, the surge model approach could be replicated at health facility level with HSS principals in the absence of Government e.g. Somalia.

KEY ELEMENTS OF SCALE UP

THE IMAM SURGE MODEL

SURGE MODEL PILOT

Mixed methods design.Key informant interviewsFocus group discussions at Health Facility, Sub-County, County and National Level. Visited 9 Health Facilities and 1 outreach, pilot and non-pilot.Data and Information analysis.

EVALUATION METHODOLOGY

PILOT SITESDistrict Weak performance Average performance Strong performance

Chalbi

(4 facilities out of 7)

Folore (level 2)

Kalacha (level 2)

Hurri Hills (level 2) Turbi (level 2)

Moyale

(5 facilities out of 12)

Bori (level 2) Godoma (level 3) Dabel (level 3)

Nana (level 2)

Butiye (level 2)

Sololo

(5 facilities out of 9)

Walda (level 3) Uran (level 3) Ramata (level 3)

Waye Godha (level 2)

Golole (level 2)

SURGE MODEL PILOT

a. Relatively small surges for malnutrition and morbidity experienced throughout the pilot period.◦ Only 1% of months had more than 3 times increase (OTP

3 average/month to 15/average per month).

b. Surge Model built on and in Health Systems Strengthening. Differences between Pilot and Non-Pilot Centres limited.

c. Pilot coincided with devolution process. Difficult to separate effects of Surge Model from those of devolution.

d. Many elements of the Surge Model and the findings of this study are context specific for the Health System in Marsabit with support from Concern.

STUDY LIMITATIONS

TRENDS IN ADMISSIONS

•No seasonal pattern

•Same for SFP and OTP

•OTP: 3 children average per month. SFP: 9 children average per month

•Spikes mostly related to local conflict

TRENDS IN ADMISSIONS

•Seasonal Pattern

•Same for Diarrhoea and Pneumonia

•Diarrhoea: 23 children average per month. SFP: 7 children average per month

SURGE MODEL PILOT

Approach is effective in supporting Health Facilities and SCHMT to manage increases in OTP and SFP admissions without undermining ongoing health and nutrition systems.

o Threshold review and adjustment process should be reviewed. More adjustment according to changes in capacity.

o Change thresholds by more and make bigger intervals between thresholds

oMonitoring system for capacity development required to measure progress towards exit strategy and inform need for changing threshold levels.

o Periodic independent capacity assessment of Health Facilities to add to threshold setting and amendment process and to measure N/HSS progress.

o Due to lack of major surges in pilot period upper thresholds and actions have not been tested. Need to include a specific scenario based approach to setting and review of upper thresholds. Now probably to low.

EFFECTIVENESS

SURGE MODEL PILOT

Approach is effective in supporting Health Facilities and SCHMT to manage increases in OTP and SFP admissions without undermining ongoing health and nutrition systems.

oThe lack of seasonal pattern and demonstrated impact of local conflict suggest the need for more emphasis on scenario planning and preparedness for the thresholds and the response activities. Conflict 1-2 months surge. Drought several months.

oActivities matrix in MoU should be simplified especially for lower thresholds. HF matrix should be simplified and number of activities planned at lower thresholds reduced.

oSimplification in data recording and analysis for the Health System and the Surge Model process are possible.

oData analysis process and use of wall charts should focus more on forward planning and preparedness.

EFFECTIVENESS

SURGE MODEL PILOT

Surge Model plus N/HSS significantly contributed to impact through:o Increased coverage. oUse of data at HF and SCHMT leveloPromoting effective communication

between SCHMT and HF.

oTest the use of “numbers in charge” rather than new admissions to set threshold levels.

oTo Plan, prepare and manage Nutrition and Health System “emergencies” use historic programme data and scenario planning rather than nutrition survey data projections except for rare extra-ordinary emergencies.

IMPACT

SURGE MODEL PILOT

Weak evidence for reduced costs due to use of Surge Model.Due to financial data not being organised to evaluate properly

Many aspects of Surge model at lower thresholds are same as or similar to N/HSS activities. So at lower thresholds costs low.

oUse a Value for Money based approach to examine increased efficiency of system implementing sustained Surge Model alongside NHSS.

oCosting of surge model to be reviewed with simplification at lower thresholds and more attention to higher thresholds.

oActivities matrix too many activities and overly comprehensive especially at lower thresholds.

EFFICIENCY

SURGE MODEL PILOT

The approach was found to be acceptable to all stakeholders and very relevant for Health Facility staff and the SCHMT.oDevelop a systematic satisfaction monitoring system for clients and staff as part of programme accountability agenda.

ACCEPTANCE RELEVANCE

SURGE MODEL PILOT

The approach has established the foundations of a sustainable approach.

oMore decentralization aspects of financial management of surge model further down to Health Facility.

oMore decentralization of supply aspects to of surge model to health facilities as part of ongoing strengthening of the nutrition supply chain. (Push to Pull).

oDevelop Surge Model analysis, monitoring and response approach for SCHMT and CHMT. Use of dashboards.

oConsider including diarrhoea as a Surge Model morbidity.

SUSTAINABILITY

SURGE MODEL PILOT

The approach has established the foundations of a sustainable approach.

oClarify roles and responsibilities, SCHMT, CHMT, for Human Resources secondment in response to surges.

oIncorporate Surge Model approach into County Health System yearly planning process. Historic data, scenario planning and self assessment of capacity.

oIncorporate Surge Model financial planning process into CHMT yearly contingency planning process.

oUse Surge Model approach to clarify the relationship between Health System contingency planning and response and the NDMA crisis modifier funding.

SUSTAINABILITY

Facility Based Nutrition Surge Model in Kenya

V. Large Emergency (<1%)

Serious to Emergency (~1%)

Alert to

Serious (~4%)

Normal to

Alert (95%)P

RO

GR

AM

ME

DATA

Nu

trit

ion

Su

rvey

an

d E

arl

y

Warn

ing

Data

Data for Plan, Prepare and Response

MoH

CO

NTIN

GEN

CY

B

UD

GET

ND

MA

Surge Planning and Financing

THANK YOU