State of Sustainability Planning & Evaluation in Community Health

Sustainability Framework (SF) 101

April 21, 2011IntraHealth with the SHOUT Group

ACKNOWLEDGEMENTS

IntraHealthRenée Kotz, Sara Margolis-Pacqué, Frances Thornhill, Kris Horvath

ICF MacroJennifer Yourkavitch, Ilona Varallyay, Kiersten Johnson, Leo Ryan

SF 101 - Outline

• Part 1: Introduction--“It’s complicated”

• Part 2: Practical Recent M&E Experience

• Part 3: SF 101: – Planning: Small but Critical Steps– M&E: Do we count all trees to measure a forest?

• Conclusion

PART 1 - INTRODUCTIONIt’s not complicated; it’s complex

Current Context Exhibit A: Global Health Initiative

• “The challenge of the next decade and beyond is to take these impressive accomplishments to the next level by helping countries achieve long-term sustainability in their health services.”

• “Building on a long tradition of U.S. government global health leadership and the unprecedented level of commitment manifested in recent years, the Obama Administration’s Global Health Initiative has the opportunity to move global health to a new level of effectiveness, with a vision of long-term sustainability led by partner countries.”

GHI— [Implementation of the Global Health Initiative: Consultation Document]

Do we need a system perspective?E.g. Urban Health System, Bangladesh (Concern)

Health Platform (MESPCC)

NGO Health

Providers

ADB, DfID, USAID, etc.

Govt & Private Health Facilities

Ward Health Committee

Community organizations

Traditional health providersYouth volunteers

Teachers

Private pharmacists

Social & religious leaders

Commissioner

City Government

MOLGRD

Chairman

Health Department

Health Inspector (in absence of Med

Off.)

MOHFW District & Sub-District

MOHFW

Sustainability in HSS: a Complex Equation?

Illustration inspired by Samir Rihani: “Complex Systems: Theory and Development Practice. Understanding Non-Linear Realities.”

“Health -programme sustainability is the ultimate manifestation of a complex web of inter-relations between health concerns, stakeholders, resources, and actions analogous to an ecosystem.”

Gruen et al. The Lancet. 2008

Common Evaluation Questions about Sustainability

Questions I don’t understand:• Is it sustainable?

• Can you ensure it will be sustained?

• Who will take over after you leave?

Questions I understand: Will you leave partners with a better chance of

facing what will come after you leave? Are you at least improving expectations for

continued progress in the context? Are you thinking “within a system?” [a.k.a. is the

“Sustainability Scenario” coherent from an internal development logic?]

Do you have information about where your efforts are heading?

Are your partners mastering the processes which produce the results?

Do the M&E systems inform the local actors about their progress?

Evolving Understanding of Sustainability— A Suggestion:

• An (emerging?) property within a local system which allows interdependent actors to maintain and improve the health status of the system’s (vulnerable) population through negotiated and coordinated social interactions, allowing the expression of their respective and collective capabilities.

Implications of this definition1. Other definitions (i.e. financial viability, organizational

capacity, policy alignment), identify potential determinants of sustainability

2. Projects can contribute to the process, or hinder it, but ultimately neither control nor define it.

3. We can, however, measure progress on hypothesized determinants of sustainability during our projects

4. More than one configuration can support the same sustainable outcome

5. Allowing systems to find their own new equilibrium might be more important than enforcing the best approach

Understanding the Role of External Agents in Building Sustainability

In a Linear Process• Attribution:

Input->Process->Output->Outcome->Impact

• Direct Accountability– “ensure sustainability in X years”

• Fits Top-Down Planning– Caveat:“involve stakeholders”– pre-determined “models”

• “We come, we fix, we go. Hey! They ‘got it’!”

• Evidence? – Partial, sub-systems, very thin.

In a Complex Process

• Contribution:Ownership? Negotiation? Alignment?

• Shared Accountability– “advance capacity & processes”

• Role of “Agents”– Agents have strategies– “models” emerge / constructed

• “Donors and Implementers as Types of Agents.”

• Evidence?– Challenging ; Growing Interest;

More robust?

PART 2 – THREE RECENT EXAMPLES

1. Prospective (USAID/NEPAL)

Point in time evaluation of progress toward sustainability by supported districts

2. Prospective & Retrospective (USAID/CSHGP/Concern Bangladesh)

Planning for sustainabilityWhat has been sustained?How much has been sustained?

3. Modeling Investments (USAID/CSHGP/Save the Children Guinea)

What could we achieve if…?

The Sustainability Framework –Time and Perspective in Evaluation

Prospective: What are the long-term odds based on the results, processes, and systems being built?

Retrospective: Were gains maintained?

Did progress continue? How did system adapt?

Constraints & Black Swans

Implement HSS

Intervention

Intervention Ends

Stakeholders Adapt & Organize

Improvements continue and/or

Achievements are maintained

SF

Source: Taking the Long View: A Practical Guide to Sustainability Planning and Measurement in Community-Oriented Health Programming. 2008. http://www.mchipngo.net/controllers/link.cfc?method=tools_sustain. And: Black Swan, Grey Swan, Sustainability. Or, the difference between Planning and Predicting. Available at: http://cedarscenter.com/resdetail.cfm?resid=125

E.g. 1- Measuring sustainability as a programming tool for health sector investments – USAID Nepal

Using Composite / Dashboard Measures

Intermediate

Promising

Poor

Nascent

Strong

Compo 1 index

Compo 4 index

Compo 5 index

Compo 6 index2

3

Source: Measuring sustainability as a programming tool for health sector investments—report from a pilot sustainability assessment in five Nepalese health districts. Int J Health Plann Manage. 2009 Oct-Dec;24(4):326-50 http://www3.interscience.wiley.com/journal/118903030/abstract

Can districts sustain the health of their mothers & children?

Kanchanpur and Rasuwa Districts, Nepal - 2006

E.g. 2- Sustained Results 5 Years Post-Project. Urban Health / Concern Bangladesh: 1999-2004↨-2009

• From Urban Child Survival Project to Urban Health System Strengthening

– Defining a Sustainability Scenario consistent with national policy

– Defining ±consistent measures (incl. outcomes and repeated capacity assessments)

– Repeated assessments & collective learning

• 18 months post-project coaching (minimal)

→ 3-year and 5-year post-project sustainability assessments

Source: Sustainability of the Saidpur and Parbatipur Urban Health Model (Bangladesh) Five Years After the End of Concern’s Child Survival Project . Final Evaluation Report – January 10, 2010. USAID, Concern Worldwide. ICF Macro/. Available at: http://cedarscenter.com/resdetail.cfm?resid=106

CWI Bangladesh: 1999-2004↨-2009How Much Sustainability in Health Outcomes?

Project Ends

E.g.3- Does it matter? HSS at District Level: Sustainability Investments and Child Deaths Averted

Source: Pro-sustainability choices and child deaths averted: from project experience to investment strategy Health Policy and Planning 2010;1–12. ; doi: 10.1093/heapol/czq042

Save the Children US in Guinea

A Simplified Traditional Scenario for Investment

Total Investment:$7.5 millions

U5 Lives Saved:2,530 [1,569 – 6,167]

Investment Scheme for Maximizing Sustainability – Phase 1

Investment Scheme for Maximizing Sustainability – Phase 2

Investment Scheme for Maximizing Sustainability – Phase 3

Investment Scheme for Maximizing Sustainability – Phase 4

Investment Scheme for Maximizing Sustainability – Phase 5

Total Investment:$7.5 millions

U5 Lives Saved:8,485 [4,169 – 8,909]

Testable Hypothesis: three-time the impact for the same investment

PART 3SF 101 - From Planning to M&E

Practical Resource

www.CedarsCenter.com

ANNEXES

MANUAL

‘Sustainability Manual’ Tools

• Any-Stage Rapid Tool– Sustainability Planning Checklist

• Step 1– Local system mapping and stakeholder analysis– Visioning activity

• Step 2– From Visioning to Developing a “Sustainability

Scenario”

Stakeholders’ Analysis for Sustainability Planning

POWER /

INFLUENCE Low-----------------------------------------High

INFORM & ANSWER CONCERNS

Core of the SystemChange Agents

MINIMAL INVOLVEMENT

KEEP INFORMED & FIND OPPORTUNITIES FOR COOPERATION WITH LOW

TRANSACTION COST

INTERESTLow---------------------------------------------------------------------------------High

‘Sustainability Manual’ Tools for M&E

• Facilitating a detailed planning workshop with local stakeholders

• Measurement tools & Environmental Scan • Notes on basing an evaluation on the

Sustainability Framework

Local actors revise sustainability vision

and scenario

Local actors are supported in

developing a (draft) sustainability vision

and scenario

Project designs interventions to

strategically CONTRIBUTE in the

most effective manner possible to the achievement of

the sustainability scenario

Projects develops its M&E plan, in

order to be accountable for its

progress. Project also adds

element of information in its M&E plan, to help

the local actors measure progress

toward their sustainability

scenario.

Participatory evidence-based

M&E activities are implemented to

help answer: [1] is the project

effective in achieving its objectives?

[2] are there indications of

progress toward the sustainability scenario of local

actors?[3] what lessons

are being learned about the

sustainability scenario (still

valid?), the value of the project’s

contribution (still strategic); and the

role played by local actors (consistent

with their own sustainability

scenario?)

Corrective measures are taken by the project and

by local actors

First Steps in Sustainability Planning– Who Does What?

THE BEAUTY AND DANGERS OF M&E FRAMEWORKS

Component 1—Health Outcomes

Component 2—Services

Component 3—Regional/District MOH Capacity & Viability

Component 4—Local NGO Capacity & Viability

Component 5—Community Capacity

Component 6—“Enabling” Environment

Proportion of pregnant women in targeted households who slept under an ITN the previous night

Quality of existing services

DPHO capacity and viability

Capacity of NRD (local NGO)

Community Health Competency in Malaria

Environment Scan

Proportion of children under five in targeted households who slept under an ITN the previous night

Coverage of health services – facilities + community based

Viability of NRD (local NGO)

Proportion of targeted households with a pregnant woman or children under 5 with at least one ITN

Community-based treatment policy

Proportion of women in targeted households who received two or more doses of IPTp during their last pregnancy in the last two years

Proportion of women in targeted households that cite IPTp as a necessary treatment during pregnancy.

National ACT + IPT supply

Proportion of children under five years old in targeted households with fever in the last two weeks who received treatment with an antimalarial according to national policy within 24 hours of onset of fever

Coverage of community groups

Functionality / Capacity of community groups

Example / Community-Based Malaria Project

Results Framework

IR1 IR2 IR3 IR4

Key Elements for Sustainability in the Results Framework of the Project

Potentially Key Elements for Sustainability in the Larger Environment of the Project and National Stakeholders

Elements in the Results Framework Sustainability Framework

Component 1

Component 2

Component 3

Component 4

Component 1

Component 2

Component 3Component 4

Component 5

Baseline 18 months

Project Objective

Sustainability Scenario

Assessing Progress Toward Sustainability

Project

Accountability

Shared

Accountability

Results Framework Aligned to the SF (Taking the Long View. CARE Nepal)

Dimension 1

IR1: Improved MN outcomes esp amongst marginalized populations

IR2:Improved

maternal & neonatal services

Dimension 2

IR3:Improved

capacity of MOH and

other sectors for MN care

IR4:Improved viability of

maternal and neonatal services

Strategic Objective: Sustained and equitable improvements in maternal and newborn health in the district of Doti and Kailali

especially amongst marginalized populations

IR5:Improved

community commitment for maternal & neonatal

care

IR6:Improved

environment for maternal & neonatal

care

The SF “Dashboard”A Picture Is Worth 1,000 Words But Can Sometimes Tell a Lie

Conclusion• Some Practical Tool and Guidance; No Insurance Policy• Nothing (Really) New Under the Sun, Except Getting

Serious:– Implications for Health and Community Systems– Implications for Scale Up Strategies– Implications for Integration of Health with Sustainable

Development in view of Global Challenges– Implications for Evaluation +++

• Don’t Lose Sight of the Forest for the Trees• The [Sustainability] Questions of Donors are also the

Questions for Countries Themselves

THANK YOU.