EVALUATION OF THE WATER AND SANITATION (WASH) PROGRAMME IN MALAWI (2007-2013)

Final Report

Prepared by

Anton Rijsdijk (Netherlands) and

David Mkwambisi (Malawi)

June 2016

ACKNOWLEDGEMENTS

The evaluation team wishes to thank UNICEF and government of Malawi officials for their support

during the mission. Many thanks to the District Water Authorities for providing assistance during the

field trips and to CPAR for making reports available to the team. The team further appreciates the

contribution from Dr Andrew Safalaoh who participated in data collection and contributed to some of

the chapters. Finally, the team appreciates the comments provided by the validation team and the role

played by the Research Assistant who participated in data collection in the field.

3

TABLE OF CONTENTS ACKNOWLEDGEMENTS .............................................................................................................................. 2

1.0 INTRODUCTION AND BACKGROUND ................................................................................... 15

1.1 General introduction ............................................................................................................................... 15

1.2 The Malawi context ................................................................................................................................. 17

1.3 Malawi WASH Programme ..................................................................................................................... 18 1.3.1 Background and financing .............................................................................................................. 18 1.3.2 Programme objectives ..................................................................................................................... 19 1.3.3 Programme implementation ........................................................................................................... 19 1.3.4 Major deviations during the course of the programme .............................................................. 20

1.4 Structure of the report ............................................................................................................................ 20 2.0 EVALUATION PURPOSE, OBJECTIVES AND SCOPE ................................................................. 20

2.1 Programme logic model .......................................................................................................................... 20

2.2 Purpose of the evaluation ...................................................................................................................... 21

2.3 Evaluation objectives ............................................................................................................................... 22

2.4 Scope of the evaluation .......................................................................................................................... 22 2.4.1 Relevance ........................................................................................................................................... 22 2.4.2 Efficiency ........................................................................................................................................... 22 2.4.3 Effectiveness ..................................................................................................................................... 22 2.4.4 Impact ................................................................................................................................................ 22 2.4.5 Sustainability ..................................................................................................................................... 23

3.0 EVALUATION METHODOLOGY ........................................................................................................ 23

3.1 Overall approach ...................................................................................................................................... 23

3.2 Evaluation Areas and Participants ........................................................................................................ 23 3.2.1 Evaluation Areas .............................................................................................................................. 23 3.2.2 Participants ........................................................................................................................................ 24

3.3 Key informant interviews ....................................................................................................................... 24

3.4 Data collection methods ......................................................................................................................... 25 3.4.1 Household surveys ........................................................................................................................... 25 3.4.2 Surveys among school learners and health centre users ............................................................. 25

3.5 Physical inspection of WASH infrastructure and field observations ............................................. 25

3.6 Literature reviews .................................................................................................................................... 26

3.7 Data processing, analysis and reporting ............................................................................................. 26

3.8 Limitations of the evaluation ................................................................................................................. 26 4.0 EVALUATION FINDINGS ...................................................................................................................... 27

4.1 Relevance ................................................................................................................................................... 27 4.1.1 Introduction ...................................................................................................................................... 27 4.1.2 Assess the internal, vertical and horizontal consistencies between sector policy/strategy and the UNICEF WASH programme ........................................................................................................... 27 4.1.3 Appropriateness of the programme objectives in the overall problem context, needs and priorities of the target groups and beneficiaries .................................................................................... 28 4.1.4 How well has the programme identified the needs of the most deprived populations and how have these been built into programme results and monitoring? ................................................ 30 4.1.5 How well has the programme identified and addressed the challenges of balancing equity and efficiency for optimal results? .................................................................................................................. 30 4.1.6 Proposed Logic Model (Theory of Change) ................................................................................ 31

4.2 Efficiency .................................................................................................................................................... 33 4.2.1 Introduction ...................................................................................................................................... 33

4

4.2.2 To what extent did programme implementation strategies contribute to national results? .. 33 4.2.3 What are the factors affecting the pace and quality of implementation and how can these be mitigated? .................................................................................................................................................... 34 4.2.5 How efficient were the procedures and approaches adopted for infrastructural works and were they in conformity with UNICEF’s regulations? ........................................................................ 38 4.2.6 Reporting and data management ................................................................................................... 38

4.3 Effectiveness ............................................................................................................................................. 38 4.3.1 Introduction ...................................................................................................................................... 38 4.3.2 What actual results were achieved against planned results as outlined in the results matrix and against key performance indicators? ....................................................................................................... 39 4.3.3 How effective were the approaches adopted by the project? (supplementary question) ..... 40 4.3.5 What capacities were developed in the sector as a result of the WASH programme and how did these contribute to the achievement of results? ............................................................................. 42 4.3.6 What advocacy interventions were carried out and how effective have these been? ............. 43

4.4 Impact ......................................................................................................................................................... 43 4.4.1 Introduction ...................................................................................................................................... 43 4.4.2 What impact did the programme achieve against its planned targets? ..................................... 44

4.5 Sustainability ................................................................................................................................................ 49

4.5.1 Introduction ...................................................................................................................................... 49 4.5.3 Capacity of the beneficiaries to maintain the water points ........................................................ 50 4.5.4 What strategies for sustainability have been adopted and how effective have these been? .. 51 4.5.5 What is the extent of programme influence on national level commitment for resource allocations and leveraging ......................................................................................................................... 53 4.5.6 To what extent are results achieved sustainable at their respective levels (National, sub-national, communities) and can they be replicated or integrated into policies, strategies and programmes? .............................................................................................................................................. 54 4.5.7 What competencies within the WASH section have been useful and what are the key human resource and skills gaps? ........................................................................................................................... 54 4.5.8 What is the likelihood of a continued flow of benefits for beneficiaries after the programme end? .............................................................................................................................................................. 54

5.0 LESSONS LEARNED ................................................................................................................................. 55

5.1 Lessons Learned ....................................................................................................................................... 56 5.1.1 Organisational and institutional set-up ......................................................................................... 56 5.1.2 Infrastructure and technical sustainability .................................................................................... 56 5.1.3 Capacity building and awareness .................................................................................................... 56 5.1.4 Baseline surveys, data management and financial transactions ................................................. 57 5.1.5 National coordination and networking in WASH ....................................................................... 57

5.2 Challenges .................................................................................................................................................. 57 6.0 CONCLUSION AND RECOMMENDATIONS .................................................................................. 58

6.1 Conclusion .................................................................................................................................................. 58 6.1.1 Relevance ........................................................................................................................................... 58 6.1.2 Efficiency ........................................................................................................................................... 59 6.1.3 Effectiveness ..................................................................................................................................... 60 6.1.4 Impact ................................................................................................................................................ 61 6.1.5 Sustainability ..................................................................................................................................... 62

6.2 Recommendations ................................................................................................................................... 63 BIBLIOGRAPHY ................................................................................................................................................ 65

5

ANNEXES........................................................................................................................................................... 69

Annex 1: UNICEF-WASH evaluation Matrix ........................................................................................... 69

Annex 2: Household questionnaire .............................................................................................................. 79

Annex 3: Questionnaire for school going pupils ...................................................................................... 89

Annex 4: Checklist for key informants ......................................................................................................... 93

Annex 5: Field research and data collection .............................................................................................. 96

Annex 6: List of people consulted ............................................................................................................... 108

Annex 7: Annual and total results of the UNICEF WASH programme .......................................... 110

Annex 8: Achievements log frame: key outputs and outcomes .......................................................... 111

Annex 9: TOR End of Term Evaluation of Water, Sanitation and Hygiene (WASH) Programme

(2007-2013) .......................................................................................................................................................... 113

Annex 10: Pictures ............................................................................................................................................ 116

LIST OF TABLES

Table 1: District WASH related data between 2007 and 2014 .................................................................. 28

Table 2 Suggested outline of a logical framework with comments on the assumptions .................... 31

Table 3: Statistics on rural water and sanitation between 2008 and 2011 ............................................ 33

Table 4: Cost of constructing one borehole by different organisations ................................................. 37

Table 5: Opinion of the beneficiaries about the water quality after the programme end.................. 39

Table 6: Responses on source of water and its usage ................................................................................. 39

Table 7: Most commonly cited factors which influence the sustainability of Hand Pumps and the results of the UNICEF project ............................................................................................................................ 55

Table 8: Selection criteria for Districts ............................................................................................................ 97

Table 9: Planned sample size ............................................................................................................................ 98

Table 10: Actual sample size for the evaluation for households and schools ........................................ 99

Table 11: Villages were FGDs were held ......................................................................................................... 99

Table 12: Information on gender of the leaner .......................................................................................... 100

Table 13: Villages with borehole and CLTS interventions in Lilongwe ................................................... 100

Table 14: Villages in Mangochi ....................................................................................................................... 101

Table 15: Villages and interventions in Mzimba District ........................................................................... 101

Table 16: Water points’ data........................................................................................................................... 103

6

LIST OF FIGURES Figure 1: Map of Malawi showing districts where the WASH programme was implemented ........... 16

Figure 2: Logical framework for the programme evaluation ..................................................................... 21

Figure 3: Programme logical framework ........................................................................................................ 32

Figure 4 Proposed Theory of Change .............................................................................................................. 33

Figure 5 Water quality of the water points constructed by the programme ......................................... 35

Figure 6 Responses on non-availability of water .......................................................................................... 37

Figure 7 Responses on whether distance to water point has reduced. .................................................. 44

Figure 8: Responses on whether the project has reduced time to access water .................................. 45

Figure 9: Responses by learners on the condition of toilets ...................................................................... 47

Figure 10: State of hygiene of the toilet ........................................................................................................ 48

Figure 11: State of toilet structure .................................................................................................................. 48

7

ACRONYMS AEC Area Executive Committee

AfDB African Development Bank

AM Area Mechanic

CBM Community Based Management

CFR Case Fatality Rate

CIDA Canadian International Development Agency

CLTS Community Led Total Sanitation

COWASH Community-Led Accelerated WASH

CPAR Canadian Physicians for Aid and Relief

DA District Assembly

DC District Commissioner

DCT District Coordination Team

DDC District Development Committee

DDP District Development Plan

DEAP District Environmental Action Plan

DEM District Education Manager

DFID Department for International Development

DHS Demographic and Health Survey

DP Development Partner

FGD Focus Group Discussion

GoM Government of Malawi

HDI Human Development Index

ICEIDA Icelandic International Development Agency

IDA International Development Association (World Bank)

IRC International Rescue Committee

JMP Joint Monitoring Programme

MDG Millennium Development Goal

M&E Monitoring & Evaluation

MoH Ministry of Health

MoIWD Ministry of Irrigation and Water Development

MTR Mid Term Review

NWDP National Water Development Programme

ODF Open Defecation Free

O&M Operation & Maintenance

SWAp Sector Wide Approach

PHAST Participatory Hygiene and Sanitation Transformation

PMU Programme Management Unit

QA Quality Assurance

RWSSP Rural Water Supply and Sanitation Programme

TA Traditional Authority

VHWC Village Health and Water Committee

VIP (latrine) Ventilated Improved Pit (latrine)

VLOM Village Level Operation and Maintenance

WASH Water, Sanitation and Hygiene

WatSan Water and Sanitation

WB World Bank

WMA Water Monitoring Assistant

WP Water Point

WS Water Supply

WSS Water Supply System

U5 (mortality) Under 5 (mortality)

8

EXECUTIVE SUMMARY

Overview of the evaluation

This report provides the findings of the evaluation of the Government of Malawi-UNICEF Water and Sanitation Programme in Malawi. The programme comprised the installation of boreholes in villages, schools and health centres, and support to interventions on sanitation and hygiene (WASH) in communities and schools in 12 out of the 28 districts in Malawi. The overall purpose of the Malawi WASH programme was to improve child health, survival rates and well-being. This purpose was to be achieved by improving sanitation, providing safe drinking water and effective hygiene education. The programme had a particular focus on children and included components for water and sanitation for schools and health centres. The Dutch government, UNICEF and the Government of Malawi funded the programme with contributions in cash and kind from the community beneficiaries. The programme was implemented from 1 January 2007 to 31 December 2013. The evaluation was commissioned by UNICEF Malawi and was carried out by one external (Dutch) and one internal (Malawian) consultant. The evaluation was carried out from September to December 2015 with most of the design and preliminary work being done in August 2015.

Evaluation objectives

The overall aim of the evaluation was to assess the degree to which the programme achieved its objectives. The evaluation set out to:

(i) pinpoint strengths and weaknesses, and identify risks and opportunities for the WASH programme;

(ii) assess the extent to which the programme focused on the deprived and vulnerable, and on systems and capacity development in the sector, including for financing, and

(iii) assess the programme contribution to risk management and internal UNICEF capacity, especially in relation to upstream engagement.

Specifically, the evaluation looked at the Relevance, Efficiency, Effectiveness, Impact and Sustainability of the programme. The evaluation laid a special focus on lessons learned to improve the forthcoming GoM-UNICEF WASH programmes in Malawi. Under Relevance, the evaluation assessed the internal, vertical and horizontal consistencies between sector policy/strategy and the GoM-UNICEF WASH programme, looking at how well the programme identified the needs of the most deprived populations and how these were built into programme results and monitoring. It also assessed the appropriateness of the project objectives in the overall problem context, including needs and priorities of the target groups and beneficiaries, and how well the programme identified and addressed the challenges of balancing equity and efficiency for optimal results. Under Efficiency, the evaluation team looked at the programme implementation strategies and how they contributed to national results. The evaluation also assessed the efficiency of the procedures and approaches adopted for infrastructural works and their conformity to UNICEF’s regulations. It looked, too, at factors that negatively affected the pace and quality of implementation and how can these be mitigated. Under Effectiveness, the evaluation team looked at how far programme objectives were achieved, while Impact looked at the most significant

9

changes due to the project as identified by beneficiaries, and the development of community capacities and how these contributed to sustainability and resilience. Under Sustainability, the evaluation looked at what strategies for sustainability were adopted, how effective these have been, and considered the likelihood of a continued flow of benefits for beneficiaries after the programme end. Intended audience The main users of the report - the Dutch Government, the Government of Malawi and UNICEF Malawi – wish to understand the extent to which the programme was successful and what could be improved upon. The report therefore provides lessons learnt that could be applied to the future scaling up of WASH interventions in Malawi. The findings could be triangulated to other WASH programmes and projects in Malawi, especially those implemented by the NGOs. The evaluation findings and recommendations could also be used to inform the design and implementation of similar programmes at national and regional level in future. Stakeholders that could benefit from the evaluation are the District Executive Committees, District Councils, development partners, education institutions, the UN family, and local and international NGOs.

Evaluation methodology

Wherever possible, both quantitative and qualitative methods were used to answer the evaluation questions. The design of the evaluation did not use experiential (or quasi-experiential) approaches, because of the absence of a control/comparison group and baseline information, which would have allowed a robust measurement of impact. The analysis of change was therefore largely limited to the use of secondary data on programme recipients or target populations. However, an effort was made to collect primary data using a beneficiary questionnaire for households and school-going children in the target areas. In total, 667 households and 115 school going students were consulted.

Qualitative methods were also adopted, in order to gather an in-depth understanding of the perspectives of the various primary targets of the programme (direct beneficiaries, members of Water Point Committees and Village Health and Water Committees (VHWC), area mechanics, district programme staff, and staff from the implementing partner agencies). The evaluation was carried out in three of the target districts - Mangochi, Lilongwe and Mzimba - representing the regional, geographical, economic, cultural and social range of the programme. Besides, overall observation, the evaluation also conducted a physical inspection of the WASH infrastructure (water points with hand pumps and school sanitation facilities) and scored the hygiene status at both community and school levels.

Main conclusions The GoM-UNICEF programme was highly relevant, as it was consistent both with UNICEF goals and

the national policy. The relatively low coverage of water and sanitation facilities in the target areas added to the programme relevance. A weak point in terms of relevance is the rather poor and confusing log-frame, which features outcomes rather than easily measurable outputs. The performance related to relevance is satisfactory.

The efficiency of the programme leaves much to be desired. The programme reached and even surpassed its targets within the timeframe and budget, but short-cuts were made in supervision and contracting, which compromised the quality of construction. Other serious shortfalls include

10

poor data management by UNICEF and, in particular, the lack of M&E in the districts. And so, one may conclude that UNICEF focussed too much on achieving the physical targets instead of concentrating on supervision and on strengthening the capacity of the implementing partners. This all leads to a less than satisfactory performance rate for efficiency.

The programme was quite effective and the majority of beneficiaries now have water of better

quality and closer to their homes. Furthermore, the programme was flexible and made some proper decisions to deviate from the original proposal. On the other hand, despite efforts to train the district authorities the programme did not leave strong and effective districts behind. Nevertheless, the performance rate for effectiveness is satisfactory.

The programme had a positive impact on women and girls in that it decreased their workload and

improved sanitation facilities at schools. However, the positive impact of the programme on school children is compromised by the lack of functioning hand washing facilities at the schools. Due to a lack of baseline data the programme impact could not be quantified, hence no performance rate for it could be established.

Despite the fact that to date the water supply system is running well, the programme has not left a

sustainable water supply behind. Key constraints include the vulnerable economic position of the area mechanics and the inability of the VHWC to collect sufficient funds for major repairs to the hand pumps. The lack of any after care and M&E by the district authorities is not helpful here, hence the performance related to sustainability is less than satisfactory.

Main recommendations

The development of a proper theory of change should be given priority in new projects.

Clear roles and responsibilities should be well defined between UNICEF and the line ministries, and programme partners at district and community level.

Quality control, strengthening of the social infrastructure, and capacity building in the districts should be placed higher on the agenda.

Communications with the districts, especially on financial matters, should be improved.

UNICEF and the district partners should improve their data management, reporting and M&E.

UNICEF and the GoM should focus more on the social aspects of WASH in rural water supply.

UNICEF and the development partners should be more innovative and allow experiments with advanced technical approaches.

The development of an exit strategy is essential and UNICEF should get the commitment of government for sustained financial support for the districts, in order to avoid the loss of valuable trained staff.

Hand washing in schools and villages should be given the highest priority in rural WASH projects.

Rethinking the present sustainability strategy is essential. All the different aspects of sustainability should be reviewed in cooperation with development partners. If necessary, UNICEF could ask for assistance from international knowledge institutions.

11

1.0 INTRODUCTION AND BACKGROUND

1.1 General introduction UNICEF Malawi commissioned an End of Programme Evaluation for the GoM-UNICEF Water, Sanitation and Hygiene (WASH) Programme in Malawi. The programme was implemented from 2007 to 2013, with financial support from the Government of the Netherlands. It aimed to contribute to progress towards the achievement in Malawi of Millennium Development Goal 7.C – to reduce by half the proportion of the population without sustainable access to a safe water supply and basic sanitation. It was implemented through the Government of Malawi, especially district government, and non-governmental organisations (NGOs). The programme was implemented in 12 of the 28 districts in Malawi, covering about 43% of the country. The districts are spread across all three political regions (see Figure 1). They are: Kasungu, Mchinji, Lilongwe, Salima, Dowa, Mwanza, Mangochi, Blantyre, Chitipa, Nkhata-bay, Mzimba and Likoma. The programme targeted vulnerable rural people, with an emphasis on improving the lives of women and children. It focused on the provision of safe drinking water, advocacy, hygiene practices and capacity building among communities, health centres and schools. This was achieved through the provision of sanitary facilities, especially latrines in primary schools and health centres, and the installation of new boreholes and rehabilitation of existing water points in order to make safe water available within walking distance. The GoM-UNICEF WASH programme evaluation was undertaken as part of a Country Programme review which is a key milestone in UNICEF’s country programming process. The evaluation provides an opportunity to undertake a comprehensive review of the progress towards achievement of the results outlined in the current country programme documents. It is also intended to give the country office a basis for making the best strategic choices for the forthcoming country programme.

12

Figure 1: Map of Malawi showing districts where the WASH programme was implemented

The evaluation was carried out in Mangochi, Lilongwe and Mzimba districts, thus representing regional variations, cultural differences, socio-economic status and demographic statistics. UNICEF guidelines (UNICEF 2010) were followed and the evaluation team prepared an evaluation matrix (Annex 1) to answer key questions. Both quantitative and qualitative methods were used. A semi-structured questionnaire (Annex 2) was administered to households that benefited from the programme and a self-administered questionnaire (Annex 3) was given to school pupils. In addition, a checklist (Annex 4) was used to solicit information from key stakeholders including government officials, NGO partners, UNICEF staff and development partners. The evaluation was carried out from September to December 2015 with most of the design and preliminary work being done in August 2015.

13

1.2 The Malawi context Malawi is a small landlocked country in Southern Africa, 118,500 sq. km. in area, located along the Great Rift Valley, with diverse geography of mountains and lakes. In 2008, the population amounted to 13 million people with a growth rate of 2.8 % (NSO, 2008). More than 80% of the population live in rural areas, and are engaged in subsistence, rain-fed agriculture. Despite these adverse conditions, the rural population density is among the highest in Africa. The UN Human Development Report (UNDP, 2015) ranked Malawi as one of the poorest countries in the world, at 173 out of 188 countries and territories on the Human Development Index (HDI), with a value of 0.445 in 2014, which showed a slight improvement on the 2010 value of 0.420. Statistics from the Demographic and Health Survey (DHS) showed that the under-five mortality rate was 133/1,000 live births (DHS, 2004). According to same survey, about 20 per cent of under-five children had suffered an episode of diarrhoea in the preceding two weeks of the survey. Six years later, the DHS showed that the under-five mortality rate was 112/1,000 live births, and about 18% of under-five children had suffered an episode of diarrhoea in the two weeks preceding the survey (DHS 2010). Women’s life expectancy at birth in Malawi increased from 48.9 years in 2007 to 58.2 years in 2013 (WHO). Cholera is endemic and outbreaks may occur in the rainy season (November–April), the severity differing year on year. The 2008/2009 season was a bad one: 5,198 cases (including 113 deaths, CFR 2.2%) were registered throughout the country in 23 districts, with the majority of cases being reported in Lilongwe (central region) and Chikwawa (southern region) (WHO, 2010). In 2008, 77% of the population in rural areas had access to safe water, and this had increased to 82% by 2011, and access to improved sanitation remained more or less constant between 2008 and 2011 (around 55%)A recent and potential serious challenge to the availability of groundwater is the environmental degradation of natural forest and farmland, in combination with the effects of climate change (USAID, 2010).

The United States Agency for Development (USAID), quoting the World Bank, reported that Malawi is considered a water-stressed country, with per capita water availability rapidly declining due to remarkable population growth, especially in the urban and peri-urban areas (USAID, 2010).

In rural areas the main source of water is groundwater from boreholes and to a lesser degree from shallow wells and surface water. The majority of rural people do still drink some water from unprotected sources including rivers, shallow wells and dams. The government of Malawi has put in place policies and strategies to address challenges associated with the provision of safe water.

USAID (2010) reported that despite the challenges of poverty, rapid population growth and shrinking water availability, Malawi has made significant improvements in drinking water coverage since 1990, including a 40% improvement in drinking water coverage. This has helped Malawi meet the Millennium Development Goal (MDG) target for access to drinking water. The World Health Organization (WHO) and UNICEF reported in 2010, however, that Malawi was behind on the MDG for sanitation, as it would need an extra 600,000 people to obtain access to adequate sanitation each year between 2008 and 2015.

The Government of Malawi has in place legal and policy frameworks that are directly linked to the WASH interventions. The Water Resources Act (GoM, 2013) and the Waterworks Act (GoM, 1995)

14

provide the regulatory framework for water resources, supply and sanitation in Malawi. The Ministry of Irrigation and Water Development (MoIWD) developed a National Water Policy that was approved by the Government of Malawi in August 2005 which promotes an integrated approach to water resource management. The MoIWD also developed the National Sanitation Policy in 2008, and a draft Integrated Water Resources Management/Water Efficiency (IWRM/WE) Plan for Malawi in 2007. The MoIWD has the responsibility to regulate, monitor, and disseminate information on water resources and sanitation.

The National Water Policy made local governments responsible for planning and coordination of the implementation of water supply and sanitation (WSS) programmes at the district level (USAID, 2010). Through the decentralization framework, the Ministry of Local Government and Rural Development (MoLGRD) is responsible for implementing the decentralization of the WSS sector. Donors supporting the WASH interventions include the World Bank, African Development Bank, UNICEF, European Union and several NGOs including WaterAid, Care Malawi, Concern Universal, Water for People and Malawi Red Cross.

1.3 Malawi WASH Programme

1.3.1 Background and financing

The Malawi WASH programme was a partnership between the Government of Malawi and UNICEF Malawi. UNICEF contributed financially to the implementation of the programme, however the main funding agency was the Netherlands Government who supported the programme with USD 16.9 million. The programme was led by UNICEF Malawi as the lead agency within the United Nations system in Malawi in rural water supply and sanitation and the major donor in rural WASH1. UNICEF Malawi has been involved in the provision of safe water supply, sanitation and hygiene promotion in rural areas and peri-urban areas for over 16 years. The 2007-2013 WASH programme is the successor to the Dutch funded 2002-2006 WASH programme. The UNICEF programme of cooperation in Malawi (2012-2016) focuses on health, nutrition, basic education, and water and sanitation. In addition, the programme covers child protection and care for orphans and other vulnerable children, and social policy, with the response to HIV/AIDS and human rights cutting across all interventions. Hence this 2007-2013 Dutch funded programme fits well into the long involvement of UNICEF in the Malawi WASH sector. The GoM-UNICEF WASH programme was part of the Dutch Cooperation Programme on Water, Sanitation and Hygiene (WASH) supporting water supply and sanitation programmes in countries in sub-Saharan Africa2. The regional WASH programme was implemented in several countries in Southern Africa in the context of the national efforts of these countries to achieve the Millennium Development Goal (MDG) 7, Target 10C, seeking to halve the proportion of their people without access to safe drinking water and basic sanitation by 2015.

The GoM-UNICEF WASH programme has three phases. The inception phase was for 6 months from

July to December 2006, followed by the implementation phase (January 2007 to December 2012) and the closing phase covered the 6-month period January to June 2013. The total budget of the programme was USD29.1 million, composed of USD16.9 million (58%) from the Dutch

1 Donors in rural water in US$ millions: IDA 2; UNICEF, 30; AfDB, 19; CIDA, 18; WaterAid, 5 (Source: World Bank, 2007) 2 UNICEF-Government of the Netherlands Cooperation Programme on Water Supply, Sanitation and Hygiene in Comoros, Malawi, Mozambique

15

government, USD1.2 million (4%) from UNICEF, USD4.7million (16%) from GoM and USD6.3 million (22%) expected from the communities.

1.3.2 Programme objectives

The programme had the following objectives to be achieved by 2013: (i) Improve access to, and sustained use of, safe water supply from 66 percent to 80 percent

in 12 districts

(ii) Improve access to, and sustained use of, adequate sanitation facilities from 10 percent to

22 percent in 12 districts

(iii) Improve water, sanitation and hygiene in 300 primary schools

(iv) Sustain use of improved hygiene practices one million population

(v) Improve community-based water resources management by the 2800 Village Health and

Water Committees (VHWC)

(vi) Build capacity of partners for effective, efficient and accountable implementation of water

and sanitation

(vii) Improved knowledge management on water, sanitation and hygiene.

1.3.3 Programme implementation

UNICEF’s lead partner at national level was the Ministry of Irrigation and Water Development (MoIWD), with input from the Ministry of Health (sanitation and hygiene components), the Ministry of Education, Science and Technology (school WASH), the Ministry of Gender, Child Welfare and Community Services (mobilisation and sensitisation of communities) and the Ministry of Finance. In the districts, the main implementer was the District Coordination Team (DCT) established by this programme under the District Assembly headed by District Commissioner. DCTs were supervised by the District Water Development Officer (DWDO) and supported by District Environmental Health Officer (DEHO), District Education Officer (DEO) and District Community Development Officer (DCDO). The DCT tasks included planning community work, training extension workers and overseeing implementation of all social aspects of the project. UNICEF Malawi supported the GoM through the MoIWD in the implementation of the programme. Specifically, UNICEF tasks were to:

(i) identify appropriate implementing agencies to undertake the various works on the ground; (ii) assist with the procurement of materials and contractors while building the communities’

capacity to carry out some procurement and to manage programme funds under written agreements with implementing partners, and

(iii) co-ordinate periodic consultative meetings with a broad group of interested government ministries and other sector agencies to discuss progress of implementation, share lessons learnt and plan for the coming years.

International NGOs such as CPAR (in Kasungu and Nkhata-bay) supported the project through a programme cooperation agreement with UNICEF. The NGOs cooperated with the district office in school sanitation, construction of water points, mapping of water sources and preparation of the strategic plans for the district. The tendering and drilling of boreholes was divided among the national government, the districts, UNICEF and NGOs.

16

At the Traditional Authority (TA) level, the extension workers formed a sub-committee of the Area Executive Committee (AEC). The AEC comprised Water Monitoring Assistants (WMA), Health Surveillance Assistants (HSA), Community Development Assistants (CDA) and NGO representatives. These committees liaised between the community institutions and the implementing agencies. At village level, the Village Health and Water Committees (VHWC) took care of operation and maintenance of the facility. The VHWCs were required to provide the agreed community contribution to the project, supervise the construction work and eventually manage and maintain the water supply and sanitation facilities. The VHWC was also responsible for fee setting and collection. Similarly, sanitation clubs at schools (Toto Clubs) in Malawi were expected to be established to facilitate and promote sanitation and hygiene interventions at school level.

1.3.4 Major deviations during the course of the programme

The programme made several key deviations from the original programme proposal submitted to the Dutch government in 2006 (UNICEF, 2006). For example, Result 5 - Improved community-based water resource management by 2,800 Village Health and Water Committees - was cancelled. Results 1.4 and 2.3 that proposed the establishment of 200 sanitation centres/San Marts were also cancelled. Result 4.2 – 12 District Coordination Teams, 500 Health Surveillance Assistants, 150 Primary Education Advisers, 300 teachers equipped with Participatory Hygiene and Sanitation Transformation (PHAST) communication skills - was amended to the Community Led Total Sanitation (CLTS) approach. The implication and impact of these deviations on the project objectives will be discussed in the findings section of this report. 1.4 Structure of the report This report is presented in six sections, which reflect the scope of work of the evaluation as provided in the Terms of Reference. Section 1 provides the introduction and background of the programme including the context in which the programme was implemented and implementation arrangements including programme objectives. Section 2 has evaluation purpose and objectives including the scope of the assignment with specific evaluation criteria. Section 3 provides the evaluation methodology and sources of data and information. The key findings are provided in Section 4, whilst lessons learnt are given in Section 5. Section 6 concludes the evaluation findings and provides some recommendations. Annexes follow the sections.

2.0 EVALUATION PURPOSE, OBJECTIVES AND SCOPE

2.1 Programme logic model

The evaluation team conceptualised a logic model based on the information provided by UNICEF Malawi. It builds on the logic model developed by Goulet and Rodie, which is in turn based on the Wisconsin model that depicts the theory of change as a series of “if-then” relationships. Goulet and Rodie (2013) isolated the following components:

Inputs: Resources that go into the program such as: money, materials, staff among others.

17

Outputs: Activities, products and participation that are generated by the program after resources are inputted.

Outcomes: Results or changes that stem from the outputs of the program. These range from short-term outcomes to medium-term outcomes to long-term impacts.

Assumptions: The beliefs that surround the program in terms of what factors need to be present and the context in which the program works.

Risks: Factors such as resources and/or barriers, which potentially limit program effectiveness.

External factors: The environment in which the program exists and the potential factors that might affect the program.

Based on the programme document, the logic model is presented in Figure 2 below.

Situa on(Seeabove)

ResultsChain

Priori es:Increaseaccesstosafewaterfrom66to80by2011Increaseby12%accesstoimprovedsanita on

• 700,000newwaterusers

• 280,000usersofrehabilitatedwaterpoints

• 180,000pupilsin300schoolsusingsafedrinkingwaterandsanita onfacili es

• Construc onofnewwaterpoints

• Rehabilita onofwaterpoints

• Training• Advocacy• Provisionofsanita onfacili es

FindingprovidedbytheDutchGovernment,GovernmentofMalawi,UNICEFMalawiandcommunitycontribu on

Input Ac vi es Reach

Outputs

TheoryofChangeA=Assump onR=Risks

A: Fund availability to support theinterven onsR:Fundingcutbacks

A:Communitywillpar cipateR:Lackofinterestfromtargetgroup

Improvedchildhealth,survivalratesandwell-being

• Increase access towater

• Water and sanita onfacili esareinplace

• Decrease in waterbornerelateddiseases

• Improved knowledgeonsanita on

• Increased access tosanita onfacili esbypupils

• Improved d i s t r i c tmanagementplans

Short-term Mediumterm Longterm

Outcomes-Impacts

A:Water and sanita on services aredesiredR:Resourcesandknowledgenotfully

u lised

A:Targetpopula onwanttopar cipateR:Targetpopula onnotinterestedwiththeprogrammes

Figure 2: Logical framework for the programme evaluation

2.2 Purpose of the evaluation The evaluation of the GoM-UNICEF WASH programme provides an opportunity for the UNICEF country office to undertake a comprehensive review of the progress towards achievement of the results for WASH outlined in the programme documents for the programme period (2007-2013). The evaluation was framed to assess the programme contribution following the standard criteria of relevance, effectiveness, efficiency, sustainability and impact. The process focused on seeking

18

information based on these criteria in relation to current programming in the areas of WASH. The purpose was to assess the 2007-2013 WASH programme in the light of these criteria in order to contribute to the strengthening of existing programmes and/or further programming. It was envisaged that the evaluation will inform the country programme document (CPD) process for the new country programme, which will commence in 2017. 2.3 Evaluation objectives The objectives of the evaluation were to:

pinpoint strengths and weaknesses, and identify risks and opportunities for the WASH programme;

assess the extent to which the programme focused on the deprived and vulnerable; systems and capacity development including for financing in the sector;

assess programme contribution to risk management and internal UNICEF capacity, especially in relation to upstream engagement.

2.4 Scope of the evaluation

2.4.1 Relevance

The evaluation assessed the internal, vertical and horizontal consistencies between sector policy/strategy and the GoM-UNICEF WASH programme, looking at the appropriateness of the programme objectives in the overall problem context, and to the needs and priorities of the target groups and beneficiaries. Information was sought to show how the programme identified the needs of the most deprived populations and whether such needs were built into programme results and monitoring. In addition, the evaluation assessed whether the programme identified and addressed the challenges of balancing equity and efficiency for optimal results.

2.4.2 Efficiency

The evaluation assessed how the implementation of the programme strategies contributed to national results. It identified the factors affecting the pace and quality of implementation and provided mitigation options. In addition, the evaluation looked at how efficient were the procedures and approaches adopted for infrastructural works and assess whether they were in conformity with UNICEF’s regulations.

2.4.3 Effectiveness

The evaluation identified the actual results that were achieved against planned results as outlined in the results matrix and against key performance indicators. It also assessed how the programme interventions in capacity-building contributed to the achievement of results. This component also looked at how advocacy interventions were carried out and how effective these have been.

2.4.4 Impact

The evaluation assessed the impact of programme activities against planned targets. It identified community capacities and competencies that were developed and how they contributed to sustainability and resilience.

19

2.4.5 Sustainability

The evaluation assessed whether strategies for sustainability were adopted and how effective they have been. It looked at the competencies within the WASH section and whether they have been useful, identifying the key human resource and skills gaps. The evaluation assessed the likelihood of a continued flow of benefits for beneficiaries after the programme, and the extent to which the programme influenced national level commitment for resource allocations and leveraging. Finally, the evaluation assessed the extent to which results achieved are sustainable at their respective levels (national, sub-national, communities) and if they can be replicated or integrated into policies, strategies and programmes.

3.0 EVALUATION METHODOLOGY

3.1 Overall approach The evaluation of the GoM-UNICEF WASH programme used a non-experiential design since no comparison could be made between ‘the actual and the counterfactual (control group)’. Even though it might have been possible to look at the ‘before and after’ situation of the programme in comparable districts sharing similar characteristics, the evaluation period was not long enough to permit this. A challenge to evaluating the programme was the lack of specific baseline indicators. The evaluation had an additional brief to draw lessons learned for the purpose of improving the implementation of similar programmes in future. Taking all of this into account, the evaluation focused on changes before and after the project within the same population. The evaluation process had four component methodologies:

Key informant interviews

Interviews with the households and school going pupils

Physical inspection of WASH infrastructure and field observations

Desk/literature reviews. As in other evaluation assignments (see Goulet and Rodier, 2013), the absence of key baseline information about the population (e.g. health status before the WASH programme, distances required for access to water, water consumption data, assessment of water quality used before the programme, awareness levels, hygiene practices, school clubs on sanitation) made it impossible to evaluate the programme entirely based on quantitative evidence. Qualitative methods were therefore adopted in order to understand the perspectives of the programme stakeholders.

3.2 Evaluation Areas and Participants

3.2.1 Evaluation Areas

The evaluation was carried out in 3 of the 12 districts where the programme was being implemented. The selected districts (Mangochi, Lilongwe and Mzimba) represent the southern, central and northern regions respectively (details in Annex 5). The Yao tribe that is directly linked to the Muslim religion dominates Mangochi, and it has strict cultural and sanitation practices. Mangochi is among districts that have high illiteracy levels (47%) and it is mainly associated with fishing communities.

20

Lilongwe district is in the central plateau area where agricultural production predominates. The majority of the people in the district are Chewas whose illiteracy levels are also high (40%). The dominant cultural practice in rural Lilongwe is Nyau. Finally, Mzimba district has a literacy level of 46% and the settlement pattern is sparse, with hilly and stony landscapes. The districts were also chosen because they generally represent the geographical and demographic features of most of Malawi, which broadly consists of highlands, the central plain and the lakeshore. Coverage of safe water before the programme was also considered, with Mangochi having high coverage (73%) followed by Mzimba with medium coverage (65%) and Lilongwe with low coverage (53%). Within the districts, the Traditional Authorities (TAs) were generally randomly selected from the TA list provided by the client. Based on the guidance received, ten TAs in all were surveyed. Most importantly, only villages, hence households, that had participated in the 2007-2013 WASH programme were involved in the study. In Lilongwe, TAs Chadza, Malili and Masumbankhumba were evaluated, whilst in Mangochi, TAs Chowe, Namavi and Chimwala were covered. Finally, TAs Mbelwa, Chindi, Mzikubola and Kampingo Sibande were evaluated in Mzimba (see Annex 5).

3.2.2 Participants

The evaluation covered both primary and secondary stakeholders. The primary stakeholders were mostly the health centre users, school going pupils, communities accessing water points and villages that were targeted by the programme. This included the area mechanics, committees and those providing the spares. The secondary stakeholders were district officials, government officers, UNICEF experts, NGOs, and other key informants included health experts, teachers and head teachers. 3.3 Key informant interviews The evaluation team consulted several stakeholders to generate an in-depth understanding of the programme and its implementation process. This method provided detailed ‘explanations’ for programme implementation events and complex descriptions of programme implementation processes which could not have been collected from structured interviews. These consultations provided information for analysis of the lessons learnt. They were used to assess: programme status, needs and plans; aspects of organisational development, and the strengths and weaknesses of programme implementation for achieving the impact and outcome objectives. A checklist was developed and administered to UNICEF staff, development partners, external organisations, private sector firms, and other stakeholders (see Annex 6). At national level, interviews were held with UNICEF staff, the deputy director of the MoIWD, staff of other development partners, such as DFID and the National Water Development Programme (NWDP), the World Bank and NGOs. At district level, interviews were conducted with District Commissioners, District Development Committee members, district staff, school and health centre staff, staff of implementing NGOs and people engaged in the private sector. Stakeholders consulted at community level included local leaders, area mechanics, water point committee members, village heads, head teachers, women at water points and local entrepreneurs.

21

3.4 Data collection methods

3.4.1 Household surveys

Although key baseline indicators such as distance to water point, health status and mortality rates were not available, the evaluation undertook a household survey. Within the identified impact villages a total of 667 households were sampled using the ‘snowball’ method. In this method the sample is identified as the survey progresses (Kelly, 2003): one individual is interviewed and he or she is asked to recommend another household that benefited from the interventions, and so on. Survey assistants who were graduates from the Lilongwe University of Agriculture and Natural Resources and from the University of Malawi collected this information. This sampling framework allowed for proxy responses to be provided for beneficiaries with regard to changes that have occurred due to the programme interventions. The questionnaire was administered to selected beneficiaries who had lived in the impact area for more than 6 years. It had several modules that assessed changes in lives and the programme interventions that had special impact. This information included: changes in sanitation status, in water quality and in level of awareness on WASH; incidence of disease; new skills and knowledge gained; participation and engagement in the planning and implementation of interventions. This methodology was chosen because it increased the sample size of beneficiaries and provided the quantitative data required to compare before and after the project and determine impacts and outcomes. In Lilongwe, 9 villages were covered in total, of which two had Community Led Total Sanitation (CLTS) interventions. In Mangochi, all 10 villages had CLTS interventions whilst in Mzimba, of the 24 villages covered, only 5 had CLTS interventions. Detailed information on fieldwork and the sampling protocol is provided in Annex 5. This quantitative approach was then strengthened with qualitative approaches applied among implementers, beneficiaries and the funding agency. 3.4.2 Surveys among school learners and health centre users

Records on UNICEF-supported schools were generally available at the district offices. The team make a random selection from the records, but the final selection prioritised schools with water points installed with UNICEF support. The team visited 17 schools in total, and the interviews with learners targeted those in Standards 6, 7 and 8 who had been in that school for at least five years. Due to the long distances and the fact that most health-related information was collected at household level, only one health centre was visited (Chadza Health Centre in Lilongwe district). 3.4.3 Focus group discussions Two focus group discussions (FGDs) were held in each of the Traditional Authorities covered in this study. The size of the focus groups ranged from 8 to 12 and comprised women only. This decision was made as the best way to gain critical understanding of the issues associated with access to water, health status and sanitation and to discover emerging issues in the WASH sector. The FGDs further allowed the team to compare the attitudes and behaviours of different cultures and tribes on issues associated with WASH. The FGDs allowed triangulation with information collected through semi-structured interviews. 3.5 Physical inspection of WASH infrastructure and field observations

22

The programme assessed the physical status of latrines, water sources, school sanitation facilities and other structures provided by the programme. A physical inspection of the facilities was carried out and different scores were given on the quality or suitability of the facility through a Field Form (see Annex 5 Table 16). At the water point, the team inspected the technical condition and yield of the Hand Pump and the taste of the water. The technical condition of the Hand Pump was checked by assessing the play of the bush bearing and by seeing if water could be pumped with a reasonable yield and without unusual friction. In addition, the condition and hygiene of the apron and drainage channel were checked. The team interviewed the water users and, if available, the members of the VHWC on satisfaction, and on financial and maintenance issues. This approach provided accurate information on the context in which the programme was being implemented, and it complemented information found in documents or derived from discussions and interviews. The 53 surveyed water points represented 1% of the 5,288 new or rehabilitated water points. Despite the small sample the results compared well with the results of previous surveys on the water points (UNICEF 2011 and UNICEF 2011a) and with the results of the household surveys for this evaluation. During household interviews, the survey assistants and team leaders were able to visit and inspect water and sanitation facilities, especially toilets. In some cases, communities were involved during the inspection and this included checking hand washing facilities, the state of the toilet infrastructure, water storage facilities and other sanitation structures such as dustbins, rubbish pits and drying structures. 3.6 Literature reviews The team reviewed reports and relevant documents from UNICEF and GoM, and background literature kindly provided by the MoIWD and other development partners. Some were accessed via internet. 3.7 Data processing, analysis and reporting Quantitative data from questionnaires was analysed using SPSS software. Qualitative data from in-depth interviews or consultations with partners, FGDs, and participant observations were transcribed, coded and then interpreted based on the programme objectives and the assignment, using thematic analysis. To ensure validity and allow for generalisation, data were triangulated across all quantitative and qualitative data collection tools, in order to build consensus on the findings. 3.8 Limitations of the evaluation The most serious constraint for this study was the non-availability of baseline data and the poor data management by both UNICEF and the districts. The team failed to obtain basic information such as number of water points per district, information on construction costs, location of structures and number of beneficiaries. The evaluation was also constrained by the fact that the logic model did not make the relationship between outputs and outcomes clear (see Annex 8).

23

The team emailed a questionnaire to all District Water Offices with a request for data and opinions. Unfortunately, despite reminders from the UNICEF WASH section, the evaluation team failed to get this information. Information on the coverage of safe water and especially on sanitation was often confusing and varied by sources. The other challenge that the team experienced during the survey was the lack of information on the UNICEF-supported structures. Most importantly, information on the names and locations of CLTS villages, and to a lesser degree on location of water points, was inaccurate or lacking. Recent maps showing the location of UNICEF-supported constructions and WASH programmes were not available, often because the structures were at remote locations. The team was dependent on the water monitors in the district to act as guides, but often these people were not available, or the districts no longer employed water monitors. Hence the choice of the TAs was influenced by the availability of a guide in the TA. But even with a guide, several water points could only be found after a long search. The lack of clear identification numbers on the water points further complicated the situation. The assessment of the cost effectiveness of the programme proved to be difficult due to a lack of comparable projects and proper data on beneficiaries. Neither UNICEF nor other organizations, such as the National Water Development Programme, the AfDB project and Concern Universal provided sufficient information to enable a cost comparison on borehole construction.

24

4.0 EVALUATION FINDINGS

This section provides the overall findings of the evaluations. The actual achievements of the programmes are provided in Annex 7. 4.1 Relevance

4.1.1 Introduction

This section attempts to answer the questions listed in the ToRs (Annex 9).

4.1.2 Assess the internal, vertical and horizontal consistencies between sector policy/strategy and the UNICEF WASH programme

The programme is consistent with UNICEF goals on children's health, nutrition and education. It is also fully consistent with both the Millennium Development Goal (MDG) 4 on reducing child mortality and MDG 7C on sustainable access to safe water and sanitation. The construction of private (girl) latrines at schools indirectly contributed to MDG 2 (increased access to children to schools); this intervention particularly reduced absenteeism among adolescent girls. The GoM-UNICEF WASH programme applied the Community Led Total Sanitation (CLTS) approach rather than fully implementing interventions as per guidelines from the National Water Policy (GoM, 2005) and the Sanitation Policy (GoM, 2008a). Findings have shown that the UNICEF’s CLTS approach was consistent with the Open Defecation Free (ODF) Malawi Strategy 2011–2015 (GoM, 2011). UNICEF implemented the programme to a large extent through the districts. This approach was different from other projects such as the National Water Development Programme (NWDP) Phase II which was implemented through a Programme Management Unit (PMU) within central government (World Bank, 2007) or the centrally managed approach to a drilling programme, such as that coordinated by Concern Universal3. The UNICEF project approach strengthened the Malawi decentralization policy of 1998 (GoM, 1998) and is consistent with the decentralization policy of the World Bank (World Bank, 2012) and the Paris Declaration on Aid Effectiveness (UN, 2005)4. Both consultations and literature reviews noted that the project was on course in retaining an explicit district focus within the GoM decentralization policies (see also UNICEF 2011). Other key partners support WASH related programmes in Malawi, for instance, the World Bank funds NWDP Phase II and the African Development Bank (AfDB) funds the Rural Water and Sanitation Strategy. However, national cooperation among the partners has not translated into a fully developed sector wide approach, or SWAp, as the National Sanitation Policy (2008) and the Sector Performance report 2011/2012 proposed.

4.1.3 Appropriateness of the programme objectives in the overall problem context, needs and priorities of the target groups and beneficiaries

For decades, Malawi has ranked low on the United Nations Development Programme (UNDP) Human Development Index (HDI). Key indicators for the HDI include poverty levels and life

3 Personal communication by the project manager 4 Statement 21: Donors commit to: Use country systems and procedures to the maximum extent possible

25

expectancy. While poverty levels have remained very high, life expectancy has shown an increase. Demographic statistics from this evaluation show that the majority of households had only primary education (66.8%) with 22% never having accessed education. Their vulnerability is further linked to their main source of income, which was farming in over 48% of cases. The study areas have many challenges including poor markets, roads and housing. There is therefore strong justification for implementing the GoM-UNICEF WASH programme in rural Malawi. The Demographic and Health Survey 2010 gave the national under-5 mortality rate as 112 per 1,000 live births, and the figure is higher (138) for uneducated, mainly rural mothers. Using the most realistic data, that from the Joint Monitoring Programme (WHO/UNICEF 2010)5, the Malawi safe water coverage (62%) was above the 45% average coverage of improved water in sub-Saharan Africa. The data show that between 2007 and 2014 more were accessing safe water in all districts covered by the WASH programme, with Mzimba (94.3%) having the highest percentage of population accessing safe water (Table 1). The average coverage of safe sanitation in Malawi stood at 56%, which was higher than the average coverage in rural sub-Saharan Africa (26%), but the reliability of the data is questionable in view of the discrepancy between the national data and the district data6. Table 1 shows that in all the districts, there was a percentage improvement in sanitation with the highest in Chitipa (68.5%), followed by Lilongwe (58.4%) and Mzimba (58.2%). However, in some districts, the improvement was low, for instance in Mangochi (22.7%) and Mwanza (26.4%).

Table 1: District WASH related data between 2007 and 2014

District Year % improved sanitation

% Access to safe water

Blantyre 2007 28 66 2014 30.6 79.3 Chitipa 2007 18 66 2014 68.5 84.5 Dowa 2007 13 61 2014 42.9 74.7 Kasungu 2007 24 54 2014 55.6 72.5 Likoma 2007 23 63 2014 - - Lilongwe 2007 9 53 2014 58.4 85.5 Mangochi 2007 15 73 2014 22.7 92.7 Mchinji 2007 11 62 2014 41.3 76.7 Mwanza 2007 5 65 2014 26.4 89.9 Mzimba 2007 15 65

5 The MoIWD-led task force preparing the National Water Development Project Phase II conclude that the JMP figures are most

realistic

26

2014 58.2 94.3

Nkhata Bay 2007 18 67

2014 36.1 75.3

Salima 2007 6 76 2014 45.9 92.6

Source: District Rural WASH plans 2007 and Malawi Endline Survey (2014)

Looking at the achievement of the Millennium Development Goals, the WHO/UNICEF JMP reports (JMP 2004 to 2015) have shown that Malawi is on track with safe water coverage, but has made little progress on sanitation. This requires critical interventions to address challenges in the WASH sector. Despite the unreliability of the data, it may be concluded that the relevance for water and especially sanitation projects in Malawi is reasonably good in the African context. By providing water, the programme addressed key priorities of the target groups, as well as stakeholders at district and national levels. Interestingly, statistics show that Mangochi, Lilongwe and Mzimba made several requests for water points, while some NGOs and partners tend to support the installation of boreholes.

4.1.4 How well has the programme identified the needs of the most deprived populations and how have these been built into programme results and monitoring?

The GoM-UNICEF WASH programme was implemented in 12 out of 28 districts in Malawi, representing 42% of the national population. The criteria for the selection of the districts, as shown in the map in Figure 1, were the under-5 mortality rate and water and sanitation coverage. The actual factors used to identify the districts are not very clear, but the districts gave highest priority to communities in greatest need. The evaluation team initially assumed that the selection of the 12 target districts was based on the division of work in Malawi among the different development organizations, rather than simply on health status and WASH coverage. Later consultations revealed that in 2008, the Government of Malawi with partners carried out a nationwide water assessment. A GPS system was used to map out all the boreholes and other water points at national level. However, the use of the information from that assessment is not strongly linked to the identification of the beneficiaries under the WASH programme being evaluated here. The evaluation found other development organizations supporting the WASH sector in Malawi, namely: Canadian International Development Agency (CIDA) working in Thyolo and Phalombe districts; GTZ implementing WASH interventions in Mangochi, Machinga and Zomba districts; the African Development Bank (AfDB) was involved in Ntchisi and Mzimba districts, and an NGO consortium including Concern Universal, World Vision, Emmanuel International, Oxfam and Fresh Water were supporting ten districts in the southern region of Malawi. The evaluation found that the development partners in the WASH sector tend to implement similar interventions with minimal collaboration, or even consultations with each other. This tends to affect the capacity of the district officials and the sustainability of the interventions. It is further a challenge for the government of Malawi to monitor the outcomes of these various projects at national level.

27

At community level, it is not clear to what extent the programme selected the most vulnerable

people within the districts. Selection was based on demand from the beneficiaries and not on a

needs assessment like that carried out by Concern Universal with funding from DFID6. Both

approaches have their pros and cons. Selection on demand might result in a higher sense of

ownership, but it could favour villages with the strongest (political) voice rather than the villages

most in need.

The districts did not have a proper mapping system or database, hence the team could not

assess if the project had indeed targeted the most vulnerable parts of the district in terms of

WASH needs. This is also the conclusion of the quality assurance report (UNICEF 2011) which

stated that: “The UNICEF method involving a water point coverage map, which highlights the

neediest areas, has not yet been fully integrated into district water point management in any

of the districts".

4.1.5 How well has the programme identified and addressed the challenges of balancing equity and efficiency for optimal results?

The evaluation found that the logical framework in the programme document (UNICEF, 2006) could have provided a more suitable balance between equity and efficiency. The quality of the programme logical framework is only moderate; it lacks hierarchic logic and has a confusing mix of purpose, results, targets and indicators. Assumptions are not always complete. Targets are difficult to measure, for example, one "result" is an" indicator" in another column. The project purpose should be child health, not both child health and access to safe water. There is also a mixture of output indicators and outcome indicators, which are difficult to quantify. In order to remedy these gaps, the evaluation team has provided comments on the assumptions in Table 2.

Table 2 Suggested outline of a logical framework with comments on the assumptions

Overall Objective

Poverty Poverty indicators Assumptions

Project purpose

Child Health Health indicators One not achieved assumption: Water stays high on the national agenda and districts are capable of continuing M&E and after care at the end of the project One missed assumption: Beneficiaries are able to finance major repairs after 10 - 15 years

Results 1) Increased access to water

Water indicators

2) Increased access to safe sanitation

Sanitation indicators

3) Improved WASH in primary school

Improved WASH at schools

4) Sustained use of hygiene practices

CLTS indicators

5) Capacity building of partners

District info management

6) Improved knowledge management

SWAp cooperation

6 Interviews with staff members

28

4.1.6 Proposed Logic Model (Theory of Change)

Based on the gaps identified in the logical framework provided in the programme document, the evaluation team proposes a new model that could be considered in the new programming process. This logic model is provided on the assumption that it will help the stakeholders to plan, implement, evaluate and communicate more effectively.

There is need to understand the problem or the context in which the new programme will be situated. This can only be achieved if a baseline or situation analysis has been undertaken in consultation with relevant stakeholders. The programme planners should clearly identify the global and national development gap that has to be addressed. They should carefully analyze the Sustainable Development Goals (SDGs) that the new programme is addressing. This can be achieved through literature review and consultations.

In addition, there is need to conduct a stakeholder mapping exercise that will allow profiling of all programmes and projects in the WASH sector, where such interventions are being carried out and who are the target beneficiaries. At this stage, the engagement of other development partners should be considered, including NGOs and the key ministries.

After identifying the problem, there is need now to conceptualize the programme logical framework, which will have activities, inputs and impacts (see Figure 3).

Problem:LackofsafewaterandinsuficentknowledgeonWASH

Inputs

Lackofsafewaterandinsufficientknowledgeon

WASH

Ac vi es Par cipa on

Outcomes-Impact

Short Medium Long

Figure 3: Programme logical framework

Figure 3 provides a platform for a new programme that will assess the type of inputs, activities and engagement and the participation framework that will allow the attainment of intended outcomes. This logic model will depict the relationship between what UNICEF and GoM want to

29

and what results they are expected to achieve7. In Figure 4, we attempted to reconstruct a Theory of Change based to better assess the key evaluation criteria - relevance, effectiveness, efficiency sustainability and impact.

INPUTS

ACTIVITIES

OUTPUTS

Money

Time

Humanresource

Web-basedtechnology

Volunteers

Researchcentre

EstablishmentofWASHSocial

LearningForums

Crea onofdatamanagement

systems

EstablishmentofWaterpoint

technicalnode

Forma onoffeedbackmechanism

framework

EstablishmentofWASHSocial

LearningForums

Crea onofscholarWASHinnova on

Clubs

OUTCOMESReducedchildmortality

Improvedaccesstosafewater

Improvedaccesstosanita onfacili es

ImprovedWASH

knowledge

Assump onsWASHremaina

priority

Communi es,schoolsarewillingtopar cipate

WASHbusinessmodelaccepted

3MillionHouseholds

REA

CH

400primaryschools400,000school

leaners

IMPACT:ImprovedChildHealth

Figure 4 Proposed Theory of Change

In this proposed model, certain activities and outputs are identified that will provide measurable outcomes. For example, the establishment of WASH Innovation Clubs both in schools and villages will promote awareness and provide a platform for advocacy on hygiene practices. A feedback mechanism will allow the interventions to be reviewed regularly. Also, the establishment of WASH Social Learning Forums will allow information to be collected throughout the programme.

7 If you don’t know where you are going, you will end up somewhere else - Yogi Berra

30

4.2 Efficiency

4.2.1 Introduction

This section attempts to answer the questions listed in the ToRs (Annex 9). Project efficiency refers to the

extent to which the various activities transformed the available resources into the intended outputs in terms

of quantity, quality and timeliness.

4.2.2 To what extent did programme implementation strategies contribute to national results?

Table 3 provides statistics on rural water supply and sanitation from 2008-2011. The results show that the population accessing water in rural areas increased from 58% in 2000 to 82% in 2011, reducing the number of households using an unimproved water supply. However, even though sanitation results show a positive trend between 2000 and 2011, the findings show that close to half of the population was still lacking sanitation services.

Table 3: Statistics on rural water and sanitation between 2008 and 2011

Attribute Status/condition % Access 2000 2008 2011

Water supply Improved 58 77 82 Unimproved 42 23 14 Sanitation Improved 50 56 53 Shared 21 24 31 Unimproved 7 8 9 Open defecation 22 11 7

Source: WHO/UNICEF JMP 2008 & 2014 The evaluation found that at country level, 1.3 million more people accessed water from new and rehabilitated water points, representing about 7% of the 13.8 million population in rural Malawi in 2013 (World Bank, 2013). However, previous data showed that between 2008 and 2011 there was only a 5% increase in clean water supply (JMP 2014). Our argument is that if the 7% increase can all be attributed to GoM-UNICEF WASH programme, then it would mean no progress in non-UNICEF districts. The same point applies to the contribution of the programme to the ODF campaign. The programme doubtless contributed to the increased access to safe water and to the reduction of ODF in Malawi. However, due to lack of data and institutional memory, the evaluation failed to assess the extent of that contribution at district and national levels. This challenge calls for appropriate data collection and reporting at national district and community levels. One critical observation is that data collection for key development indicators is collected by different agencies using different techniques. This lack of standardisation poses a critical challenge for those making decisions that affect national development. On school WASH, according to the 8th and final programme report, the programme improved the WASH situation in 331 primary schools, increasing the number of schools with safe water from 1,491 to 1,822 (58% to 70%). The numbers of learners per drop hole in UNICEF schools (134 on the average) was in line with the country average (133), but above the country target of 114 learners per drop hole (MoIWD, 2012). This is in contrast to the figures given in the Sustainability

31

Check (MoIWD, 2012a) that reported an average of 73 learners per drop hole. The programme certainly had a positive influence on the national averages for safe WASH in schools and villages, but due to the unreliability of the data, it is not possible to quantify the effects on the national scale.

4.2.3 What are the factors affecting the pace and quality of implementation and how can these be mitigated?

The programme had a somewhat slow start (which is not usual in development projects) but gained speed when drilling and construction began. At the end, it achieved or exceeded all its targets (see Annex 8). The slow start in achieving ODF villages was because of a considered decision to change from PHAST to CLTS. At the end of the programme period (2013) all targets had been achieved (and often exceeded). Due to a lack of information both at the District Water Offices and at the UNICEF Country Office, the evaluation team could not find any records on the differing pace of progress among the districts. Findings show that the borehole tendering and drilling process was divided among the national government (MoIWD), the districts, UNICEF and occasionally NGOs. Initially, most drilling was under direct contracts awarded by UNICEF and the MoIWD. The progress of drilling commissioned by the MoIWD was very slow and at the end of the project MoIWD had drilled only 300 out of the total 3,241 boreholes drilled under the programme. The literature review showed that later on, the drilling contracts were outsourced to the districts. According to the MTR (UNICEF, 2011a), decentralization of major activities like tendering and borehole drilling to district level resulted in more effective and efficient implementation compared to procurement by UNICEF and central government. Consultations revealed that this decision added to the effectiveness and efficiency of the programme. However, some of the findings of the MTR are not confirmed by this evaluation. For example, initially, drilling could only be tendered in batches of 10 to 20 boreholes at a time, due to the low capacity of the districts to manage borehole contracts. Worse still, the disadvantages of the decentralized approach resulted into sub-standard construction work, caused by poor supervision of the drilling activities in the districts. The Quality Assurance (QA) report mentioned serious shortcomings in the siting and construction of boreholes (UNICEF, 2011). In 42% of the cases, the contractors sited boreholes by "divining", a cheap, but a completely nonsense methodology akin to faith healing in medicine. Other contractors applied knowledge of geophysics, but not always using the most appropriate methods. In addition, district staff made a good business by assisting the contractor in siting with such speed that one may fear for the quality of the siting choices (UNICEF, 2011). Additional short-comings include wells which did not penetrate the aquifer fully and poor construction of the apron around the wells. MoIWD (2012) concluded that 36% of the contractors did not use steel mesh in the concrete of the apron (see pictures in Annex 10). However, the majority of the boreholes (82%), produced good quality water in sufficient quantity. These checks at the water points are confirmed by the results of household surveys (see Figure 5) and by the results of the QA (UNICEF, 2011) which reported that 76% of the boreholes had good quality water in sufficient quantity.

32

Figure 5 Water quality of the water points constructed by the programme

Water point checks revealed that 7% of the boreholes had salty water whilst the household surveys indicated that the water was salty in 13% in of the cases in Mangochi, 10% in Lilongwe and 3% in Mzimba. The QA (UNICEF, 2011) reported that water from 12% of the boreholes was salty. Salty water is not always unavoidable. Clever targeting (with the help of geophysics) of the fresh layer on top of the salty aquifer could still result in water of acceptable quality even if the groundwater is salty. Another solution could be the construction of a shallow dug well. Another constraint appears to be the low yield in the dry season (11% of the boreholes in this study and 12% of the boreholes in the QA report had low yield in the dry season). The household surveys found higher figures, which makes low yield to be the second biggest challenge. This could be associated with lower rainfall across the country, which can further result in waterborne diseases due to non-availability of safe drinking water. Reduced flow in the dry season could indicate an improper location of the borehole or that the borehole is not sufficiently deep. In two cases (4%), users mentioned muddy water from the well during the rainy season. This indicates an inflow of surface water into the well through a leaky (or non-existent) sanitary seal around the casing. The 4% is probably a low estimate, because the Sustainability Check (MoIWD, 2012) concluded that 20% of the contractors omitted the sanitary seal. The lack of a sanitary seal is a serious construction flaw because the borehole could deliver contaminated water to the people. The QA report also mentioned that some contractors had used the wrong drilling methods, which could damage the aquifer around the well. The drillers who were directly contracted and paid by UNICEF performed significantly better on nearly all parameters than the contractors with district or MoIWD contracts. This calls for UNICEF to find a better capacity-building mechanism for the government officials, and to put in place district centres for capacity building in WASH related interventions. In the contracts managed by the districts the drillers complained about corruption (for example, asking for favours or selling of water points) by the districts, or delays in producing lists of

33

potential drilling locations. Also late payments by the district was mentioned in one case, and the QA survey (UNICEF, 2011) discovered that questionable procedures were being applied in the awarding of contracts by the districts. On the other hand, the districts complained that UNICEF drillers turned up without any warning, leaving them no time to do the pre-drilling training. According to the QA report (UNICEF, 2011), 8% of the villages did not receive pre-drilling training and another 8% did not receive post-drilling training, but it is not clear if this only applies to "UNICEF contractors” or to all contractors. UNICEF reacted adequately to the findings of the report by organizing a workshop with all concerned (contractors, districts). At the workshop a number of improvements in supervision and contracting were proposed. Findings revealed that in some cases water was not available from water points, for varying lengths of time, and perhaps once a week or once a month. Several reasons for this were provided (see Figure 6). The majority of households indicated a damaged water source (60%), followed by drying up of the water source (32%), while the non-availability of spare parts hardly plays a role.

Figure 6 Responses on non-availability of water

One of the causes for the drying up of water sources could be the inadequacy of drilling practices due to insufficient supervision, as mentioned above. Despite the existence of VHWCs and a functioning network of area mechanics, maintenance is still problematic (see discussion in the section on Sustainability). The drilling costs of boreholes are variable and, according to one expert, even decreasing. Table 4 provides the indicative price of a complete borehole (about 40-50 metres deep, complete with an Afridev hand pump installed) in US$.

Table 4: Cost of constructing one borehole by different organisations

Name of organisation Year Cost (US$)

CPAR (*) 2012 35008 Concern Universal 2015 4000 UNICEF 2007 50009 UNICEF 2011 680010

8 In the framework of the UNICEF WASH programme 9 Costing according UNICEF

34

(*) Partner of UNICEF in the 2007-2013 WASH project The results show that on average the cost of drilling a borehole was US$4000, although the cost might increase because of other factors, such as training (in the case of costs for UNICEF boreholes). The results further show that the costs did not necessarily go up over time - the cost of a CPAR borehole costed less in 2012 than in a UNICEF one in 2007. Comparison is difficult, because some prices include software and supervision costs, as well as training. Probably for these reasons, UNICEF costs appear to be slightly higher than those of its partners. Focus group discussion and consultations with local leaders as well as health centre officials revealed that the programme did not affect any social related issues. The inclusions of local people through committees strengthen the relationship between government officials and the communities. However, it was clear that the communities were not fully involved in the design and implementation of the programme especially budgeting, purchasing of materials and reporting.

4.2.5 How efficient were the procedures and approaches adopted for infrastructural works and were they in conformity with UNICEF’s regulations?

The UNICEF MTR (2011) recommended that the programme should improve its performance at district level. The report pointed out the need for the timely and consistent disbursement of funds, more effective communication, greater emphasis on the coordination of multi-sectoral planning, and keeping to set targets during implementation. The QA survey (UNICEF, 2011) investigated the contracting procedures and concluded that the MoIWD/District contracting procedures in particular left much to be desired and should be upgraded to UNICEF standards. These findings are consistent with the present evaluation, since officials from the three surveyed districts indicated that there was late and irregular disbursement of funds from UNICEF. This affected the implementation of the interventions and the completion of the work plans. As pointed out in section 4.2.3, the supervision by UNICEF of the construction activities in the districts was not adequate, resulting in sub-standard construction work and, in 18% of cases, the DCT did not deliver sufficient training to the VHWC. Also, the exit strategy at the end of the project left much to be desired, resulting in a lack of after-care by the districts (see details in section 4.5.8).

4.2.6 Reporting and data management

The narrative parts of the progress reports are informative and clear, but the management and reporting of data leaves room for improvement. In the progress reports and final report there is hardly any systematic overview of outputs vs. targets (see also Annexes 3 & 4). Most of the progress towards targets was given as outcome data, which makes verification difficult, especially if the relationship between outcome and output is not clear. For example, the final report indicated a total of 3,241 new water points (output) with the outcome of 700,000 users (i.e. 216 users/water point) and 2,014 rehabilitated water points (output) with the outcome of 240,000 users (i.e. 117 users/water point). However, the basis for this data is unclear. The evaluation found that the M&E and data management at district level had several gaps. The conclusions on M&E and data management of the water quality assessment study (UNICEF, 2011) mentioned the poor state of the information management systems. This present evaluation identifies this as one of the critical challenges of the programme. For example, none of the

10 From Sustainability Check (MoIWD 2012)

35

districts which were visited by the team had an adequate and up-to-date mapping and data management system in operation. The district officials in all programme sites failed to provide data to the evaluators. In addition, the structures under the programme did not bear a proper identification mark, other than vague numbers scratched in the plaster. These findings are in line with those of the MTR, which concluded that there was need for adequate project progress reports and documentation of institutional learning needs to be further developed in the project. (UNICEF, 2011a). 4.3 Effectiveness

4.3.1 Introduction

This section attempts to answer the questions in the ToR (see Annex 9).The evaluation process looks at the extent to which the UNICEF-WASH programme’s objectives were achieved and how they were achieved.

4.3.2 What actual results were achieved against planned results as outlined in the results matrix and against key performance indicators?

The programme targets were as follows:

700,000 new users of safe and sustainable sources of drinking water within 500 meters of walking distance of the new facilities constructed.

240,000 users of newly rehabilitated of safe and sustainable sources of drinking water with the rehabilitation and repairs of non-functioning water facilities.

180,000 pupils from 300 primary schools use safe drinking water and sanitation facilities and practice proper use of them.

1,000,000 new sanitation users in 200,000 households, using child- and women-friendly sanitation and water facilities. This number includes an estimated number of 600,000 people who are currently using unsanitary open pit latrines.

1,000,000 new practitioners (including 20% of pregnant women) adopt three key hygiene practices, especially hand washing with soap or ash.

The assessment has found that the programme achieved and even exceeded most of its targets (Annex 7 & 8) although the link from output to outcome is not very clear. The programme was effective in delivering water of good quality (Table 5) to rural people in the target districts.

Table 5: Opinion of the beneficiaries about the water quality after the programme end

District Water quality Improved No change Worsened

Mzimba 76.0 24.0 0.0 Lilongwe 69.5 27.0 3.5 Mangochi 84.0 9.0 7.0

The evaluation found that the improvement of water quality was confirmed in all schools covered. District consultations and analysis of documents revealed that the programme provided safe

36

drinking water to 1.3 million people. However, if there had been better targeting of the villages, the programmes would have been more effective on provision of water. Information from household surveys revealed that the project was indeed effective in reducing water related challenges faced by the communities (Table 6). The results show that on average, 94.2%, 85.2% and 92.2% of the households were using a borehole as the main source of water for drinking, hand washing and cooking respectively, which is good progress from the coverage in 2008. Some communities were still using shallow wells for cooking (average 5.8% across the districts surveyed, with a much higher percentage (10.5%) in Lilongwe district). The results further show that in Lilongwe 53% of households were accessing safe water, while the figures for Mangochi and Mzimba were 73% and 65%respectively.

Table 6: Responses on source of water and its usage

District Usage Water Source (%) Borehole Shallow well Unsafe water

source No Water within the

village

Mzimba Drinking 93.0 5.0 2.0 0.0 Hand- Washing

90.5 6.0 3.5 0.0

Cooking 91.0 6.5 2.5 0.0 Lilongwe Drinking 89.5 9.5 1.0 0.0

Hand- washing

79.0 17.0 4.0 0.0

Cooking 88.0 10.5 1.5 0.1 Mangochi Drinking 100.0 0 0 0.0

Hand- washing

86.0 0.5 9.0 4.5

Cooking 97.5 0.5 0.5 1.5 Total Drinking 94.2 4.8 1.0 0.0

Hand- washing

85.2 7.8 5.5 1.5

Cooking 92.2 5.8 1.5 0.5

More than 72% of the respondents indicated that the nearest water source was closer than before the programme, thus reducing the average time it takes the beneficiaries) to access water to less than 30 minutes. This is a positive outcome for women, who have more time to carry out other household chores including taking care of their children. All the communities can now access safe water very much more safely. The evaluation failed to obtain actual figures for the number of users per water point. For optimum effectiveness, the recommended number of persons per water point would be 250, however the number was quite variable in the programme. The data from the Sustainability Checks by the MoIWD that used different categories are more or less comparable to the UNICEF findings (MoIWD, 2012). Having very few households per water point is not efficient (unless justified by need because the area is sparsely populated), and very many households per water point endangers sustainability because of heavy use of the infrastructure.

37

4.3.3 How effective were the approaches adopted by the project? (supplementary question)

The implementation of the programme through the districts was successful in terms of quantity (all proposed boreholes drilled or rehabilitated), but this achievement had its consequences in the technical quality of the boreholes (as described in the previous section on Efficiency). The approach used by UNICEF in the Malawi programme is commonly used in many rural WASH projects around the world as well as Malawi (see AfDB, 2009). The Malawi programme is based on a demand responsive approach, with obligatory contributions to be made by the local population (mainly by providing building materials), thus ensuring their ownership of the facilities. The beneficiaries received both pre- and post-construction training and the programme created VHWCs with a focus on female membership. On the technical side, the approach was also fairly standard. The well-known Afridev hand pump was installed on all the water points, and care takers of the hand pump were trained in maintenance, while advanced technicians were trained up for more complicated repairs. The sale of spare parts for the hand pump is left completely to the private sector (local traders and a national chain of supermarkets, such as Chipiku). Both the social and technical approach on rural WASH (including the commercialisation of the spare parts system) is in line with the policy of the government and other donors). On hygiene, the CLTS approach is standard in many development projects. The programme strategy was effective in ensuring clean water to the communities, but has sustainability constraints (see section sustainability). 4.3.4 How effective was the flexible programme approach? (supplementary question) The programme demonstrated an ability to change course in view of lessons learned. The evaluation found that in several cases, the programme made a considered choice to deviate from the original programme proposal submitted to the Government of Netherlands. For example, the evaluation found that in the case of Result 5 - Improved community-based water resource management by the 2,800 VHWCs - implementation of the sub-components was only attempted during the first two years, when a total of 211,800 trees and 2,000 fruit trees were planted in three of the 12 districts. However, these efforts were discontinued when it was felt that this was turning into a mere re-afforestation initiative, without a clear link to water resource management.

The cancellation of the water resource management component after two years was a sound decision. Water resource management projects are very relevant in view of large-scale deforestation in the country and the unpredictable effects of climate change. According to the World Bank (2007) the availability of water resources in Malawi is considered satisfactory on the whole, but per capita water availability is declining at a rapid rate due to population growth, and Malawi may start experiencing water stress after 2025. However, water resource management is a complicated activity, involving many different components, such as hydrogeology, soil erosion, agriculture and social issues), and should only be carried out by specialized agencies, not as a side- show to a rural water supply programme. The programme cancelled the proposed establishment of 200 sanitation centres/San Marts. Consultation revealed that UNICEF had doubts about the sustainability of these centres. This was a sound decision because in literature reviews their sustainability is questioned (Rijsdijk, 2013). In addition, the programmes abandoned the PHAST approach and applied the CLTS methodology in rural WASH. UNICEF thus followed the latest trends on rural WASH, albeit a bit late because CLTS was already the preferred approach in many rural WASH projects before 2008.

38

On the other hand, the programme was not very imaginative in its approach to rural water supply. This was also observed during the Mid-Term Review (UNICEF, 2011) where it was reported that the programme did not sufficiently emphasize different technology choices and low-cost options – only high-cost boreholes were constructed as the new water points. The MTR further concluded that the programme had limited technical solutions for sanitation and hygiene development, e.g. it focused on VIPs to replace latrines without ventilation. Document analysis and study of the MTR revealed that in the last years of the programme alternative options were considered. For example, in 2011, 76 new shallow wells and 20 gravity piped water supply systems were constructed, and in 2012, 236 new shallow wells, 69 rehabilitated shallow wells and 25 taps/piped WSS were built. As far as village water points were concerned, the programme installed only (Afridev) hand pumps and did not consider alternatives such as small-scale solar power reticulation systems. In densely populated villages, solar powered WSS are a sustainable and effective alternative to hand pumps.

4.3.5 What capacities were developed in the sector as a result of the WASH programme and how did these contribute to the achievement of results?

Consultations at community level revealed that the programme built local capacity, especially where communities were involved in installation of water points and received training. This allowed communities to be equipped with knowledge on how to carry out work on their own, in contrast with cases where NGOs or contractors were involved without the participation of communities. In addition, communities were fully involved in CLTS measures, as shown by their readiness to carry out CLTS activities even after UNICEF support ended. At national level, the programme facilitated the development and approval of the Sanitation Policy, which called for sanitation issues to be recognised at district level. The policy allowed for CLTS issues to be advocated at district and community levels. At district level, capacity was enhanced through the provision of computers and vehicles, including motorbikes, to all the districts participating in the programme. The programme developed the capacity of District Coordination Teams (DCT) by providing them with periodic briefings. DCT members were trained in several areas, including the integrated approach to WASH, which enabled them to develop the District Strategic Investment Plans (DSIP). These DSIPs helped the district to develop baseline indicators on the status of WASH. The DCTs were also trained to produce Annual Work Plans. The programme was a core facilitator in bringing the sectors of water, community development and health together for the first time at country level. Bringing officers from these sectors together helped them to deliver their interventions using a more coordinated approach. Most of the districts participating in the programme were able to produce local WASH maps using GPS tools. These maps assisted the districts in identifying the number of non-functional boreholes. District consultation revealed that there was no direct funding from Government on WASH related interventions. The GoM-UNICEF WASH programme was the main funding agency for WASH interventions. Districts therefore depend a great deal on donors, including DFID, AfDB and CIDA, to support the WASH programme. Consultations revealed that most development partners are still using structures that were put in place under the UNICEF funded WASH programme, such as the case of ICEIDA funded interventions in Mangochi.

39

At community level, the capacity related interventions consisted of orientation and training provided to VHWCs, however in 16% of the cases the VHWC did not receive either the pre- or the post-drilling training (UNICEF, 2011). There was no clear intervention to enhance the capacity of communities to manage basic sanitation practices. In addition, it would be possible for the government to introduce some short courses on borehole maintenance and sanitation marketing through Community Colleges. The training to area mechanics was successful, because the area mechanics gained the necessarily technical skills to carry out major repairs on hand pumps. However, these training sessions should continue on a regular basis into the future, because some area mechanics might move away to the towns where there may be better opportunities for them. The evaluation found that teachers and health workers were participating in capacity building interventions on an ad hoc basis, rather than as planned interventions. Sanitation lessons were not provided to school learners, who could have passed the messages on in their villages. There is need to have properly designed capacity building interventions for schools, colleges and health centres. There was a lack of clear capacity building interventions at community level, and so the evaluation has mixed findings here. Some communities contributed to the programme whilst other regarded the interventions as coming from government. There is a case in Mangochi where some materials were provided to the communities, but they were not eager to contribute simple materials such as bricks, sand or stones. Communities are supposed to understand their responsibility to do something to balance the WASH subsidies. For example, in some cases, the evaluation found that CLTS – driven by the people - has complemented the district efforts.

4.3.6 What advocacy interventions were carried out and how effective have these been?

Both focus group discussions and household interviews failed to get information related to advocacy, or indeed awareness-raising at community level. The findings have shown that although this intervention was included in the programme document nothing tangible was implemented in the districts visited. The programme could have used several ways of advocating the WASH components through schools, health centres and community groups. For example, all the schools visited indicated that the Toto Clubs were not engaged in the advocacy part of the programme. In addition, there was no deliberate effort to make sure that community interventions were providing programming feedback to upstream structures. Future programming should clearly design an advocacy framework that will involve relevant stakeholders, including the media, to make sure that clear messages are given to the targeted communities. University students could be engaged to stay in the communities and implement several awareness raising interventions, at the same time acting as role models for the youth. 4.4 Impact

4.4.1 Introduction

This section attempts to answer the questions in the ToR (see Annex 9).The evaluation looked at whether the WASH interventions resulted in improving access to quality water, hygiene practices and general livelihoods of the targeted communities. In particular, the evaluation looked at the proposed causal change or logic model of the WASH problem by examining change in the identified

40

determinants of WASH related problems such as behaviour, attitude, health and hygiene status and to some extent environmental and societal factors.

The evaluation was carried three years after the programme closure and all recommendations deal with the situation at programme closure. Any changes in the approach of UNICEF in the period since the programme end have not been considered.

4.4.2 What impact did the programme achieve against its planned targets?

(i) Impact on access to water As explained in the section on effectiveness, the programme has tremendously improved access to safe water by the rural population. The results of the evaluation showed that 84% of the respondents indicated that their main source of water was a borehole funded by UNICEF. In addition, over 50% of the respondents indicated that water point is much closer now than before the project (see Figure 7).

Figure 7 Responses on whether distance to water point has reduced.

This outcome is of significance to beneficiaries as they can have enough time for other productive work, especially women. The impact was also very clear among school going girls who indicated that they no longer take time to access water, leaving more time for their studies and other educational activities. The assessment did not examine the effect of reducing the distance to water point on pass rates or absenteeism. Figure 8 presents responses on whether the programme interventions reduced the time spent accessing water. Most of the responses confirm that the programme did reduce this time. Reduction in time taken to access water allows women to undertake other household chores, and gives them more time to take care of their children.

41

Figure 8: Responses on whether the project has reduced time to access water

In terms of time spent, most of the respondents indicated that they take less than 30 minutes to access water. However, in some communities especially Mangochi, the results found that the project has only partially reduced time, because the increase in population means that the waiting time at the water point is longer. A married lady at Nankungu School said: The borehole has helped women to access water near homes. This has even allowed young girls to have time for education unlike before the borehole. The disease incidences have greatly reduced and most of women go to the lake to wash clothes as part of a social outing. Community meetings with women revealed that provision of water has also improved personal safety, since before the programme, women had to get up in the very early morning to go to fetch water because the distance to the water point was quite long. A school girl at Chadza Primary school in Lilongwe said: When I was in Standard 3, I used to accompany my mother to draw water almost 6kms from home before the project, but now the water is just close my home and I have enough time for study before coming to school. The programme has also had an impact on teachers and their families. For example, some schools that did not receive funding under the programme face critical challenges for water. One effect of this is that the teachers have to travel long distances to fetch water. This is especially a problem for young male teachers, and it influences their decision to teach in rural areas. But with funding from UNICEF, certain primary schools now have water available, and teachers have more time to prepare their lessons and support learners. There has been a remarkable reduction in dropout and/or absenteeism rates among girls because of the improved menstrual hygiene facilities introduced under the project. For example, in Mzimba it was reported that the presence of separate girl and boy toilets and hand-washing facilities enabled girls to still attend classes during their menstrual periods, unlike before. This means that there has been greater equity in access to schools. This has also helped to reduce teacher/pupil conflicts, because pupils used to bother teachers for drinking water when there

42

were no boreholes. In some incidences, the lack of drinking water in school stopped some pupils from attending. In terms of economic benefits, the project has reduced the expenses for treating water and for health problems. Before the project, communities used to buy or receive chlorine for water treatment. Due to the project, over 80% of the households surveyed no longer need to treat their water. Focus group discussions (FGD) showed that Health Surveillance Assistants (HSA) stopped donating chlorine, which contributes to the sustainability of the programme. In summary, the programme has had a positive impact on women’s workload by reducing the time and effort spent fetching water. In addition, the programme decreased absenteeism among school children, especially girls, and reduced the costs incurred by using chemicals for water treatment. (ii) Impact on health The household survey revealed that 76%, 70% and 85% of respondents from Mzimba, Lilongwe and Mangochi respectively indicated that water quality had improved. FGDs at TA Bakili and TA Makanjira revealed that before the project, communities were accessing water from the lake for household use. Despite some treatments like boiling and filtering, the incidence of disease was quite high before the programme. Household surveys showed that the incidence of disease reduced due to the programme. On average 81% of the respondents indicated this, with Mangochi delivering the highest rate (86%). These results were confirmed by Health Centre workers in Lilongwe, who indicated that in TA Chadza there has not been a single incidence of cholera in the last 7 years. Moreover, in Mzimba, no cholera cases have been reported since 2008 and there has been an 81.5% reduction in cases of diarrhoea. However, health statistics without a proper baseline study are often unreliable and notoriously difficult to interpret. Progress data on sanitation and hygiene education could function as proxy indicators. Surveys with learners confirmed that the programme improved the sanitation infrastructure and access to safe water, despite the fact that the communities also access water from most of the schools with improved WASH facilities. The majority of learners said that the toilets were reasonably clean (see Figure 9), and over 80% of learners indicated that they use the toilets available at the school. Learners now spend less time finding a decent toilet, and have no need to use the bush and other areas. Indirectly, the improved school sanitation infrastructure has contributed to better hygiene in the communities and has reduced the incidence of disease.

43

Figure 9: Responses by learners on the condition of toilets

The survey results showed that the lack of learning materials and the shortage of teachers were the main factors impacting on learners’ education, and not primarily sanitation facilities or associated health factors. Hand washing in schools leaves room for improvement, especially in terms of the hardware. None of the schools had a functioning UNICEF hand washing facility. In most cases only the concrete stand was left. At least11 6 schools had buckets with water on metal frames placed in front of the classrooms, but none of the facilities had soap. One challenge facing school hygiene is theft by pupils of the buckets provided, but despite these problems schools took the hand washing issue seriously. It was noted that when lower classes had gone home, some students were stealing their hand washing materials. This prompted the schools to remove the buckets and soap very early in the day, thus preventing students from senior classes from using them. A mechanism needs to be found to deter students from stealing the materials. It should also be noted that some schools complained that people from the village were using school toilets. In general, the hygiene of the surveyed school latrines was reasonable (41% good, 53% reasonable, 6% poor). The hygiene of the schoolyards was good (81% good, 19% reasonable, 0% poor). On the level of awareness, the evaluation found that the project did not have a vibrant sanitation awareness model being advocated among learners. Results showed that 39%, 48% and 78% of learners from Mzimba, Lilongwe and Mangochi respectively were washing hands after using the toilet. The high percentage in Mangochi was directly linked to religious practices rather than interventions from the project. Concerning the situation in the communities, the hygiene around the borehole left much to be desired, as in only 38% of cases was it considered good, in 51% it was moderate and in 11% of cases the hygiene was really poor. In nearly all cases this was caused by poor infiltration of

11 Not counted because some schools were visited after school hours

44

wastewater from the well into the soil, with pools of stagnant water accumulating around the wells, occasionally with happily splashing pigs. Household findings revealed that, in general, the state of hygiene of household toilets in all the districts was good (see Figure 10). However, only 59%, 55% and 54% of households in Mangochi, Mzimba and Lilongwe respectively confirmed that the toilet structure was clean/in good condition.

Figure 10: State of hygiene of the toilet

The level of poverty and cultural issues both contribute to the standard of toilets and also to accommodation structures. The findings revealed that 70%, 57% and 58% of the respondents in Mzimba, Lilongwe and Mangochi respectively considered the hygiene of their yard to be clean.

Figure 11: State of toilet structure

45

In many communities the use of sanitation facilities has not been fully adopted especially in relation to hand washing with soap, despite the fact that 83.5% of the respondents indicated that they have toilets. A direct link was observed between the poor quality of dwelling houses and high poverty levels, especially in Mangochi and Lilongwe. In summary, and although no baseline study is available, it can be said that progress in school latrine construction and the greater cleanliness of household facilities have reduced diseases related to water and sanitation among both school learners and village people. Progress with hand washing is only moderate; the construction of hand washing facilities left much to be desired, as none funded by UNICEF was operational. Fortunately, the schools faced with these challenges took on the responsibility to install makeshift facilities themselves. 4.4.3 Developing community capacities and competencies At health centre level, the majority of respondents (59.2%) indicated that they had experienced an improvement in cleanliness, which was probably due to the presence of toilets and hand washing facilities. For example, in Lilongwe district the general consensus among respondents was that the WASH project had improved the health status of the target communities. In terms of habits, the increased access to improved toilet facilities in schools and health centres has improved sanitation practices such as hand washing and the use of rubbish pits. The evaluation found that there are several impacts in terms of community empowerment. This has been in several areas, including community empowerment to monitor projects and activities of contractors; increased ownership of activities through establishment of CLTS, and enhanced local capacity such as via the training of area mechanics and committees. However, no community capacities have been developed to build their own permanent toilets after the programme end. In summary, many communities, especially in Lilongwe, have toilets that leave a lot to be desired. There is need for the GoM to provide guidelines to all partners on the standard toilet for rural as well urban Malawians. Any future WASH programme should avoid supporting communities with resources to construct toilets as advocated by several NGOs. Also, the awarding of certificates to villages that have stopped open defecation is a mockery to the nation. Instead, the primary and secondary school curricula should be designed to include education on such sanitation issues.

4.5 Sustainability

4.5.1 Introduction

This section provides ‘field facts’ from the evaluation survey. The sustainability of the water points is assessed based both on the results of the field surveys, the findings of a technical mission on water quality (UNICEF, 2011) and the Sustainability Check (MoIWD, 2012). The remainder of this section attempts to answer the specific questions in the ToR (see Annex 9). 4.5.2 Field facts on the condition of the water points The evaluation team surveyed 53 water points in total - 22 in Lilongwe, 21 in Mangochi and 10 in Mzimba. All were boreholes, fitted with Afridev hand pumps. The majority (64%) of the surveyed boreholes were in good working order, but a sizeable proportion (34%) of the hand pumps were suffering from overdue maintenance. In only one case was incompetent repair the reason for the non-functioning of the borehole. This is more or less in line with the observations of the area mechanics who estimated that between 0% to 10% of all (UNICEF and non-UNICEF) water points in

46

their area did not work. The final Sustainability Check (MoIWD, 2012) reported that 94% of the water points were functioning. The QA report (UNICEF, 2011) stated that 29% of the boreholes were in good condition and 71% required some repairs; also, 3% of the hand pumps were permanently out of action (compared to 2% found in this study). The difference between the surveys might be attributed to the methodology used 12 or sampling differences. Despite these differences, one can assume that the number of functional water points and the state of maintenance did not deteriorate between 2011 and 2015. Reduced flow in the 53 water points in the dry season (affecting 11% of the boreholes) could indicate unsuitable location, or that the borehole was not sufficiently deep. If the water table drops in the future due to poor water resource management, these wells could dry out permanently. The apron around the well was in reasonable condition in 63% of the boreholes, with occasional cracks in the drainage channel (27% of the cases). The condition of 10% of the aprons was poor. Some had large cracks in the concrete platform, which in combination with a leaky sanitary seal, could lead to contamination of the well. It is likely that the rather high percentage of aprons (37%) with some damage is due to improper construction. Poor construction of the boreholes and of the concrete aprons is the result of insufficient supervision by the districts. This certainly will have negative consequences on sustainability, but it is not possible to know to what extent. Households gave several reasons why they did not access water at certain times from particular water points. In the majority of cases the reason was that the water source (60%) was damaged, and in 32% of cases it was because the water source had dried up. However, the survey revealed that spare parts are easily accessible, which should make the boreholes sustainable. The number of users per water point influences sustainability. More users than the design capacity (about 250 users) will negatively influence the lifespan of an Afridev hand pump. According to the QA report (UNICEF, 2011) and the Sustainability Check (MoIWD, 2012) about a quarter of the water points are overused.

4.5.3 Capacity of the beneficiaries to maintain the water points

All surveyed water points were maintained by a VHWC or by the school. The VHWCs had about 50% female membership, and were able to do maintenance and simple repairs. They knew where to find an area mechanic for extensive repairs. All VHWCs collected funds for maintenance or repairs of the hand pumps, but it appears that gradually the VHWCs are shifting from the collection of funds for preventive maintenance to the collection of money only when the need arises. The QA report (UNICEF, 2011), quoting the Sustainability Check (MoIWD, 2012), mentioned that an average of 18%-21% did not have a prepaid system for maintenance. This evaluation found that this percentage has increased to 53%. From the fact that 98% of the water points were still functional at the time of the survey and the state of maintenance did not decrease between 2011and 2015 illustrates that the beneficiaries are capable of maintaining their hand pumps. However, the VHWCs’ shift from collecting funds for

12 The UNICEF 2011 study focused on the down-the-hole parts of the Afridev (the seal of the piston) while the present study inspected the bush bearings and checked on possible anomalies during pumping.

47

preventive maintenance to collection of money only when the need arises left little confidence that they had the ability to pay for major repairs. In cases where the borehole was owned by the school and used both by the school and the community, maintenance was shared by the school and the community, although the team received some complaints that the community neither paid for nor cared for the hygiene around the water point.

4.5.4 What strategies for sustainability have been adopted and how effective have these been?

The findings of this evaluation study give the government of Malawi, UNICEF and other partners the basis for considering alternative strategies. For example, there is need for strengthening the capacity of the VHWC to collect and keep funds for maintenance. However, studies throughout sub-Saharan Africa have shown that the stability and the capacity of the VHWC are two of the main bottlenecks in the sustainability of rural WASH. The introduction of bank account as a strategy for sustainability has not worked in this programme since most of the accounts were not used. Instead, the new programme could try the Village Savings and Loans approach, whereby a business model could be developed for water issues at community level. To date, however, no clear solutions to this problem have been found. One of the interesting approaches is the phone banking system being developed by GOAL Uganda. In addition, leading organisations in rural WASH such as IRC and WaterAid might be a source of useful ideas. All major donors have created a system of rural technicians (area mechanics) who can do the more advanced types of repairs. In the framework of the WASH programme, UNICEF claimed to have trained 600 area mechanics. This number could not be confirmed, yet the system is still working well and plays an important role in the sustainability of the water points in the districts. The six area mechanics interviewed provided a fairly consistent picture of their trade. They were happy with their jobs and some proudly showed their Identity Cards, but they could not make a real living of their profession. One had a small bicycle repair business, whilst others were farmers. Their monthly income from maintaining hand pumps ranged from MK2,000 (US$3.5) to MK10,000 (US$18). They charged between MK1,000 and MK3,500 per repair. Some complained about non-paying or late-paying customers. Area mechanics felt that they were competent to do all repairs. According to them, the number of non-working hand pumps ranged from 0 to 10 %. The main reasons for this non-functionality were that the villages could not pay for the spare parts, and that the villagers had damaged the well and were not able to repair it. The system of area mechanics is essential to keep the boreholes working. 72% of the VHWCs had consulted an area mechanics to assist them with difficult repairs in the past. The importance of having competent area mechanics was confirmed by the District Water Officer in Mzimba who stated that a major reason for the failure of water points in the district was incompetent repairs by the VHWC. These mainly consisted of dropping items in the borehole which could not be recovered, causing the borehole to be written off. The UNICEF WASH programme equipped all boreholes with the Afridev hand pump - a Village Level Operation and Maintenance (VLOM) hand pump. The VLOM design of the Afridev has its pros and cons. Afridevs can indeed be repaired by the villagers but a steady flow of spare parts is

48

required. Thanks to the standardization of the Afridev, the chain of spare parts is not a major constraint any more. However, its open source design holds the risk of poor quality, of the hand pump itself and of spare parts. The QA report (UNICEF, 2011) mentioned that most Afridev pumps originated from reputable Indian factories, but it also discovered several cases of poor quality (or non-standard) parts. Replacement of an Afridev at the end of its service life is also problematic. WaterAid (2013) estimated the service life of a properly installed hand pump in a properly constructed borehole to be about 10-15 years depending on the use (i.e. size of the user population) and the care it receives. Replacement and major repairs are probably beyond the financial capacities of the local villages.13 On the other hand, the Afridev hand pump was developed in the 1980’s and one may wonder if the hand pump concept does not need a major overhaul. With current technology it should be possible to design hand pumps which are both affordable and of better quality than the present Afridev. In conformity with the approaches of the major donors, the trade in spare parts is completely commercialized. The availability of spare parts was not problematic in any of the surveyed districts, even if not all parts were standardized14. Although some VHWCs mentioned the long distance to shops, nearly all VHWCs and area mechanics knew where to get the parts. Indeed, the spare parts supply system is improving. A manager of a large chain of shops (Chipiku) in Malawi informed the team that due to increased demand the shops will again stock spare parts, after the discontinuation of this service years ago. Experiences of other donors support this positive trend. AfDB (2009) mentioned having spare parts constraints in Mzimba in 2008, but this evaluation did not find such constraints in 2015. Other donors, (DFID, World Bank and Concern Universal) had similar positive experiences about the availability of spare parts in their projects15. The ready availability of spare parts is a result of the policy of the GoM to use the same hand pump (Afridev). Harvey (2011) estimated that, for a sufficient turnover of spares and a sufficient profit for the retailer, approximately 200 hand pumps within a 20km radius are required. This is probably not yet reached in most of the districts in Malawi, but the combined efforts of donors are a step in the right direction. In virtually none of the WASH projects in rural Africa are the beneficiaries able to finance major repairs and replacement of hand pumps. Hence the budgeting for the regular replacement of hand pumps nearing the end of their service life is a point of discussion among development partners. The programme did not take the sustainability of water resources into account and this evaluation did not assess the long-term hydro-geological trends in Malawi. But it is likely that the continuation of the deforestation process and population growth could result in declining water tables. Also, the influence of climate change might play a role in the local groundwater situation. But is not yet known how, and to what degree these effects could contribute to the drying up of wells.

13 Several AMs mentioned that MK20,000 Kwacha (about US$25) was the maximum amount villages could afford. 14 QA report (UNICEF 2011) mentioned the existence of incompatible parts of the Afridev hand pump 15 Mentioned during interviews

49

The involvement of different groups of people -from government officials and community leaders to marginalised groups such as women and children - who voiced their needs in areas such as the siting of boreholes implies that the WASH programme was socially inclusive, thereby enhancing sustainability through ownership building. Indications that toilets were appropriate for both children and disabled people suggest social sustainability. More importantly, the continued use of buckets by schools and village people as the most common basic hand washing facility demonstrates that the practice of hand washing has been socially accepted by some wards in the TAs, despite the irregular use of soap and absence of hand washing facilities in some places.

The presence of Open Defecation Free (ODF) villages shows that the sanitation activities have had some social impact and can be sustained because a change of social norms has occurred at village level. For example, in Lilongwe, through the Community Led Total Sanitation (CLTS) approach, every household in Nkhuku village had a toilet and a waste disposal pit. It was clear from the community members that open defecation and failure to construct a toilet by any household are socially unacceptable. However, this CLTS intervention should have come from the communities themselves rather than being externally driven. Literature review yields that the sustainability of the CLTS approach is all but ensured (PLAN 2013). In addition, the willingness of community members to contribute to and pay for maintenance of water points at village level shows that WASH projects were being implemented in a socially sustainable manner.

4.5.5 What is the extent of programme influence on national level commitment for resource allocations and leveraging

The programme approach of implementing interventions through the districts strengthened capacity and ownership at district level (UNICEF, 2011a). This evaluation study agrees with the conclusions of the MTR on this. however, despite the good intentions of the programme, institutional sustainability appears to be the least successful component. External follow-up and continued training are a (highly) critical component in ensuring long-term sustainability. VHWCs start enthusiastically, but after some years key persons become less motivated, leave the village or die. The same applies for the area mechanics for whom repairing water points is not the main source of income. Moreover, persons with some technical skills could probably find better paying jobs without much difficulty. The districts are supposed to monitor the water points and provide assistance with training and encouraging VHWS and area mechanics. Unfortunately the districts have hardly any capacity to carry out these tasks. In the districts that were visited by the team, UNICEF provided cars and motorcycles but they were not functioning any more. According to the district coordinator in Mangochi, this is because decentralisation is not yet complete: Vehicle maintenance appears to be a task of the MoIWD, and they do not carry it out. The final Sector Performance Report of 2012 (MoIWD) mentions the low priority given by the GoM to the water sector. According to this report, 4.6% of the national budget is allocated to the water sector and at district level the water sector gets 0.34% from the Government of Malawi’s recurrent budget. This has resulted in a vacancy rate in the water sector across district and national levels of 67.9% for professional established positions. This is visible in the districts, especially in Mzimba, one of the largest districts of Malawi. Ten TAs had only 3 water monitors between them to do all the monitoring and support to the villages. The

50

district received MK200,000 (US$355) a month for the District Water Office. The district of Lilongwe did not have a special budget for the District Water Office.

4.5.6 To what extent are results achieved sustainable at their respective levels (National, sub-national, communities) and can they be replicated or integrated into policies, strategies and programmes?

The national development environment is a favourable factor in ensuring long-term sustainability. All major donors are in line with the present policy on rural water supply. Critical factors such as the creation of ownership, installing VWHCs, decentralisation, the area mechanics system and commercialisation of spare parts are all supported by the main players. Districts mentioned the existence of ‘loose cannons’, such as religious organisations and politicians that install boreholes without following the proper procedures, but this does not appear to be a major problem. It is not clear if the communities are conversant with the National Water Policy. The Sustainability Check (MoIWD, 2012) mentioned that the major lesson learnt by the communities is that if they wait long enough the district will come and fix their water point. However, this is not the experience of this mission, for none of the VHWC members interviewed mentioned anything about the need for district assistance.

4.5.7 What competencies within the WASH section have been useful and what are the key human resource and skills gaps?

All surveyed boreholes were maintained by functioning VHWCs (or were maintained by the school). These VHWCs were able to do maintenance and simple repairs and knew where to find an area mechanic for extensive repairs. About half of the VHWCs had an organized system for collecting maintenance funds, although few members actually knew how much cash they had collected. Some collected money only when there was a need to buy new spare parts or to pay an area mechanic. At the time of the survey 98% of the water points were still functional and the state of maintenance did not deteriorate between 2011 and 2015. These facts illustrate that the beneficiaries are still capable of maintaining their hand pumps. While the communities might still have the technical capacity to keep the hand pump operational in the short term, constraints might arise when the hand pumps approach the end of their service life.16 In consultations the area mechanics estimated that the maximum amount villages could afford is about MK15,000-MK20,000 (US$23-US$31). This amount is sufficient for routine maintenance, but not for major repairs. Hence, with or without a money collecting system it is unlikely that the village can keep the water flowing when hand pumps are nearing the end of their service life. One should note that for many of these villagers MK100 (US$0.18) is already a substantial amount.

4.5.8 What is the likelihood of a continued flow of benefits for beneficiaries after the programme end?

At present the system is still running satisfactorily, but it is not stable in the future. With the ageing of the hand pump, costs will increase to levels which cannot be sustained by the villages. Due to the low involvement by the district, the system of VHWCs and area mechanics might gradually weaken. Decreasing functionality rates are likely in the coming 5-10 years. Also, the sustainability of the CLTS is not guaranteed, according to studies in other African countries (PLAN Netherlands 2013).

16 For an Afridev HP 10-5 years, depending on the number of users and the care they receive.

51

The conclusion of this evaluation, although somewhat optimistic, is in line with the conclusion of the water quality report (UNICEF 2011): "The current review is carried out on water points that average 2 years and do not exceed 5 years in age. All of the above indicate that hand pump functionality is heading for the 80% or even 70% mark by the end of 5-10 years of constant usage by ever-expanding user communities." Field studies on the sustainability of rural water points confirm the complexity of the issue. There are many often interrelated causes for the poor sustainability of the rural water points. Some are critical, while others only have a minor influence on sustainability. Table 7 ranks the most commonly cited factors in sustainability studies in sub-Saharan Africa and shows how this programme scores on the key factors.

Table 7: Most commonly cited factors which influence the sustainability of Hand Pumps and the results of the UNICEF project

Area Factors (*) (**) Technical Maintenance – preventative 2 2

Spare parts availability 2 2 Maintenance – major repairs or replacement 4 3 Standardization of components (especially for hand-pumps) 4 1 Tools and equipment availability (especially for hand-pumps) 4 1

Financial Adequate tariff for recurrent costs 1 2 Adequate tariff for capital replacement or system expansion costs

3 3 Community and Social Community management capacity 2 2

User satisfaction, motivation and willingness to pay 2 2 Involvement of women 3 1 Social capital or cohesion 3 ? Continued training and capacity building 3 4

Institutional and Policy External follow-up support 1 4 Continued training and support to sanitation and hygiene education interventions

2 3 Private sector involvement in goods, services and management contracts

3 1 Legal frameworks for recognition of water committees and

ownership 3 2

Supportive policy and regulatory environment 3 2 Clarity over roles for operation and management 4 2

Environmental Water source production, quality and conservation 2 ?

(Adapted from Lockwood et al., 2003) (*) Rating key: 1 = of highly critical importance; 2 = of critical importance; 3 = of less critical importance; 4 = of limited importance (criteria by Lockwood, 2003) (**) Results of the UNICEF project as judged by the evaluation team on the basis of this evaluation: 1= Good; 2 = Reasonable; 3 = Moderate; 4 = Poor

It is a concern that the UNICEF programme scored poorly on some of the most critical factors in sustainability, especially external follow-up and support, continued training and capacity building. On the other hand, UNICEF scored Reasonable to Good on the issues of preventive maintenance, availability of spare parts, involvement of women and privatization.

5.0 LESSONS LEARNED

52

5.1 Lessons Learned

5.1.1 Organisational and institutional set-up

Whilst the programme was supposed to be under the Ministry of Irrigation and Water Development as custodian of the programme, the evaluation found that UNICEF ran the overall coordination at country level. Some of the Ministries, e.g. Education, Science and Technology and Ministry of Health, were not fully involved in the implementation of the programme. At district level, despite clear guidelines from the MoIWD, committees responsible for the implementation of the programme were composed of different departments depending on the district. Due to the use of government officials and the lack of a Programme Implementation Unit, there were challenges in keeping track of records, especially with high rates of staff turnover and transfers in the government systems. There was a lack of documentation and proper reporting in many districts and this was also a challenge at community and/or village level. Due to challenges in the government structures, beneficiaries in some communities were not able to differentiate between the boreholes of other agencies and those funded by UNICEF. The actual roles of ministries and other stakeholders were not very clear and some contracts were only being provided to NGOs in the course of programme implementation.

5.1.2 Infrastructure and technical sustainability

One of the most important lessons from the past programme is that the capacity of the DWOs to implement the programme and to carry out the necessary quality control is much overrated. Also, the lack of capacity of the DWO to proceed with M&E and to support the VHWCs and the area mechanics after the programme end should have been anticipated. Concerning UNICEF, poor data management is a major concern and should be improved in future programmes Sustainability is one of the most thorny issues in rural water supply in sub-Saharan Africa. Probably all rural WASH programmes struggle to leave a sustainable water supply behind and it is not surprising that also the GoM-UNICEF WASH programme falls short on sustainability. Serious issues needing improvement are the poor M&E and the aftercare capacity of the DWOs, and the insufficient quality control during the construction work. Unfortunately, achieving sustainability is complex and there isn't yet a silver bullet approach. For the moment, UNICEF should work on a number of measures to ensure greater sustainability in the forthcoming programmes. Most of these should be undertaken within a national or even international framework. For Malawi, cooperation among WASH stakeholders (preferable within a SWAp) is suitable. One thing the partners could do is strengthen the capacity of the DWO to provide the necessary aftercare. This should be done on a national scale and in the framework of the (yet to be established) SWAp. Also, discussion on the out-dated design of the Afridev and on the life-cycle approach in rural WASH should be held on a national and international scale. Although there is not yet an example of a working approach, there are many new experimental and promising approaches to sustainability in other African countries. For example, the phone banking approach and experiments with commercial O&M in Uganda might result in greater sustainability. In addition, there are international research institutions that could assist UNICEF. 5.1.3 Capacity building and awareness

The WASH programme interventions require transfer of skills to communities through training and awareness programmes. This is very critical to interventions that are promoting sanitation and

53

hygiene practices. Even though the programme supported some training to district and community members, these were not enough to critically change the mind-set of the beneficiaries. In many cases, the whole programme has been regarded as a UNICEF or government programme. Future programmes should have a well-designed and delivered capacity building component that will engage all key stakeholders throughout the process. In particular, special attention should be given to schools and health centres that are participating in the programme. Special training could be provided to specific students or pupils who could in turn train other children to make sure that sanitation and hygiene practices are supported at community level.

5.1.4 Baseline surveys, data management and financial transactions

Whilst the government has data on WASH related indicators, there was need for the programme to develop its own baseline indicators. Future programmes should put in place strategies that will provide clear baseline information. In addition, there was very poor data and information management in all the districts covered by the survey. A proper data management portal could be developed where community data could feed into a district and national data bank. The financial transaction and release process used by UNICEF has to change to make sure that activities are not delayed.

5.1.5 National coordination and networking in WASH

One critical factor making Malawi a poor country is the poor coordination of major players in critical sectors such as WASH. Major development partners have taken over the mandate of the Government of Malawi, thereby constraining the government system. This situation is made even worse considering the number of NGOs taking over the government’s role. There is need for the GoM to design a very clear framework to be followed by development partners in implementing WASH programmes. Some of the organisations championing WASH interventions should be inspected and be given the required support to address any problems. 5.2 Challenges

Weak coordination by key Government ministries: For effective delivery and ownership of similar programmes in future, there is need for one government ministry to own the interventions. At the time of the evaluation, various NGOs and agencies besides UNICEF were supporting WASH projects, with weak coordination by the government. This tended to constrain the district staff and in many cases led to a duplication of effort.

Delayed disbursement of funds: The delay in transferring funds to districts jeopardised the smooth implementation of activities. In some cases, funds were released 6 months after the plans had been approved.

Stability of human resources: Lack of stable staffing affected knowledge retention at district and community levels. Those that had been trained in specific areas were no longer at their duty station. Indeed, at community level some of the founding VHWC members were no longer holding office. It was noted, incidentally, that those holding positions in one committee were also on other committees.

Dependency syndrome: The approach taken by many development partners has resulted in communities depending on donors for each and every development effort. The self-help spirit has been eroded and this is affecting ownership of interventions. Due to a lack of

54

proper engagement guidelines or consultations during the design phase, most beneficiaries were asking government to provide for their basic needs.

6.0 CONCLUSION AND RECOMMENDATIONS

6.1 Conclusion In this section we provide conclusions based on the findings of the evaluation discussed above.

6.1.1 Relevance

The programme has been consistent with both national and global goals that aim to reduce the child mortality rate, promote education among girls, and provide safe water to rural communities. The programme also facilitated the decentralization process whereby district structures were strengthened. The programme had the right beneficiaries in that it targeted communities with high poverty and illiteracy levels. However, the programme failed to put in place the right mechanism for identifying beneficiaries. A specific needs or situation assessment that will provide criteria for identification of beneficiaries will be required for the new programme. Provision of safe water was and still is relevant in the development framework of the country, which considers the natural and human related challenges that may affect access to safe drinking water. Overall, the implementation of the programme activities supported the attainment of programme objectives. Strengths 1. The programme was fully consistent with the UNICEF goals and the ODF strategy. The

implementation of the programme activities through the district was consistent with the national policy on decentralization and with the Paris Declaration on Aid Effectiveness.

2. The programme was highly relevant in terms of poverty and sanitation, and it was reasonably

relevant in terms of water coverage. 3. The programme supported the attainment of both national and global goals to promote the

education of the girl child, to reduce the child mortality rate and to provide safe water to communities.

4. The interventions targeted the right beneficiaries, especially looking at the levels of poverty and

illiteracy in all the target areas. The inclusion of women at community level allowed them to have more time for other important household activities.

Weaknesses 1. The quality of the logical framework was only moderate. It lacked a hierarchic logic, and had a

confusing mix of purpose, results, targets and indicators. In addition, the outcome indicators were difficult to quantify.

2. It was not clear if the programme targeted the most vulnerable villages because selection was

demand based and not based on a needs assessment study.

55

3. Coordination between the development partners and key Government ministries appeared to

be weak. For effective delivery and ownership of similar programmes, there is need for a single government ministry to own the interventions. At the time of evaluation, other NGOs and agencies were also supporting WASH projects, with weak coordination by the government.

Conclusion for Relevance In general, it can be concluded that the programme attained its objectives in relevance because it allowed beneficiaries to access safe water, it provided sanitation infrastructure and supported national practices at community level. The evaluation found that all the communities supported by the programme are still accessing safe water and the water point infrastructure is still functioning. The provision of separate toilets for girls in schools has reduced absenteeism. Finally, a strong and effective coordination framework at national level among partners would allow the government to meet its targets at national level. The performance rate for relevance is satisfactory.

6.1.2 Efficiency

The programme has contributed to national results, especially on the provision of safe water, which is directly linked to reducing child mortality and the burden on women and girls. It has also resulted in the reduction of open defecation. The programme has improved sanitation in several primary schools and reduced the number of learners per water point. The change in the implementation of the programme from PHAST to CLTS affected the pace of implementation, but all the targets have been achieved. The process of borehole tendering could be reconsidered, in order to develop the capacity of district teams to handle contractual issues. Overall, the quality of water points is good and the building of community capacity has helped to sustain the boreholes. Strengths 1. Despite initial delays, the speed of the programme implementation was sufficient to reach its

targets within the programme budget.

2. The inclusion of schools and health centres in the programme allowed it to address key development challenges in education and health. In addition, the approach to establish the DCT and use the decentralization process meant that government capacity was not only built at national level, it also allowed teams to share lessons and do joint planning at district level.

Weaknesses 1. Although implementation through the districts added to the relevance of the programme, it

had serious consequences for its efficiency. Construction quality, siting of boreholes and training of beneficiaries were compromised by both the poor supervision by the districts and the inferior drilling contract standards accepted by the MoIWD and the districts.

2. Communication between UNICEF and the districts left room for improvement, especially on

financial issues. This caused long delays in payments and hampered the implementation by the district.

3. Data management and reporting left much to be desired. The M&E in the districts was poor or

non-existent. There was hardly any data on either the WASH infrastructure or the progress of the CLTS approach in the villages. The narrative parts of the progress reports were informative

56

and clear, but the reporting of data left room for improvement. The indicators in progress reports were mainly outcomes, which were difficult to verify because the relationship between outputs and outcomes was unclear.

4. The approach of having three different mechanisms for engaging drilling contractors (UNICEF,

Government and districts) resulted in several challenges and differences in the quality of work. 5. There are indications that UNICEF costs were somewhat higher than those of its partners per

borehole, but this could not be proven because financial data were hard to find.

Conclusion for Efficiency UNICEF focused too much on achieving the physical targets instead of concentrating on supervision and on strengthening the capacity of the implementation partners. The links between the district and community implementers were very weak, and feedback mechanisms to the headquarters were inadequate. These challenges were reflected in the coordination between UNICEF and the key ministries. The performance rate for efficiency is less than satisfactory.

6.1.3 Effectiveness

It can be concluded that the programme managed to achieve all the planned activities and it was effective in the delivery of safe water to the communities and schools. The challenges faced by communities in accessing water are caused by population increase as well environmental challenges or the poor siting of boreholes. The strategies to work through communities, install Afridev hand pumps, support area mechanics and establish VHWCs have all been significant in the effectiveness of the programme. The provision of separate toilets for girls in schools has reduced absenteeism. Strengths 1. The programme was effective because it reached and even surpassed its targets. The majority

of the beneficiaries now have water of improved quality closer to their homes. Women take less time to access water, thus addressing issues of insecurity associated with going to fetch water. The provision of separate toilets for girls in schools has reduced absenteeism.

2. The project certainly had a positive influence on the national average for safe WASH in schools

and villages, but it was not possible to quantify the effects because data was not available. 3. The programme was flexible and made proper decisions to deviate from the original proposal

and focus on the CLTS approach, to cancel the establishment of San Marts and to abandon the watershed management component.

Weaknesses 1. UNICEF focused too much on physical targets, e.g. the number of installations constructed, and

made short cuts on important issues such as quality control, and the social aspects of sustainability and supervision (especially the M&E capacity) in the districts.

57

2. The ratio between the number of persons and water points does not always follow the standards for Malawi (250 persons/water point). This could compromise both the effectiveness and the sustainability of the water points in the face of rapid population growth.

3. The programme mainly constructed boreholes equipped with the Afridev hand pump, and

managed by a water point user committee. While this is a commonly used approach, it is out-dated and has many sustainability constraints.

4. The programme invested a great deal in the training of the district authorities, but most

knowledge has been lost because of high staff turnover and a lack of financial support from central government.

5. The capacity building in schools for both teachers and learners was quite weak. This challenge

could be partly attributed to weak engagement on the part of the relevant Ministry. Programme support could be more effective if there was a clear framework to encourage beneficiaries to learn about issues concerning hygiene and sanitation.

6. The lack of clear interventions to promote hygiene practices in schools and at community level

resulted in poor knowledge and skills among the beneficiaries.

Conclusion for Effectiveness This was an effective and flexible programme that reached or even surpassed its targets, but it did not leave strong and effective districts and community structures. In general, the performance rate for effectiveness is satisfactory.

6.1.4 Impact

The evaluation found several impacts attributable to the interventions, including improvements in hygiene and health in schools and villages. These improvements have reduced the incidence of water-borne diseases and the associated costs for health services. The interventions have reduced pressure among some health workers because fewer patients are presenting with sanitation related diseases. HSAs can now undertake other work, therefore the programme has contributed to community development in general. The interventions have also promoted rural education, especially among girls. The impact of hand washing interventions is weak, especially with regard to the availability of soap and other facilities for both schools and households. Strengths 1. The project had a positive impact by considerably reducing women’s workload, since over 50%

of the respondents mentioned a reduction in the distance they have to walk to the water point. The waiting time at the water point was also reduced, but this was partly undone by the effects of population increase. The construction of school latrines reduced the absenteeism of girls in schools.

2. There are indications of a positive impact of the programme on health, but due to a lack of

baseline studies this could not be quantified. However, the relatively good hygiene in the school latrines and yards, combined with the efforts of most schools to ensure that children wash their hands, could lead to improved health among school children. These health benefits extend to the general population as well, since they are less likely to be affected by water-borne or sanitation related diseases or infections.

58

Weaknesses 1. The lack of baseline indicators and critical gaps in continuous data collection resulted in

challenges in providing accurate estimations of the health benefits of the programme. Even though national indicators have been used, these have their own limitations, e.g. the methods for setting them are inconsistent.

2. The efforts to improve hygiene behaviour (hand washing) leave much to be desired. None of

the hand washing structures at the schools are still functional, and the makeshift installations that some schools have made risk being stolen, too. In addition, the hand washing efforts in the villages were not very successful.

Conclusion for Impact The programme has achieved a good impact on water availability, but the impact on health could have been better if the programme had put more emphasis on hygiene promotion. Because of insufficient baseline data it is not possible to establish the performance rate for impact.

6.1.5 Sustainability

The sustainability of this programme lies in infrastructure works and the capacity of districts and communities to maintain the boreholes. It can be concluded that the interventions to empower communities, the training of area mechanics and the use of simple technologies added to sustainability. However, sustainability is very difficult to achieve and more should have been done to leave a sustainable water supply behind. Water quality and quantity may stay good for a year or so, with few challenges with boreholes. Economic issues remain the challenge, since high poverty levels and the ageing of the borehole might have an impact on sustainability. The sustainability of committees at local level also remains doubtful, as most of the members have moved on. Strengths 1. In general, the WSS was running well and nearly all water points were still in operation,

although a sizeable number of hand pumps were in need of maintenance. Specific positive aspects include the standardization of the hand pumps, the well-functioning commercial distribution system for spare parts, the sense of ownership by the beneficiaries, and the harmonization of the donor approaches. The VHWCs knew where to get technical assistance for more advanced repairs and were able to collect funds for maintenance and minor repairs.

2. The hand pump repair system based on the area mechanics is still functional. 3. The hand pump used by the project (the Afridev) is highly suitable for village level operation

and maintenance, and is widely used in the country.

Weaknesses 1. The VHWCs are did not collect sufficient funds for major repairs. They are still vulnerable, and

without periodic support and encouragement they will become less effective in the near future.

59

2. The poor M&E capacity and the district authorities’ failure to provide aftercare to the programme is a critical flaw in the sustainability of the water points.

3. The economic situation of the area mechanics is weak and not very stable, because once

trained mechanics are able to find more profitable employment. 4. The effects of climate change and the deterioration of the watershed on the water table have

not been investigated. This could become an issue needing attention in the future. 5. The hand pump which was used in this programme (the Afridev) is out-dated and vulnerable to

improper maintenance. Conclusion for Sustainability The sustainability approach as used in this programme is often applied in other countries in sub-Saharan Africa, but is rarely successful. Despite the good performance to date, long-term sustainability is not yet ensured; hence the performance rate is less than satisfactory.

60

6.2 Recommendations

Relevance 1. The development of a proper theory of change should be given high priority in the new

programme. 2. UNICEF should combine the existing demand-based approach with the identification of priority

areas based on the local WASH situation. For this to done, UNICEF should carry out a short needs assessment study in the target areas for the new programme.

3. Roles and responsibilities should be well defined between UNICEF and the line ministries, as

well as between those at district and community levels. This will require the MoIWD to provide guidelines for the WASH sector, to be adhered to by all relevant stakeholders. The powers of funding agencies, coordinating ministries and the implementing districts should be very clear and distinct.

Efficiency 1. In future UNICEF programmes, more emphasis should be placed on the quality of construction.

UNICEF, together with the local and national governments, should develop strict quality control procedures and the work contracts should at least comply with UNICEF standards. This will require a deliberate effort to build the capacity of district teams and communities to monitor the works.

2. In view of the long delays in transferring the implementation funds to the districts, UNICEF and

the districts should agree on a more efficient protocol for financial issues and or assist the districts with financial management.

3. M&E and the proper management of data should become a priority of both UNICEF and the

districts in future programmes. UNICEF and the districts should agree on a sustainable M&E system for physical data on water points, data on the performance of the VHWCs and area mechanics, and health data. The districts should be capable of continuing to conduct M&E long after the closure of the programme. Progress should be reported in output indicators rather than in outcome indicators, and the relationship between outputs and outcomes should be clear.

4. In the forthcoming programme the different drilling contracts should be harmonized and the

highest standards adopted. 5. Data on construction costs should be kept readily accessible in UNICEF files until the final

evaluation of the programme.

Effectiveness 1. In future programmes UNICEF/GoM should be less focused on physical targets such as the

number of water points and should prioritize the social aspects of sustainability and supervision (especially the M&E capacity) in the districts.

61

2. In selecting drilling locations, UNICEF and the districts should follow the Malawi (and international) standards of 250 persons/water point. This would improve both effectiveness and sustainability, and will require a general approach to WASH that can be linked to demographics and the issues involved in reducing population growth.

3. UNICEF programme proposals should incorporate the most recent views on WASH

methodology propounded by the leading WASH research institutions. Rural water technology has made great progress in recent years (e.g. solar power, phone banking payment options). Other organizations such as GOAL, Fair Water and IRC have developed innovative approaches in rural water supply. In the forthcoming programme UNICEF should encourage experiments with advanced technical approaches.

4. In the programme planning stage, UNICEF should ensure long-term financial commitment from the government to avoid the loss of trained staff in the districts after the project end.

5. In future UNICEF/GoM programmes, capacity building interventions in schools and

communities should be better planned and executed. This will require the development of a graduate WASH learning programme where university students or ‘WASH champions’ would be able to deliver important packages/products and modules to schools and communities.

6. UNICEF-/GoM should initiate a deliberate effort to design a WASH Learning Forum where pupils and teachers will be able to learn more about WASH issues. Partnerships with centres of higher education could facilitate this.

Impact 1. In order to quantify the impact of a WASH programme in the districts on both the health and

the economic situation of the beneficiaries a baseline survey should be carried in the proposed programme and comparison locations.

2. Hand washing in schools and in villages should be given the highest priority in rural WASH

projects. In particular, the design of the hardware should be appropriate to the needs of school children.

Sustainability 1. UNICEF should continue to focus on the stability and capacity of the VHWC to collect funds to

ensure long-term maintenance. New payment options, for example, phone banking, should be piloted. In addition, an aftercare approach should be developed.

2. UNICEF and its development partners should enhance the capacities of the districts to organize

M&E during rural WASH projects and aftercare upon their completion. A sustainability and exit/aftercare strategy should be negotiated with the government and development partners (preferably within the framework of a SWAp). Without such a strategy, long-term sustainability will not be ensured.

3. UNICEF and development partners should develop initiatives to improve the economic situation

of the area mechanics. This can be done by improving their entrepreneurial skills and including them in the supply chain of hand pumps and spare parts.

62

4. For long-term sustainability, there is need for stakeholders to carry out a joint assessment of the vulnerability of the groundwater resources in Malawi.

5. A deliberate effort should be made to initiate the development of low cost and maintenance-

free hand pumps to replace the obsolete Afridev and Indian Mark II/III hand pumps. More generally, the lack of sustainability is a common problem for all major WASH donors in Malawi and worldwide. Hence, UNICEF and development partners should attempt to develop sustainable approaches on WASH in Malawi. Unorthodox solutions such as the life cycle approach in rural water supply should be seriously considered. International knowledge institutions such as IRC could provide valuable assistance here.

63

BIBLIOGRAPHY

AfDB (2009) Malawi Integrated Rural Water Supply and Sanitation Project for Ntchisi and Mzimba. Final report. African Development Bank, Lilongwe, Malawi. Concern Universal (2011) Post Project Evaluation of Ntcheu and Thyolo Water Supplies and Sanitation Projects. Concern Universal, Lilongwe, Malawi. Goulet, L. and Rodie, J. (2013) Evaluation of the WASH project in the Integrated Rural Village of Nkurye and Murembera in the Commune of Giharo in the province of Rutana, Burundi. UNICEF, Burundi. Government of Malawi (2012) The Sustainability Check report. Ministry of Irrigation and Water Development, Lilongwe, Malawi. Government of Malawi (2012) Sector Performance Report 2011/12. Ministry of Irrigation and Water Development, Lilongwe, Malawi. Government of Malawi (2011) Open Defecation Free (ODF) Malawi Strategy 2011-2015, MoIWD, Lilongwe, Malawi. Government of Malawi (2008) National Population and Census Report. National Statistical Office, Zomba, Malawi. Government of Malawi (2008) National Sanitation Policy. Ministry of Irrigation and Water Development, Lilongwe, Malawi. Government of Malawi (2008) Water and Sanitation Sector Joint Sector Review Report. Ministry of Irrigation and Water Development, Lilongwe, Malawi. Government of Malawi (2005). National Water Policy. Ministry of Irrigation and Water Development, Lilongwe, Malawi. Government of Malawi (2004). Malawi Demographic and Health Survey. National Statistics Office, Zomba, Malawi. Government of Malawi (1995). Water Works Act. Lilongwe, Malawi. Government of Malawi (1969). Water Resources Act. Lilongwe, Malawi. Harvey, P.A (2011) Sustainable Supply Chains for Rural Water Services linking local procurement of hand pumps and spare parts supply. Rural Water Supply Network. Field Note No 2011-1. Lockwood, H., Bakalian, A., Wakeman, W. (2003) Assessing sustainability in rural water supply: the role of follow-up support to communities. Accessed on May 14, 2016 from http://www.aguaconsult.co.uk/uploads/pdfs/WBAssessingSustainability.pdf

64

Paris Declaration on Aid Effectiveness (UN, 2005) and Accra Agenda for Action (2008) Plan Nederland. Mid-Term Review Pan African CLTS program 2012-2013. (PLAN Netherlands 2013).

Rijsdijk, A. (2013) Study and Supply Chain for Operation and Maintenance of Rural Water Supply and Sanitation. Final Report for AfDB. African Development Bank, Tunis, Tunisia. UNICEF (2013) Detailed progress and final report for 2007-2013 and work plan for 2014. UNICEF, Lilongwe, Malawi. UNICEF (2011) Quality Assurance of UNICEF Drilling Programmes for Boreholes in Malawi Report. UNICEF, Lilongwe, Malawi. UNICEF, 2011a Final Report of Mid-Term Review of UNICEF Malawi implemented WASH project. UNICEF, Lilongwe, Malawi (2011) UNICEF (2010) UNICEF-adapted UNEG Evaluation Report Standards. UNICEF, New York, USA. UNICEF (2007) District investment plans of Salima, Lilongwe, Blantyre, Nhakta Bay, Mangochi, Mzimba, Chitipa, Dowa, Kasungu, Likoma, Mchinji and Mwanza. UNICEF, Lilongwe, Malawi. UNICEF (2006) Malawi-WASH Programme Proposal submitted to Dutch Government. UNICEF, Lilongwe, Malawi UNDP (2015) Human Development Report: Work for Development. UNDP, New York, USA. UN‐Water (2008). Status Report on IWRM and Water Efficiency Plans. UN, New York, USA. USAID (2010) Malawi Water and Sanitation Profile. USAID, Washington, USA. WHO/UNICEF (2010). Joint Monitoring Programme for Water Supply and Sanitation (JMP). Progress on Sanitation and Drinking Water. WHO, Geneva, Switzerland. World Bank (2012) The Republic of Malawi Joint IDA-IMF Staff Advisory Note on the Second Malawi Growth and Development Strategy. World Bank, Washington, USA. World Bank (2007) Project appraisal for a second water development project 2007. World Bank, Lilongwe, Malawi.

65

66

ANNEXES

Annex 1: UNICEF-WASH evaluation Matrix

Relevance

Main evaluation questions

Sub questions Indicator Document review National interviews

Districts interviews Communities interviews

UN

ICEF in

tern

al

do

cum

ents

Natio

nal strate

gy

do

cs and

statistics D

istrict

investm

ent p

lan &

m

on

itorin

g repo

rts

Mo

WD

&I

Mo

H

Natio

nal N

GO

's

CP

AR

Private

sector

District Im

pl.

team

s

Scho

ols

Health

cen

tres

Imp

lemen

ting

NG

O's

Private

sector

Ben

eficiares

Area m

echan

ics

Village w

ater

com

ms

Village h

eads

Private

sector

Visu

al insp

ection

s

Assess the internal, vertical and horizontal consistencies

How does the project strategy compare with national Wat/San strategy

Strategies are comparable

X X

How does the project objectives compare with UNICEF goals

Objectives and UNICEF goals are not different

X

Appropriateness of the project objectives in the overall problem context, needs and priorities of the target groups and beneficiaries

Has UNICEF the needs of the target groups properly identified

Needs assessment reports are of good quality

X X X X

Is the lack of water perceived as the most urgent

The main problem is indeed the lack of water

X X X

Stakeholders confirm the relevance

67

Relevance continued

Main evaluation questions

Sub questions Indicator Document review National interviews

Districts interviews Communities interviews

UN

ICEF in

tern

al d

ocu

men

ts

Natio

nal strate

gy

do

cs and

statistics D

istrict investm

ent

plan

& m

on

itorin

g rep

orts

Mo

WD

&I

Mo

H

Natio

nal N

GO

's

CP

AR

Private

sector

District Im

pl. team

s

Scho

ols

Health

cen

tres

Imp

lemen

ting

NG

O's

Private

sector

Ben

eficiares

Area m

echan

ics

Village w

ater

com

ms

Village h

eads

Private

sector

Visu

al insp

ection

s

How well the programme has identified the needs of the most deprived populations and how these have been built into programme results and monitoring;

Did UNICEF identified the poorest districts

Country and district statistics

X X

Did UNICEF identify the most vulnerable area's in the districts

Proper needs assessments

X X

Did UNICEF identified the most vulnerable persons in the districts

Proper needs assessments

X X X

How well the programme has identified and addressed the challenges of balancing

Is the logframe coherent?

Logframe check

X

68

Efficiency

Main evaluation questions

Sub questions Indicator Document review National interviews

Districts interviews Communities interviews

UN

ICEF in

tern

al

do

cum

ents

Rep

orts o

f oth

er im

plem

entin

g orgs

Natio

nal strate

gy d

ocs an

d statistics

District in

vestmen

t p

lan &

mo

nito

ring

repo

rts

Mo

WD

&I

Mo

H

Natio

nal N

GO

's

CP

AR

Private

sector

District Im

pl. team

s

Scho

ols

Health

cen

tres

Imp

lemen

ting

NG

O's

Private

sector

Ben

eficiares

Area m

echan

ics

Village w

ater co

mm

s

Village h

eads

Private

sector

Visu

al insp

ection

s

What are the factors affecting the pace and quality of implementation and how can these be mitigated

Did the programme experience serious delays and what did UNICEF to mitigate this

Activities which went (not) according to plan

X X

Where there any significant differences in costs and implementation speed between the districts

Differences in costs / implementation speed between the districts

X X

Did the interventions met the intended results at the least cost

Differences in costs / implementation

X X

How efficient were the procedures and approaches adopted for infrastructural works and were they in conformity with UNICEF’s regulations

Check UNICEF regulations and compare with practises in the district

Districts follow UNICEF procedures

X X X

How was the reporting to the donor

Quality of reporting to the donor

X

69

Effectiveness

Main evaluation questions

Sub questions Indicator Document review National interviews

Districts interviews Communities interviews

UN

ICEF in

ternal

do

cum

ents

Rep

orts o

f oth

er

imp

lemen

ting o

rgs

Natio

nal strate

gy

do

cs and

statistics D

istrict investm

ent

plan

& m

on

itorin

g rep

orts

Mo

WD

&I

Mo

H

Natio

nal N

GO

's

CP

AR

Private

sector

District Im

pl. team

s

Scho

ols

Health

cen

tres

Imp

lemen

ting

NG

O's

Private

sector

Ben

eficiares

Area m

echan

ics

Village w

ater

com

ms

Village h

eads

Private

sector

Visu

al insp

ection

s

To what extent did programme implementation strategies contribute to national results

Compare 2007 JMP data with those of 2013

Differences in national statistics

X X

Compare district data of 2007 with 2013

Differences in district statistics

X X

What actual results were achieved against planned

Compare log frame figures with actual results

See log frame X

What capacities were developed in the sector as a result of the WASH programme

Where the capacity development plans relevant to the needs identified in the districts

Comparison of needs assessments with capacity development plans

X X X

What advocacy was carried out and how effective these

Discuss advocacy with partners

What advocacy was carried out

X X X X X

70

have been;

Impact

Main evaluation questions

Sub questions Indicator Document review National interviews

Districts interviews Communities interviews

UN

ICEF in

tern

al d

ocu

men

ts

Rep

orts o

f oth

er

imp

lemen

ting

orgs

Natio

nal strate

gy

do

cs and

statistics D

istrict in

vestmen

t plan

&

mo

nito

ring rep

orts

Mo

WD

&I

Mo

H

Natio

nal N

GO

's

CP

AR

Private

sector

District Im

pl.

team

s

Scho

ols

Health

cen

tres

Imp

lemen

ting

NG

O's

Private

sector

Ben

eficiares

Area m

echan

ics

Village w

ater

com

ms

Village h

eads

Private

sector

Visu

al insp

ection

s

What impact did it achieve for against its planned targets

Is there change in WASH related diseases

Data in health statistics X X X X X

What is the current state of hygiene in HH and institutions

State of maintenance and hygienic condition

X X X

Change in school attendance of girls

Attendance of girls to schools before and after the project

X X

Did the project have positive influence on gender

Involvement of women in the decision process

X X X

Impact on work load of women and children?

X X

Other unintended positive or negative impacts of the program

71

Sustainability

Main evaluation questions

Sub questions Indicator Document review National interviews Districts interviews Communities interviews

UN

ICEF in

ternal

do

cum

ents

Rep

orts o

f oth

er

imp

lemen

ting

orgs

Natio

nal strate

gy

do

cs and

statistics D

istrict

investm

ent p

lan &

m

on

itorin

g repo

rts

Mo

WD

&I

Mo

H

Natio

nal N

GO

's

CP

AR

Private

sector

District Im

pl.

team

s

Scho

ols

Health

cen

tres

Imp

lemen

ting

NG

O's

Private

sector

Ben

eficiares

Area m

echan

ics

Village w

ater

com

ms

Village h

eads

Private

sector

Visu

al insp

ection

s

What strategies for sustainability have been adopted and how effective these have been

Compare the UNICEF approach on sustainability with other approaches (2007 & 2015

Changes in approaches

X X

Did the project take external sustainability factors into account (for example availability of (future) water resources)

UNICEF approach includes all aspects of sustainability

X X

What competencies within the WASH section have been useful and what are the key human resource and skills gaps;

Is UNICEF staff up to date with the latest sustainability approaches in rural WASH?

The new strategy is up to date on sustainability

X

Sustainability continued

Main Sub questions Indicator Document review National Districts interviews Communities

72

evaluation questions

interviews interviews

UN

ICEF in

tern

al d

ocu

men

ts

Rep

orts o

f oth

er

imp

lemen

ting o

rgs

Natio

nal strate

gy

do

cs and

statistics D

istrict in

vestmen

t plan

&

mo

nito

ring rep

orts

Mo

WD

&I

Mo

H

Natio

nal N

GO

's C

PA

R

Private

sector

District Im

pl.

team

s

Scho

ols

Health

cen

tres

Imp

lemen

ting

NG

O's

Private

sector

Ben

eficiares

Area m

echan

ics

Village w

ater

com

ms

Village h

eads

Private

sector

Visu

al insp

ection

s

What is the likelihood of continued flow of benefits for beneficiaries after the project end

What is the present condition of the existing water points

Technical condition X X X X

What are the present (managerial / financial) capacities of the Village health and water committees?

Number of active members, condition of admin / financial situation / gender

X X X X X X

What is the average down time of a water point

Days before repair and % of defect pumps

X X X X X

Are the area mechanics still actively engaged with the maintenance of the water points?

Satisfaction with his job/ commercial interest / future plans. How many are still in function?

X X

Sustainability continued

Main evaluation

Sub questions Indicator Document review National interviews

Districts interviews Communities interviews

73

questions

UN

ICEF in

tern

al

do

cum

ents

Rep

orts o

f oth

er

imp

lemen

ting

orgs

Natio

nal strate

gy

do

cs and

statistics D

istrict in

vestmen

t plan

&

mo

nito

ring

repo

rts

Mo

WD

&I

Mo

H

Natio

nal N

GO

's C

PA

R

Private

sector

District Im

pl.

team

s

Scho

ols

Health

cen

tres

Imp

lemen

ting

NG

O's

Private

sector

Ben

eficiares

Area m

echan

ics

Village w

ater

com

ms

Village h

eads

Private

sector

Visu

al insp

ection

s

What is the likelihood of continued flow of benefits for beneficiaries after the project end (continued)

Is the supply chain of spare parts still functioning?

Availability of spare parts at distribution centres

X X X X

Is the district still actively monitoring the water / sanitation / health situation

Up to date monitoring reports

X

Condition and hygiene of latrines (all places)

Condition and hygiene of latrines (all places)

X X X X X X

Condition of hand washing facilities at all places

Condition of hand washing facilities at all places

X X X X X X

School WASH clubs still active?

Clubs still meet and have activities

X X

Village tidy and still ODF?

Village tidy and still ODF?

X X

Sustainability continued

Main evaluation questions

Sub questions Indicator Document review National interviews

Districts interviews Communities interviews

74

UN

ICEF in

tern

al d

ocu

men

ts

Rep

orts o

f oth

er

imp

lemen

ting

orgs

Natio

nal strate

gy

do

cs and

statistics D

istrict

investm

ent p

lan &

m

on

itorin

g repo

rts

Mo

WD

&I

Mo

H

Natio

nal N

GO

's

CP

AR

Private

sector

District Im

pl.

team

s

Scho

ols

Health

cen

tres

Imp

lemen

ting

NG

O's

Private

sector

Ben

eficiares

Area m

echan

ics

Village w

ater

com

ms

Village h

eads

Private

sector

Visu

al insp

ection

s

What is the extend of programme influence on national level commitment for resource allocations and leveraging

Is rural WASH still a high priority for the Malawi government?

Budget on rural WASH did not decrease

X X X X X X

To what extent results achieved are sustainable at their respective levels (National, sub-national, communities) and if they can be replicated or integrated into policies, strategies and programmes

Is the sustainability strategy of the project still valid or is revision required

Comparison between existing strategy and latest views on sustainability

X X X

What is the interagency cooperation

State of the Cooperation with Developm. Partners

X X X X

What community capacities and competencies were developed and how did these contribute to sustainability and resilience

Do the beneficiaries and village committees still have knowledge of the training

Do the HH still remember the most important aspects of training and do the use these?

X X X

Sustainability Continued

Main Sub questions Indicator Document review National interviews Districts interviews Communities interviews

75

evaluation questions

UN

ICEF in

tern

al

do

cum

ents

Rep

orts o

f oth

er

imp

lemen

ting

orgs

Natio

nal strate

gy

do

cs and

statistics D

istrict

investm

ent p

lan &

m

on

itorin

g repo

rts

Mo

WD

&I

Mo

H

Natio

nal N

GO

's

CP

AR

Private

sector

District Im

pl.

team

s

Scho

ols

Health

cen

tres

Imp

lemen

ting

NG

O's

Private

sector

Ben

eficiares

Area m

echan

ics

Village w

ater

com

ms

Village h

eads

Private

sector

Visu

al insp

ection

s

What community capacities and competencies were developed

How is the gender balance of the village / district committees

M/F ratio of the committees

X X X X

76

Annex 2: Household questionnaire

Questionnaire for Households (FINAL OCTOBER 22, 2015)

This questionnaire basically serves to get an independent programme review of the UNICEF- WASH Programme; at the household level. The questions target the project beneficiaries to understand the impact of the programme since 2007. The information will be collected on the following thematic areas: (i) Water Supply (ii) community sanitation (iv) Hygiene promotion (vi) Capacity building (vi) awareness on hygiene promotion and (vii) health.

MODULE 100: DEMOGRAPHIC DATA Date |____________| 101 Enumerator Code |_______________| 102 Household Serial Number |____|(Unicef 1 will use 1-10, Unicef 2, 11-20 and so on) 103 District Mzimba |__1__| Lilongwe |__2__|Mangochi |__3__| 104 Traditional Authority |____________________________________| 105 Group Village Headman (A Gulupu anu ndani?) |________________________________| 106 Sex of respondent |____| 1= Male; 2 = female 107 Age (Kodi mulindizaka zakubadwa zingati? |____| Years 108 Period of residence in the area (Kodi mwakhala nthawi yaitali bwanji mmudzi muno?)|____| Years 109 Education level |____| 1 = Primary, 2 = Secondary, 3 = Vocational 4= University/ College 5= None

(Kodi sukulu munafika nayo patali bwanji?)

110 Marital status |____| 1 = Single, 2 = Married, 3 = Widowed, 4= Separated (Kodi mulu pa banja? Ngati ayi,funsani ngati anakwatiwapo. 111 Type of household (Kodi mutu wa banja lino ndi ndani?)|____|1=female headed, 2= male

headed; 3=child headed 112 Are you the head of household |____| 1=yes 2=no 113 If not, what is your relationship with the head of household? |____| 1=spouse 2=brother/sister

3=uncle, 4=aunt 5=grandparent 6=other (specify) (Kodi ubale wanu ndiotani ndi mutu wanyumbayi?)

77

114 Household Size (Kodi pakhomo pano mumakhala angati?) |____| 115 Main source of income business (Kodi ndinjila iti yomwe mumadalila popezela ndalama?)

|____|1 = Farming, 2 = Informal Business, 3= Formal business,4 = Formal employment, 5= Informal employment 6= Other (specify)_________________________

78

MODULE 200: WATER SUPPLY 201 What is the household’s main source of water for the following? (Kodi madzi mumatunga kuti?) Drinking Handwashing Cooking Borehole |_1_| |_1_| |_1_| Shallow well |_2 | | 2_ | | 2_ | Piped water in building |__3| | 3_| |_3_| Public stand pipe |__4| |_4_| |_4_| Unprotected surface water (river, dam, etc.) |_5_| |_5_| |_5_| No water available within the village |_6_| . |_6_| |_6_| Other (specify) ____________________________________________________________ 202 Has there been a period of non-availability of water from the main source? (Kodi munakhalako munyengo yosowekela madzi kumalo komwe mumatunga madzi kawirikawiri?)

Yes |_1__| CONTINUE No |_2__| SKIP TO Q207

203 If yes,for how long? (Kwanthawi yaitali bwanji?) One day |_ 1 _|

One week |_ 2 _| One month |_ 3_ | More than One month |_ 4 _|

204 What are the reasons? (Kodi vuto lakusowa kwamadzi linabwela bwanji?)

Damaged water sources |_1__| Drying of water sources |_2__| Poor design |_3__| Theft of parts |_4__| Non-availability of maintenance parts |_5__| Lack of skilled workforce |_6__|

Other (specify) |_7__| 205 If it was damaged, how long did it take to repair the water facility? (Kodi kunatenga nthawi yaitali bwanji kuti vuto likonzedwe?)

Immediately after it was reported |_1_| 1-2 days |_2_| 3-7 days |_3_| 7-14 days |_4_| More than 14 days ______________ |_5_| Not yet fixed |_6_|

206 If the water point was not working, what was the alternative source of water?

(Munthawi imene madzi analikusowa,madzi mumawatunga kuti?)

Immediately after it was reported |_1_| Alternative borehole |_1_| Unprotected wells |_2_| Other (specify)__________________ |_3_|

207 What is the yield of water during dry season?

Kodi madzi amakhala ochulika bwanji mu nthawi ya chilimwe pa mjigo wanu?

79

Same yield |_1_| Lower yield |_2_| No yield |_3_|

208 Do you pay for the water use? (Kodi mumalipila ndalama iliyonse pa madzi omwe mumagwilitsa ntchito?) Yes |_1__| CONTINUE No |__2_| GO TO Q210 209 How much do you pay for water per month? (Kodi mumalipila ndalama zingati pa mwezi?)

Free water |_1__| Less than K1000 |_2__| K1000-K5000 |_3__| More than K5000 |_4_ |

210 Do you contribute money towards maintenance of your main water point? (Kodi mumasonkha ndalama zokonzetsera malo omwe mumatunga madzi?) Yes |_1__| CONTINUE No |_2__| GO TO Q212 211 How do you make your contributions towards maintenance of your main water point?

Maperekedwe a ndalama pofuna kukonzetsera malo omwe mumatunga madzi amakhala otani? Monthly |_1__| Yearly |_2__| Only when it is broken |_3__|

212 Since 2007,has the distance between your home and the water point you use changed? (Kodi pali kusintha kwa mtunda umene mumayenda kukatunga madzi kuchokela chaka cha 2007?)

It is closer |_1__| No change |_2__| Farther Away |_3__| Not sure |_4__|

213 Since 2007, how has the water quality changed? Chiyambireni mchaka cha 2007, kodi kolity ya madzi yasintha motani?

Improved |__1_| It has remained the same |__2_| It has worsened |__3_|

214 What do you think of the performance of the water point management committee? Kodi comiti yomwe imayanganila malowa mukuiona bwanji?

There is no committee 1

They do a bad job 2

Neither bad not good 3

They do a good job 4

80

215

What is the most important reason for your assessment? Fotokozani chifukwa mwayakhula motelo? ............................................................................................................................................................................... ..............................................................................................................................................................................

216 Has the water point reduced your time to access water? Kodi nthawi imene mumatenga kuti mupeze madzi, yasintha ndikubwera kwa chitukukochi?

No 1

Partially 2

Neither 3

Mostly 4

217 What is the taste of water? Kodi madzi apamalo pano amakoma /amamveka bwanji mukawamwa?

Good 1

Salty 2

Smell 3

Oily 4

Other (Specify)__________

218 How often is this water point functional? Kodi mipopi/mijigoyi imagwila nchtito nthawi zonse?

Occasionally 1

Often 2

Always 3

Other (Specify)__________ 4

219 What was your role or contribution during the installation of the water point? Kodi munatenga gawo lanji pothandizila kukhazikitsa chitukukochi?

Labour contribution 1

Cash contribution 2

Material contribution 3

No payment 4

Meal contribution 5

220 Is water treated in any way before drinking? (Kodi madzi amene mumamwa amatetezezedwa munjila iliyonse?

Always treated |_1_|……….. CONTINUE Sometimes treated |_2_|……….. CONTINUE No treatment |_3_|……….. SKIP TO 224.

221 If chlorine/ water guard was used who supplied the chlorine? (Kodi chlorineyo/ water guard amapeleka ndi ndani?)

Government |_ 1 _| WASH Programme |__2_ | Purchased by committee |__3 _| Health Extension Officers |__4 _| Other (specify) ______________________________

222 How do you ensure that water is not contaminated after treating? Kodi mumapanga chani kuti madzi akhalabe otetezedwa/ osamalika nthawi zonse?

81

Re-chlorination |_1_| Covering the container |_2_| Other (Specify) |_3_|

222 How frequently do you visit your main water point? Kodi mumabwela pafupipafupi bwanji kumalo omwe mumatunga madzi kawirikawiri?

....... times per day

Less frequently or irregularly

223 How many buckets do you normally extract from the main water point per day? Kodi mumatunga zidebe zamadzi zingati patsiku pa mjigowu?

....... ........... Buckets/day

................. MK/ bucket

224

How much do you pay for the water? Kodi mumalipila ndalama zingati kuti mutunge madzi?

................. MK/ week

………….MK /Month

………… don’t pay

225 Is the water price affordable for you? Kodi mitengo yamadziyi mukuiona bwanji?

Yes 1

No 2

226

Apart from yourself, who else of the household normally comes to fetch water? (You may tick more than one answer) Kupatula inu,ndindani wina wapakhomo panu amabwela kudazatunga madzi pano?

Male adults (18 or older) 1

Female adults 2

Boys 3

Girls 4

227

What means of transport do you normally use to get to the water point? (Tick only one answer) Mumayenda bwanji pokatunga madzi?

Walking 1

Bicycle 2

Cart 3

Other 4

228

How do you normally carry the water? (Tick only one answer) Nanga manyamulidwe a madzi amakhala bwanji?

By hands 1

On the head 2

On a wheelbarrow 3

On a cart 4

Bicycle 5

228

How long does the roundtrip from home take when fetching water?) Kodi mumatenga nthawi yaitali bwanji potita ndikubwela kumadzi?

Less than 30 minutes 1

More than 30 minutes 2

Don’t know 3

229 Did you already have access to an improved source of water in 2007? Kodi munali ndi madzi a unkhondo mu chaka chan 2007?

Yes 1

No 2

230

Since 2007, has the distance to your water point changed Kodi kuchokela chaka cha 2007 pali kusintha kwa mtunda okatunga madzi?

It is closer 1

Same distance 2

Farther away 3

Since 2007, has the quality of the water at your water point It improved 1

82

231 changed? Kodi kuchokela chaka cha 2007,ukhondo wamadzi wasintha pa malo pano?

It remained the same 2

It worsened 3

232

Since 2007, has the time you spend in the queue changed? Kodi chichokeleni chaka cha 2007 pali kusintha kwinakulikonse pa nthawi yomwe mumakhala panzele kuti mutunge madzi?

It has diminished 1

No change 2

It has increased 3

233

Has the water point reduced your time to access water? Kodi nthawi imene mumatenga kuti mupeze madzi, yasintha ndikubwera kwa chitukukochi?

No change 1

Partially 2

Neither 3

Significantly 4

234 How best can this water point be improved to meet your needs? Kodi ndi zinthu ziti zimene mungafune kuti zikonzedwe pamalo amene mumapezela madzi kuti zikwanilitse zofuna zanu? ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

MODULE 300: HYGIENE AND SANITATION 301 Do you have any of the following functional facilities? (Kodi ndiziti mwazinthu izi zimene munalinazo?

Toilets |_1__| Rubbish pit |_2__| Cloth drying line |_3__| Waste bin |_4__| Bathroom |_5__| Utensil drying rack |_6__|

302 Is the toilet accessible to children or people with disabilities? (Kodi chimbuzicho ndi chakuti ana komanso olumala angathe kugwilitsa ncthito?)

Yes, |_1__| No |__2_|

303 How do you dispose faecal matter for children? (Assess if the family has children) kodi mumataya kuti mabi wa ana? Toilet |_1__| Bush |_2__| Rubbish pit |_3__| No children |_4__| Other (specify)_____________________ |_5__| 304 What do you use for hand-washing?

83

Mumagwiritsa ntchito chani posamba mmanja mukachoka ku chimbuzi? Water only |_1__| Soap |_2__| Ash |_3__| Ash and soap |_4__| Nothing |_5__| Sometimes use soap sometimes not |_6__| Other (specify) |_7__| 305 How frequent do you use the handwashing facilities? Ndi mowilikiza bwanji pamene mumasamba mmanja?

Always |_1__| Sometimes |_2__| Never |_3__|

306 How did you come to know about hand washing with soap? (ASK IF WASH HANDS WITH SOAP) Kodi zosamba m’manja ndi sopo munazimvera kuti? WASH training sessions |_1__| Through the radio |_2__| Friends |_3__|

Health Extension Workers |_4 _| Other (Specify) |_5__| 307 At what times do you wash hands? Ndi nthawi ziti zomwe mumasambira m’manja? Before and after eating |_1__|

After using toilet |_2__| Others |_3__|

308 In the last few years, which of the following hygiene interventions have you been exposed to?? Mu zaka zingapo zapitazi ndi njira ziti za ukhondo zotsatirazi zomwe mwazidziwa? Training |_ 1 _|

Awareness campaign |__2_| Advocacy and lobbying |__3_| Attending meeting |__4_| Leading materials on hygiene |__5_| Heard message on hygiene on radio |__6_|

None |_ 7 _|

MODULE 400: HEALTH 401 Which of the following water borne related diseases have you suffered from in the past 2 weeks?

309 During 2014/2015 were you visited by and received any WASH extension message from the following? Mçhaka cha 2014/15 ndi mabungwe ati mwa otsatirawa amene akuyendelani ndikulandila uphungu wa za madzi ndi ukhondo? Officials from government |_1_| Officials from NGOs |_2_|Health Surveillance Assistant |_3_| Local leaders |_4__| UNICEF |_5__| None |_6__|

84

Kodi ndi matenda ati otsatirawa obwera chifukwa cha madzi omwe mwadwala pa masabata awiri apitawa?

Diarrhoea |_1_| Dysentery |_2_| Cholera |_3_| Bilharzia |_4_| Malaria |_5_| None |_6_| Other (specify) |_7_|

402 Has the incidence of diarrhoea in your household changed compared to 2007? Pali kusintha kwinakulikonse kwa ka bweledwe kamatenda otsekula mmimba chichokelani chaka cha 2007?

It has diminished 1

No change 2

It has increased 3

403 Over the last two weeks, how frequently have you visited the health center in this area? Kodi pama sabata awiri apitawa mwapita kangati ku chipatala cha m’dera lanu lino?

1 to 2 times 1

3 to 4 times 2

5 to 6 times 3

More than 7 times 4

404 Do you pay for the services? Kodi mumalipila mukalandila chithandizo?

Yes 1

No (Go to Question 408)

2

405 If yes, How much on average do you pay per visit? Kodi mumalipila ndalama zingati mukalandila chithandizocho?

........... MK/visit

406 In a month , how much do you spend on health related problems? Kodi pamwezi mumagwilitsa ntchito ndalama zingati pamavuto okhuzana ndi zaumoyo?

........... Mk/month.

407 Since 2007, have the costs of health services changes? Kodi chichokeleni chaka cha 2007 pali kusintha kwotani pa ndalama zimene mumalipila kuchipatala?

Increased 1

Remained the same 2

Decreased 3

408 How far is the health centre from your village? Kodi chipatala chino ndichotalikilana bwanji ndi mudzi wanu?

Close to dwelling house 1

Less than 1Km 2

2- 3 km 3

3-5km 4

Over 5km 5

409 Since 2007, has the distance between your house and the health center you use changed? Kuyambira mchaka cha 2007, kodi pali kusintha kotani pa mtunda womwe mumayenda kuchokera kunyumba kwanu ndi kudzafika ku chipatala?

It is closer 1

Same distance 2

Farther away 3

410 Since 2007, has the cleanliness and hygiene of the health center changed? Kodi chichokeleni chaka cha 2007 pali kusintha kwina kulikonse pa zaukhondo wa pachipatalachi?

It improved 1

No change 2

It worsened 3

411 Since 2007, has the quality of the service changed? It improved 1

85

Kodi chichokeleni chaka cha 2007 pali kusintha kwina kulikonse kumbali ya chithandizo chimene mumalandila?

No change 2

It worsened 3

412 Since 2007, has the time you spend in the queue changed? Kodi chichokeleni chaka cha 2007 pali kusintha kwina kulikonse pa nthawi yomwe mumatenga kuti mulandile chithandizo?

It has diminished 1

No change 2

It has increased 3

413 Since, 2007 have there been changes regarding the access to medicines? Kodi chichokeleni chaka cha 2007 pali kusintha kwinakulikonse pakalandilidwe ka makwala?

It improved 1

No change 2

It worsened 3

MODULE 500: OVERALL IMPACT 501 What are the changes at the household level with regard to knowledge about WASH programmes

or activities? Pa khomo pano, mwawona kusintha kotani pa nkhani yokhuzana ndi ntchito zoti muli ndi madzi abwino komanso ukhondo?

___________________________________________________________________________________ ___________________________________________________________________________________ ________________________________________________________________________________ 502 How could similar WASH programmes or activities be improved in future?

Kodi ndondomeko zoona kuti muli ndi madzi abwino komanso ukhondo mungafune kuti zizakonzedwenso motani mtsogolo muno?

____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Thanks You. Zikomo

86

Annex 3: Questionnaire for school going pupils

QUESTIONNAIRE FOR LEANERS

Dear Friend,

From 2007 to 2013 UNICEF- Malawi with funding from the Government of Netherlands implemented a project on Water, Sanitation and Hygiene (WASH), in support of water supply and sanitation programmes in Malawi. This programme has been implemented in 12 Districts including Kasungu, Salima, Mzimba, Likoma, Lilongwe, Mwanza, Dowa, Mchinji, Nkhata-bay, Blantyre, Chitipa and Mangochi. The programme targeted 180,000 pupils from 300 primary schools to use safe drinking water and sanitation facilities.

At this moment, we are making an evaluation of this programme. We want to find out whether it contributed to a change in the lives of school going pupils. For that reason, we would like you to participate in a survey we are holding with leaners. In this classroom one of our team members will explain to you how this survey works. Please listen carefully to his or her explanation. If you have any query, don’t hesitate to raise your hand and ask for help.

Before starting answering the questions, please have a look at the list of questions. You will see that each question is accompanied by a series of answers. You should respond to a question by marking the answer that best corresponds to your particular situation. Tick the right answer by placing an “X” in the indicated space.

In most cases, you will be asked to mark only one answer. In some cases, however, you may find that more than one answer applies. In these cases, we indicate that you may tick all the applicable answers.

In general, you will follow the questions in the normal order from the first to the next. However, in certain cases, a particular answer results in specific follow-up questions. In these cases, you will find after the answer you ticked an indication of the question you have to proceed to. The instructions in the right-hand side column guide you through the questionnaire. Please follow them carefully and only answer questions that apply to your specific situation.

Please note that we do not ask for your name because we will treat your answers confidentially and anonymously. This means that nobody will know how you as an individual responded to the questions. Therefore, please feel free and secure to respond as close the real situation you live in as possible.

True answers make this survey reliable. A reliable survey will help us to assess the performance of the project supported by UNICEF and the Government of Netherlands correctly and to provide the UNICEF staff with the correct instructions for improving its work.

Thank you very much for your cooperation

87

TO BE FILLED IN BY THE SURVEY ASSISTANT

Date

School

District

TA

Enumerator code

TO BE FILLED IN BY THE RESPONDENT

1 How old are you? .......................... Years

2 How long have you lived in this area? I was born here (1)

More than 5 years (2)

3 Which class are you in? Standard..........................

4 Are you a boy or a girl? Boy (1)

Girl (2)

5 What factors affect negatively your school attendance?

Poor toilets (1) You may tick more than one Teacher shortages (2)

Distance to school (3)

Work at home or elsewhere (4)

Harassment (5)

Abuse (6)

Lack of food (7)

Lack of uniform (8)

Lack of learning materials (9)

Health-related illness (11)

Others (12) (specify) ………………

6 What is the condition of the toilets at this school?

Clean (1)

Somewhat clean (2)

Not clean (3)

Sometimes clean (4)

7 Do the water points have water all the time?

Always (1)

Sometimes (2)

No water (3)

8 Is there always washing soap? Always (1)

Sometimes (2)

No soap (3)

9 If children don’t use the toilets, What could be the reason for young children not to use the toilets?

Smell (1)

Darkness (2)

Privacy (3)

Other (4)

10 If girls don’t use the toilets, What could be the reason for girls not to use the toilets?

Smell (1)

Darkness (2)

Privacy (3)

Other (4)

11 Do you use the toilet every time that you want to help yourself?

Yes (1)

No (0)

12 Specify where you go Nearby bush (1)

Home (2)

Other(9) (specify) ………………

88

13 How many times do you wash your hands after using the toilet?

Always (1)

Sometimes (2)

Never (3)

14 Where does the school water come from?

Shallow well (1)

Tap (2)

Borehole (3)

Stream/river (4)

Other (9) (specify)

15 What role do you play in keeping the toilets clean?

Mopping (1) Multiple answers may apply

Sweeping (2)

Brushing (3)

None (4)

Bush clearing (5)

Others (9) (specify) ____________

16 What role do you play in making sure that the water points and hand washing facilities are in good condition?

Reporting when damaged (1) Multiple response may apply Cleaning (2)

Bush clearing (3)

17 Do water points function throughout school term?

Yes (1)

No (0)

18 Do you have safe water at home? Yes (1)

No (0)

19 Have you been trained on sanitation and hygiene?

Yes (1)

No (0)

20

What is the importance of hygiene practices that you were taught?

Promote good health (1)

Improve learning (2)

Environment (2)

Other (9) (specify)

21 Who taught you? Government official (1) NGO official (2)

Teachers (3)

Mother group (4)

Committee members (5)

WASH (6)

Community Sanitation committee(7)

Family members(8)

Other (9) (specify)

22 What else will you want for hygiene and sanitation

More toilets (1) Hand washing facilities (2)

Wash rooms (3)

Soap (4)

Water point(5)

23 Which hygiene/sanitation practices do you do at home?

Washing hands before and after eating (1)

Washings hands after using toilet (2)

Others (9)

24 Compare the condition of home and school toilets

Same (1) School better than home (2)

89

Home better than school (3)

No toilet at home (4)

25 What health-related challenges are there at your school?

No water (1) No washrooms (2)

No toilets (3)

No hand washing facilities (4)

Other(9) (specify)

Thank you very much for your collaboration. Now please hand in the completed form to your survey assistant.

90

Annex 4: Checklist for key informants

A: MINISTRY OF WATER DEVELOPMENT AND IRRIGATION

What is/was your role in the project?

What is your opinion of the UNICEF-funded project?

In what way are you related to UNICEF under this project?

What constraints have you faced in the implementation of this project?

How did the project contribute to government strategies or policies including SWAp.; water and sanitation policy?

To what extent were the project activities decentralised? In what ways? Eg financial management

How did the project contribute to capacity building of the ministry?

How does the project integrate aspects of water resource management?

How ere other ministries/departments involved in the projects?

What behavioural changes have taken place due to the project?

How best can a similar project be implemented?

B: MINISTRY OF HEALTH

What are the perceived water-related health challenges at national level?

Which districts are the most affected or critical?

What strategy was used to identify the beneficiaries? Districts?

What is/was your role in the project?

What were the institutional arrangements for implementation of the project between UNICEF, your ministry and other partners?

What is your opinion of the UNICEF-funded project with respect to sustainability?

What constraints/challenges have you faced in the implementation of this project?

How did the project contribute to government strategies or policies including SWAp, Sanitation Policy/WASH Strategy.

To what extent were the project activities decentralised? In what ways? E.g. financial management

How did the project contribute to capacity building of the ministry?

How does the project integrate aspects of water resource management?

How ere other ministries/departments involved in the projects?

What is your perception of the impact of the project?

Are there any lessons learned or best practices that can be up-scaled? How does this compare with other similar projects?

What behavioural changes have taken place due to the project?

How would you want similar projects planned and implemented in future for success?

91

UNICEF- MALAWI

What strategy was used to identify the beneficiaries? Districts?

What were the institutional arrangements for implementation of the project between UNICEF and partners?

What constraints/challenges have you faced in the implementation of this project?

How did the project contribute to government strategies or policies including SWAp, Sanitation Policy/WASH Strategy.

How was advocacy planned and /or carried out?

To what extent were the project activities decentralised? In what ways? E.g. financial management

How did the project contribute to capacity building of the ministry?

What is your perception of the impact of the project?

What are the lessons learned or best practices that haven been up-scaled?

What behavioural changes have taken place due to the project?

How would you want similar projects planned and implemented in future to ensure success? DISTRICT LEVEL QUESTIONS (District officials, NGOs and key informants)

What is/was your role in the project?

What strategy was used to identify the beneficiaries? Districts?

What were the institutional arrangements for implementation of the project between you and UNICEF?

What advocacy mechanisms were used and how effective were they?

How did they engage the communities?

What behavioural changes have taken place due to the project?

How did they take into consideration the needs of the people?

What constraints/challenges have you faced in the implementation of this project?

To what extent were the project activities decentralised? In what ways? E.g. financial management

How did the project contribute to capacity building at district level?

How did the project integrate aspects of water resource management?

How were other stakeholders involved in the project?

What is your perception of the impact of the project?

Are there any lessons learned or best practices that can be up-scaled? How does this compare with other similar projects?

What is your opinion of the UNICEF-funded project with respect to sustainability?

How would you want similar projects planned and implemented in future for success?

What are the social aspects that affect water, toilet and washing facilities?

How was data and information collected and managed?

How did the programme contribute to district plans?

How did they engage the communities?

How did they take into consideration the needs of the people?

COMMITTEES

How did the project approach and work with the committee?

What is/was your role in the project?

How were the needs identified?

Assess the functionality of the committees

What advocacy mechanisms were used and how effective were they?

How did they engage the communities?

92

What behavioural changes have taken place due to the project?

How did they take into consideration the needs of the people?

What constraints/challenges have you faced in the implementation of this project?

What type of training did they participate?

What is your perception of the impact of the project?

Are there any lessons learned or best practices that can be up-scaled? How does this compare with other similar projects?

What is your opinion of the UNICEF-funded project with respect to sustainability?

How would you want similar projects planned and implemented in future for success?

How was data and information collected and managed at community level?

How the programme did contributed to community development plans?

How did they participate in the implementation of the project?

How were interventions promoted?

What challenges have you faced with water points?

How did they take into consideration the needs of the people?

How best can the project be implemented in future?

How do you manage operational maintenance of boreholes?

93

Annex 5: Field research and data collection

1.0 TRAINING OF SURVEY ASSISTANTS The evaluation exercise started with training that was held from 19 October to 21 October, 2015. In order to identify qualified personnel, all survey assistants participated in an aptitude test to key issues and challenges that are linked to WASH programmes and interventions in Malawi. Upon selection of the required number of assistants, the facilitators provided the back ground to the study including the objectives and the methods to be used. Further information was provided for the study sites and deliverables to the client. The study tools that were covered during the training were the household and school questionnaire to be administered to communities that benefited from the programme. In addition, two more questionnaires to administer to health centre and water users were also discussed. This process covered key terms of the evaluations and mechanisms of going through the responses. In addition, the questions were translated to local language - Chichewa. This process provided common understanding of the tool and involved simulations of how the questions would be asked in an actual face to face interview. During this exercise, some questions, where necessary, were rephrased to make them more meaningful or some were completely removed. In other cases, additional questions were added to ensure a good flow of the questionnaire. The other tools that were covered include checklists for health workers, key informants and other officials. The questionnaire was pre-tested in T/A Chadza’s village. All the tools were tested including the questionnaire for School Learners at Chadza Primary School. Among others things, the team inspected the condition of the toilets, boreholes and the surrounding environment in general, in order to inspect the impact of WASH related programmes implemented at the school. Later on household interviews were administered. Water users were intercepted at water points and interviewed, health service users were also intercepted and interviewed at Chadza Health Centre and finally, one FGD was conducted with women in the village to gather more insights on how they benefited from the WASH interventions. During debriefing to discuss the outcome of the pre-testing, it was agreed that the households, water users, and health service user’s questionnaires will be merged as the information collected was similar. All the survey assistants were instructed to always seek consent before any interview and maintain confidentiality on all data collected. For example, the team was asked to get consent first from the school administration when they get to a school. Upon being granted that consent they should only interview boys and girls who have been at school for more than 5 years, those in grade 6, 7 and 8 and complete the observation form for the school. At household levels, consent had to be sought from the village heads. Arrangements were made in advance to visit the targeted areas and interview respondents who have stayed in the area for more than 5 years either males or females and further observe and complete the household toilet observation form. Logistical issues were also discussed on how the teams should operate. In conclusion, the merging of the three study components into one questionnaire; thus Household, Health and Water User questionnaires eased pressure on the execution of the fieldwork as the team was only concentrating on households instead of going to the health facilities or water points to conduct interviews. In such areas, information was collected through key informant interviews. 2.0 FIELD RESEARCH After the training, two separate teams visited the target Districts (see below). One team with 10 local surveyors (6 female and 2 male) students and one WASH expert did the household surveys while the other team with two local surveyors (1 female and 1 male) and the other WASH expert inspected the Water

94

Points (WP) and the schools. The second team also did the interviews with the District authorities (names of the persons consulted in Annex 1 and the summary of the interviews in Annex 3). 2.1 Selection of target Districts Based on the information provided on programme impact in Districts, three Districts were selected for evaluation. Table 1 presents criteria for selecting the target Districts.

Table 8: Selection criteria for Districts

District (*) Coverage of safe water (%)

Region (S)outh / (C)entral /(N)orth

Location (L)ake / (H)ighlands)

Blantyre 66 S H Chitipa 66 N H Dowe 61 C H Kasungu 54 C H Lilkome 63 N L Lilongwe 53 C H Mangochi 73 S L Mcinji 62 C H Mwanza 65 S H Mzimba 65 N H Nkhatabay 67 N L Salima 76 C L

Source: District water and sanitation plans (2007) The Districts were selected based on the difference in coverage of safe water: (low, 53%), medium (65%) and high (76%). In addition, the regional representation of South, Centre and North were taken into account. The topographic and landscape issues were also considered for lakeshore, central plain and northern highlands. The Districts were selected as follows:

Mangochi: South region / lake and high water coverage

Lilongwe: Central region / highlands and low water coverage

Mzimba: North region / highlands and medium water coverage 2.2 Selection of TA's and villages Within the Districts, the TA's were, generally, selected randomly from the TA list (maps in Annex 5). However, since the help of local guides was vital, the team limited the selection of visited locations to TAs where local guides were available. The guides led the teams to the villages and water points. Based on the available guides, ten (10) TA's were visited. Within the TAs, only the villages that participated in the 2007-2013 WASH programme were involved. The selection was done with assistance of the District Coordinating Committee. Lilongwe: Chadza, Malili, Masumbankhumba Mangochi: Chowe, Chimwala, Namavi, Makanjila Mzimba: Mbelwa, Chindi, Mzikubola, Kampingo Sibande

95

In Lilongwe District, the first visited TA was Chadza, located in the Eastern part of the District with easy access to most villages and about 15km from the urban areas. The second visited TA was located in the Western part of the District with villages situated about 25km to 45km from the urban centre. In Mangochi District, the two visited TAs were located on the Eastern bank of the Lake Malawi. The TAs are occupied by the Yao tribe adhering to Muslim religion and values. The TAs are located over 60km from the District headquarters and are difficult to access. The Chowe TA was only covered for schools and borehole assessment and not for household interviews. In Mzimba, four TA's were visited. They were generally sparsely populated, difficult to access, located mostly in hilly areas. 2.2 Selection of water points Within the visited TAs the water points were selected randomly (based on the list provided by the Districts, if available). However, the inspection of water points was time consuming because they were widely spread in the TA's and often far from the main road. For this reason, the team inspected in total 53 water points (Lilongwe 22, Mangochi 21 and Mzimba 10) instead of the target of 90. Table 9 presents the complete lists of water points inspected. The sample size represents 1 % of the total number of water points. 2.3 Selection of schools and Health Centres (HC) Records of UNICEF target schools were generally available at the Districts. The team made a random selection of schools from the records, however the final selection was also dependent on the travel plan in which water points were given a priority. The team visited in total 17 school toilets, which represented 5 % of all the UNICEF rehabilitated school toilets. Due to the long distances and that most of health related information was collected at household level, only one health centre was visited (the Chadza Health Centre in the Lilongwe District). 2.4 Selection of communities and target household (HH) In the inception report, we provided a generalized sample size irrespective of population size, cultural issues and geographical coverage. The sample size was calculated using simple random sampling technique as follows:

Where: n is the required sample size

N is the population size of the District P and Q are the population proportions (set to 0.5) Z is the confidence level (set to 1.96) E is the sample proportion accuracy (set to 0.03)

The formula above gives the sample size for the Districts as in Table 9.

Table 9: Planned sample size

District Population Ideal sample

Actual sample size

%

Lilongwe 560,000 204 200 0.04 Mzimba 895,000 377 200 0.02 Mangochi 805,000 280 200 0.02

96

The following TA's where UNICEF implemented the WASH project were visited. Table 10 provides the actual sample size for the survey for both households and schools.

Table 10: Actual sample size for the evaluation for households and schools

District TA Number of households

Number of schools (number of students)

Lilongwe Chadza 58 1 (12) Malili 142 2 (21)

Mangochi Namabvi 34 0 Makanjira 166 1(10) Chowe 0 3 (31)

Mzimba

Mzikubola 37 4(41) Kampingo Sibande 22 0 Chindi 122 0 Mbelwa 19 0

Total / Average 600/ 67 11 (115)

Within the TAs, a list of Group Village Headmen were purposively selected from which a list of villages participating in the study was randomly selected. Only the villages participating in the 2007-2013 WASH Programme were visited. The research team adopted a rigid mechanical walk targeting households in their communities. Both adult male and female respondents were interviewed as long as they met the study criteria. The interviewers targeted any household member who had lived in the area for more than 5 years and was a direct beneficiary of the interventions. This was done to ensure that only respondents who had experience with WASH interventions provided feedback on their perceived impacts and challenges of the programme. Apart from household interviews (FGDs), key informant interviews and school learners’ interviews were also conducted in the intervention areas. The FGDs targeted only women. The learners’ interviews targeted students in classes 6, 7 and 8 who studied in their schools for at least five years. 2.4 Focus group discussions Two focus group discussions were held in each of the TAs covered in this study. The size of the focus groups ranged from 8 to 12 participants. The villages visited in each TA are listed in Table 11.

Table 11: Villages were FGDs were held

District Traditional Authority Villages

Lilongwe Chadza Ngala Malili Mbilizi Mangochi Makanjila Binali Bakili Ng’ombe Mzimba Mzukubola Binali Chindi Chindi

2.5 Interview and inspection methodology At water points the team inspected the technical condition of the hand-pumps (HP), yield and the taste of the water. In addition, the team checked the condition and hygiene of the apron and drainage channel. The technical condition of the HP was checked by assessing the play of the bush bearing and if water was

97

pumped with a reasonable yield and without unusual friction. The team also interviewed the water users and, if available, the members of the WHWC and satisfaction, financial and maintenance issues. 3.0 FIELD SAMPLE OBSERVATIONS 3.1 Household data During the survey, majority of the participants were females (82.3%, n=494) while males comprised 17.7% (n=106). Of these, only 33.5% were heads of the household. Majority of the survey participants attained primary school education (66.8%) with few having attained secondary school education (10%), some had vocational training (0.5%), and university or college education (1.2%). The remaining 21.5% of interviewed people had no education. 78.2% of the participants were married, 9.5% widowed, 6% single, 6.3% separated. Although the statistics show that most of the participants were female, it transpired that majority of the households were male-headed (80.3%) with only 19% and 0.2% reported as female or child-headed respectively. 3.2 School learners data The study covered 115 students of which 35.7% were in Mangochi, 35.7% in Mzimba and 28.6% in Lilongwe district (see Table 12).

Table 12: Information on gender of the leaner

District TA Male Female

Lilongwe Malisi 10 11 Chadza 5 7 Mangochi Chowe 14 17 Namabvi 5 5 Mzimba Mzikubola 19 22

In terms of age, the youngest participant was 11 and the oldest was 19 years old with an average age of 14.6 years old. Majority of the study participants (66.1%) were born in the areas where they were interviewed, 33.9% stayed in those areas for more than 5 years. The study covered students who were in form 3, form 4, standard 7 and Standard represented by 3.5%, 5.2%, 33.9% and 57.4% respectively. 4.0 Information on villages visited during field survey The following section provides information on the number of villages that were visited in each District and the interventions provided: boreholes and CLTS. Table 13 shows that in Lilongwe, in some villages the boreholes and CLTS projects were separate.

Table 13: Villages with borehole and CLTS interventions in Lilongwe

District GVH

Malili Chadza

Borehol CLTS Borehole CLTS

Lilongwe

Kaoche 5

Nsipuwansalu 38

Chamadenga

13

Makanga

43

Mpokosa

1

Kalambo 1

34

Nkhuku

3

98

Mwimba 9

Kamtambo 5

Mbiliza 43

1

Chimwano

5

Total 100 42 58

In Mangochi (Table 14) the villages visited had both interventions (boreholes and CLTS). More villages covered with GVH Makonjeni and Binali

Table 14: Villages in Mangochi

GVH Makanjira TA Namabvi TA Boreholes + CLTS Boreholes + CLTS

Namabvi 15 Makonjeni 42 Njelenje 26 Binali 41 Bakili 23 Mutiya 2 Maleza 1 Chapola 20 Tumbwe 20 Ng'ombe 10 Total 145 55

Table 15 provides similar statistics for Mzimba District and shows how spread are the villages.

Table 15: Villages and interventions in Mzimba District

GHV

Mzimba Traditional Authority Mzikubola Mbelwa Chindi Kampingo Sibande

Borehole CLTS Borehole CLTS Borehole CLTS Borehole CLTS

Chikadawanga Lusale

7

Kandodo Chisi

12

Mpetama Gondwe 9

Kafoteka Mkandawire 24

German Mahowe 12

Sindani 7

Chikadawanga Lusale

17

taja Chavula 1

George Nkosi 13

Gundungu Chadewa 37

Bango 6

Peter Ndawandawa

7

Grandwell Lunkhere 2

Mpeyama 13

Yobe Gama 12

Vostes Moyo 1

Chakwandamira 2

Saulosi Mwanza 3

Bongo Moyo 2

99

David Chingwalo 1

Chimembe 2

Kanyalu Chadewa 3

Hanock Lukhepo

1

Sudan Chimwambo 6

Total 24 24 12 19 67 50 3 1

100

Table 16: Water points’ data

Lilongwe

#

Date

of

ob

servatio

n

BH

#

TA

Village

Scho

ol

Co

nd

ition

of

HP

Low

yield

d

urin

g dry

seaso

n?

Co

nd

ition

of

apro

n

Hygie

ne

Taste o

f

wate

r

Ow

ner

Co

mp

ositio

n

of V

HW

C (M

/ F)

Co

mm

un

ity

paym

en

y (C

/R)

Extern

rep

air use

d

(Y/N

)

Ye

ar of

con

structio

n

Re

marks

1 21-10-15 Chadza Ngala

Ngala primary school 1 2 2 OK S R Y

Suffered major breakdown costing 70,000 KW paid by an organization

2 21-10-15 44

6 Chadza Mnonksa 2 2 2 Salty C 10/6 R

3 21-10-15 2 Chadza Mnonksa 1 1 2 Muddy 2012 Milky water in rainy season

4 21-10-15 Chadza Chimombe CDSS 1 1 2 S

2011

Used by community, but they do not contribute

5 23-10-15 Chadza Mbilize 2 2 2 Ok C 6/4 R N 2009 contribution is a problem /locked

6 23-10-15 Chadza Mchedwa 2 3 2 Ok S S Y

Collect money from learners School will charge villagers for use

7 23-10-15 Malili Chiwenga 3 2 3 S R Y Rods too short area mechanic could not repair

8 23-10-15 Malili Kathanga 1 Low 3 2 OK C 6 /4 C N Apron broken water can infiltrate / bank account 6000

9 23-10-15 Malili Kawale 1 1 2 R Y

Suffered major breakdown, but repaired by an organization

10 23-10-15 Malili Kalema I 2 3 1 UNICEF?

11 23-10-15 Malili Kalima II 1 1 3 OK C 4/6 R 2012

12 23-10-15 Malili Chikute 2 Low 1 1 Ok C R 2012

101

#

Date

of

ob

servation

BH

#

TA

Village

Scho

ol

Co

nd

ition

of H

P

Low

yield

d

urin

g dry

season

?

Co

nd

ition

of ap

ron

Hygie

ne

Taste o

f w

ater

Ow

ner

Co

mp

ositio

n o

f VH

WC

(M

/ F)

Co

mm

un

ity

paym

en

y (C

/R)

Extern

repair u

sed

(Y

/N)

Ye

ar of

con

structio

n

Re

marks

13 23-10-15 Malili Kuthindi 1 1 1 OK C 6/4 C N Collected already 10000 Kwacha

14 23-10-15 Malili Mwera Pre -school 2 2 2 Salty C 10/8 R 2010

15 23-10-15 Malili Mbeta 2 1 2 Salty

16 24-10-15 14

7 Malili Chala School 1 2 1 OK S 6/4 R Y 2011

costs for repairs 6000 Kw very high for community

17 24-10-15 6 Malili Dzoore 1 1 3 Salty 2011 Too salty to drink

18 24-10-15 49 Malili Chimvite School 1 1 1 OK S 2011

19 24-10-15 26

Masumbankhumba Madumbza 1 2 1 OK 10/8 R N 2010

20 24-10-15

Masumbankhumba Nkalema 1 2 2 OK 3/3 R N 2010

21 24-10-15 25

Masumbankhumba Nderema 2 1 2 OK 5/5 N 2011

Don't know where to get SP

22 24-10-15 2

Masumbankhumba Dzodzwe 2 1 2 OK 5/3 R Y 2009

BH was broken for 3 months and no money to repair

102

Mangochi

#

Date

of

ob

servatio

n

BH

#

TA

Village

Scho

ol

Co

nd

ition

of H

P

Low

yield

du

ring d

ry se

ason

?

Co

nd

ition

o

f apro

n

Hygie

ne

Taste o

f w

ater

Ow

ner

Co

mp

ositio

n o

f VH

WC

(M

/F)

Co

mm

un

ity p

ayme

ny

(C/R

)

Extern

rep

air use

d

(Y/N

)

Ye

ar of

con

structio

n

Re

marks

23 26-10-15 Chowe Dimu Mangale 2 1 2 Salty s/c 5/3 N

The BH has salty water hence the community don’t use the water for drinking and only uses it in the dry season

24 26-10-15 Chowe Chimbende 1 2 1 Ok C 11/6 C Y

UNICEF BH, but revised by another organisation in 2015

25 26-10-15 Chowe St Martens School 2 1 2 Salty C 6/4 C Y 2008

26 26-10-15 Chowe Malukula school 1 2 2 Ok s 10/6 R N

27 26-10-15 Chowe Kasila 1 3 2 Ok C 10/6 C Y 2011

28 27-10-15 Chowe M'baluku 1 1 1 Ok C 10/7 C N 2011

problem for the committee to collect money from the community as some say that the BH was given by the Government for free hence no need to pay for the water

29 27-10-15 Chowe Nkulumba Msinjiri 2 2 3 Muddy S/C 7/5 C Y

the water changes color in the rainy season , it turn milky and muddy

30 27-10-15 Namavi Lungwena School 2 1 2 Ok C 10/10 C N

Still OK, but community fears that they cannot pay repairs

31 27-10-15 Namavi Taliya 2 1 2 Ok C 10/6 C Y

103

#

Date

of

ob

servation

BH

#

TA

Village

Scho

ol

Co

nd

ition

of

HP

Low

yield

du

ring d

ry seaso

n?

Co

nd

ition

of

apro

n

Hygie

ne

Taste o

f

wate

r

Ow

ner

Co

mp

ositio

n

of V

HW

C

(M / F)

Co

mm

un

ity

paym

en

y (C

/R)

Extern re

pair

used

(Y/N

)

Ye

ar of

con

structio

n

Re

marks

32 27-10-15 25 Namavi Tumbwe 1 1 2 Salty C 10/6 C Y 2011

33 27-10-15 23 Namavi Chilawe 1 1 2 Ok C 3/2 C Y 2011 Large community depending only one BH

34 28-10-15 Chimwala Kabota 2 1 1 Ok C 10/5 C Y 2009

35 28-10-15 Chimwala Abina 2 1 1 Ok C 4/3 C Y 2009

36 28-10-15 Chimwala Gunga 2 Low 2 1 Ok C 10/5 R Y

37 28-10-15 Chimwala Mwaye school 2 Low 2 1 Ok S/C 6/5 C Y

Low yield of the BH in the dry season

38 28-10-15 1 Chimwala Muonjeza 1 1 1 Ok C 10/5 R Y 2007

The BH overworks as the community is so large and it has only one working BH

39 28-10-15 2 Chimwala Josamu 2 Low 1 1 Ok C 10/6 C Y 2011

40 28-10-15 Chimwala Chimpeni 1 1 2 Ok C 6/4 C Y 2010

41 28-10-15 28 Chimwala McLoud 1 1 2 Ok C 10/8 C Y 2011

Village need external assistance for all maintenance

42 28-10-15 Chimwala Bango 2 1 1 Ok C 10/5 Y 2009

People don’t contribute anything for maintenance due to unwillingness

43 28-10-15 Chimwala Tsamilani 1 1 3 Ok C 8/6 C Y

Management of the BH is difficult-the community is very big and they have this one BH

104

Mzimba

#

Date

of

ob

servation

BH

#

TA

Village

Scho

ol

Co

nd

ition

of H

P

Low

yield

d

urin

g dry

seaso

n?

Co

nd

ition

of ap

ron

Hygie

ne

Taste o

f

wate

r

Ow

ner

Co

mp

ositio

n o

f VH

WC

(M

/ F)

Co

mm

un

ity

paym

en

y (C

/R)

Extern

rep

air u

sed

(Y/N

)

Ye

ar of

con

structio

n

Re

marks

44 30-10-15 Mbelwa Kafoteka 2 1 2 Ok C 3/7 C Y 2011

Village AM migrated and they are now dependent on external expertise. They have 8000 Kwacha in cash

45 31-10-15 Chindi Tadeyo Mseteka 2 Low 1 2 Ok C R 2011

46 31-10-15 Chindi Damaseke Gondwe 2 2 2 Ok C 10/2 R Y 2009

The BH frequently damages, they have K500 and SP

47 31-10-15 Chindi Tonthowere 1 1 1 Ok S 2011

48 31-10-15 Chindi Mbogo Moyo 1 1 1 Ok C

49 31-10-15 Chindi Ziyaka Ngwira 1 1 1 Ok C

Ladies at the WP did not have info

50 31-10-15 Chindi Andrea Phiri 1 1 1 Ok C

Ladies at the WP did not have info

51 31-10-15 Chindi David Chibambo 1 3 3 Ok C 10/5 R Y

The apron has some serious damages, they contribute K300 when damaged and they currently have K4,200

52 31-10-15 Chindi Matchonisa Msuku 1 1 2 Ok C

Ladies at the WP did not have info

53 31-10-15 Chindi Gwala Banda 1 1 1 Ok C

Ladies at the WP did not have info

105

Annex 6: List of people consulted

Name organization Position E-mail Phone

Mr. Paulos Workneh UNICEF Chief of WASH pworkneh@unicef.org 0999964210

Ms. Sarah Ahmed Mirza

UNICEF Chief planning and M&E

sahmad@unicef.org 0994961201

Mr. Mekonnen Ashenafi Woldegorgis

UNICEF Evaluation Specialist

mwoldegorgis@unicef.org 0994163668

Mr. Nipon Chizani UNICEF M&E specialist nchigami@unicef.org 0999963146

Mr. Chimwemwe Nyimba

UNICEF WASH specialist

Mr. Christopher Davids

UNICEF M&E specialist cdavids@unicef.org 0994961201

Mr. Kyaw Myint Aung UNICEF Chief of health kaung@unicef.org 099 9964 211

Ms. Emma M. Mbalame

MoWD&I Deputy director Water Supply service

emmambalame@gmail.com 0888 857 831

Mr. Tchaka Thole CPAR Country manager

tchakathole@yahoo.com 0888438079

Joseph Zimba CPAR Programme Manager

josephz.cpar@gmail.com 0888838082

Olivia Kachuma CPAR M and E Officer

oliviakachuma@yahoo.com 0999964227

Ms. Marvel Tibu District Lilongwe

District Water Officer (DWO)

mnakanga@yahoo.com 0999270206

Mr. Jack A.K. Nguluwe District Mangochi

District Commissioner

jack.nguluwe@yahoo.com 0888873283

Mr. K Mamba District Mangochi

District Environmental Health Officer (DEHO)

ngangakulaiti@yahoo.com 0995 650623/ 0888307670

Mr. Hassan Maluwa District Mangochi

Hassanmaluwa@ gmail.com

Mr. F. Kapide District Mangochi

Mr Kondwani Andreah

District Mangochi

District water development officer

koandreah@yahoo.co.uk koandreah@gmail.com

0999932603 / 0888932602

Mr. Anthony Zimba District of Mzimba

District comm. Dev. officer

anthony.zimba@yahoo.com 0888586680

Mighty Kayoyo District of Mzimba

District Health coordinator

migthykayoyo@gmail.com 0888610548

Andrew Somaliland District of Mzimba

Environment Health Officer

kurzsomaliland@rockefmail.com

0888117476

Mr. Ephraim C. Mbewa

District of Mzimba

District Water officer

mbewaephraim@yahoo.com ephraimohasiya@gmail.com

0999278450 / 0884266045

Ms. Prisca Kutengule National Water Development Programme

Community participation specialist

pkutengule@hotmail.com 0999225472

Mr. Lazarus Phiri AfDB project Project coordinator

laphiri@gmail.com 0999955534

106

Mr. Masauku Mthunzi Concern Universal

WASH programme manager

masauko.mthunzi@concern-universal.org

0991389222

Mr. Smorden Tomoka

Concern Universal

Programme Coordinator - WASH

Smorden.Tomoka@concern-universal.org

08888 79627

Mr. James Mambulu DFID Infrastructure advisor / WASH

j-mambulu@dfid.gov.uk

0999934483

Mr Evalista Mfune Ngala Primary School (Lilongwe)

Teacher

Francis Kavala Ngala Primary School (Lilongwe)

Teacher

Simeone Getala Ngala Primary School (Lilongwe)

Teacher

Simeyi Bikiele Ngala Primary School (Lilongwe)

Head Teacher

107

Annex 7: Annual and total results of the UNICEF WASH programme

Result description

as per project

document

PERIOD

2007 2008 2009 2010 2011 2012 2013 All: 2007 -2013

Output Outcome - people served

Output

Outcome - people served

Output

Outcome - people served

Output

Outcome - people served

Output

Outcome - people served

Output

Outcome - people served

Output

Outcome - people served

Total out- puts for all milestones

Outcome - people served

New community drinking WS 0 0 226 56,500 642 160,524 610 152,500 601 137,770

1,169

292, 306 0 0 3,248 799,600

Community rehabilitated water supply 678 170,000 75 18,750 213 53,250 75 18,750 193 48,250 766 191,560 0 0 2040 510,310

ODF villages 0 0 36 346 495 527 613 223 2240

Household latrines 0 0 1,770 62050

37,733 188,665

17,130 85,650

64,538 322,690

21,450 77,250

20,775 163,396 964,215

Schools with water 0 0 113 68,000 122 73,620 118 22,800 125 69369 0 0 0 0 331 198,600

School with sanitation / latrines 0 0 113 68,000 122 73,620 118 22,800 125 69369 0 0 0 0 331 198,600

Schools with hygiene 0 0 113 68,000 122 73,620 118 22,800 125 69369 0 0 0 0

Hand- washing 0 0 75,250 188,665 64,538 484,675 0 76,625 0 26,700 916,453 916,453

District Strategic Investment Plans 0 0 12

108

Annex 8: Achievements log frame: key outputs and outcomes

Results (*) Key Outputs (*) Achieved (**) Outcomes (*) Achieved (**)

RESULT 1: Improved access to, and sustained use of, safe water supply from 66 % to 80 % in 12 districts by 2012. (500 m)

2800 new boreholes 1010 rehab boreholes

3,241 new WP 2,042 rehabilitated WP

700 K new users 240 K from rehabs

Calculation method unclear, probably inflated

3850 WWHC are trained and established

Yes. This is confirmed by the WQA (UNICEF 2012). Less than 1% of the WP did not have a VHWC

80 % of 3850 VHWC raise revenues to buy spare parts

All surveyed water points have working VHWS, but at least 50 % of these only collect funds in case of a breakdown

Training of 600 Area Mechanics

No data 80 % of the 600 Area mechanics are still in business

This could not be proved, but the AM based maintenance system is still intact

Water supply constructed at 100 Health centres

No data 100 Health centres have safe water facilities

No data

60 % of the 3850 VHWC have women in key positions

All surveyed VHWC have female representation. On the average 50 % of the VHWC members is female

RESULT 2: improved access to, and sustained use of, adequate sanitation facilities from 10 percent to 22 percent in 12 districts by 2012.

600 area masons are trained

Not substantiated by internal reports

200 K HH have safe and hygienic latrines

?

5 vulnerable people / village receive demo latrines

Not substantiated by internal reports

At least 60 % of the houseolds are vulnerable and poor

Construct 4 latrines in 100 health centres

No data 100 Health centres have safe latrines

No data

80 Percent of 200 San centres function and provide service

Cancelled

RESULT 3: Improved water, sanitation and hygiene in 300 primary schools by 2012.

300 water and sanitation facilities constructed in schools

331 schools have water and sanitation facilities

300 primary schools have water within 200 m, 300 schools have safe sanitation facilities

The condition and hygiene of the sanitation facilities on schools was in general reasonable, but most schools lacked hand-washing facilities

RESULT 4: Use of Training of 300 school clubs 300 schools have active

109

improved hygiene practiced by one million

school clubs

Health officers trained in each of the Districts

Districts have functioning promotion team

50 % of the population has correct hygiene knowledge

Result 5: Improved community-based water resources management by the 2800 VHWC

Number of 3850 VHWCs practising integrated water resources management

Cancelled after two years Achieved a total 211,800 trees and 2000 fruit trees planted in three of the 12 Districts by 2009.

Percentage of 3850 water facilities in the 12 Districts with reduced yields

RESULT 6: Capacity building of partners for effective, efficient and accountable implementation of water and sanitation

Every District has a data management plan

Yes, 12 Districts with capacity to ensure quality for management and M&E

In none of the Districts the M&E system was working adequately. The Districts also lacked funds to support a network of water observers Every District has a M&E

system Yes, but barely working due to hard- and software constraints

Up to date District M&E system in place

Water point water point data in an up to date database

RESULTS 7: Improved knowledge management on water, sanitation & hygiene

Manuals with best practices produced

Manuals are still in use

RESULT 8: Strategic partnership developed

Coordination meetings are held

UNICEF had annual strategy meeitngs with the districts and has coordination meetings with the main DP's

Functioning SWAP and donor coordination

Up to a certain degree the SWAP is functioning. The development strategies on WASH are comparable

(*) Source: Detailed Progress and “Final” Report for 2007 - 2013 (**) Source: Field observations and interviews

110

Annex 9: TOR End of Term Evaluation of Water, Sanitation and Hygiene (WASH) Programme (2007-2013)

1. BACKGROUND UNICEF received funding from the Bill and Melinda Gates Foundation, DFID and the EU, for the ongoing country programme. It is anticipated that UNICEF will continue to receive donor funds after this date and be called upon to influence the strategic direction of National WASH development and demonstrate a clear comparative advantage in ensuring accountability, strengthening sector governance, innovation and ‘value for money’. Given these external drivers of change, there is a need for UNICEF to reflect on recent experience and review the outputs from the UNICEF-Netherlands partnership programme in order to identify the risks and opportunities for future programming, and ensure that the next country programme is both strategic and relevant; and geared to more effective risk management. The UNICEF-Netherland WASH programme evaluation was undertaken as part of a Country Programme review which is a key milestone in UNICEF’s country programming process, and provides an opportunity to undertake a comprehensive review of the progress towards achievement of the results outlined in the current programme documents. The evaluation will also be the basis for making the strategic choices that the country office needs to make for the forthcoming country programme. This will inform the country programme document (CPD) process for the new country programme, which will commence in 2017. UNICEF is therefore seeking to hire a consultant to assess the achievement of project objectives of UNICEF-Netherlands WASH in the 12 target Districts between the period 2007 and 2013. · To pinpoint strengths and weaknesses, and identify risks and opportunities for the WASH programme · To assess the extent to which the programme focused on the deprived and vulnerable; systems and capacity development including for financing in the sector; · To assess programme contribution to risk management and internal UNICEF capacity, especially in relation to upstream engagement. 2. SCOPE OF WORK The Consultant will specifically carry out the following tasks: The criteria of relevance, efficiency, effectiveness, impact and sustainability will be applied to programme objectives and strategic results areas (i.e capacity development, equity, policy, women and children empowerment and risk management). Specifically, the evaluation will seek to answer, but not limited to, the following questions: Relevance · Assess the internal, vertical and horizontal consistencies between sector policy/strategy and the UNICEF WASH programme; · Appropriateness of the project objectives in the overall problem context, needs and priorities of the target groups and beneficiaries; · How well the programme has identified the needs of the most deprived populations and how these have been built into programme results and monitoring; · How well the programme has identified and addressed the challenges of balancing between equity and efficiency dilemma for optimal results. Efficiency Project efficiency refers to the extent to which the various activities transformed the available resources into the intended outputs in terms of quantity, quality and timeliness. Specifically answer the following questions: · To what extent did programme implementation strategies contribute to national results; · What are the factors affecting the pace and quality of implementation and how can these be mitigated; · How efficient were the procedures and approaches adopted for infrastructural works and were they in conformity with UNICEF’s regulations? Effectiveness

111

· What actual results were achieved against planned results as outlined in the results matrix and against key performance indicators? · What capacities were developed in the sector as a result of the WASH programme and how did these contribute to the achievement of results? · What advocacy was carried out and how effective these have been? Impact · What impact did it achieve against its planned targets? · What community capacities and competencies were developed and how did these contribute to sustainability and resilience? Sustainability · What strategies for sustainability have been adopted and how effective these have been? · What competencies within the WASH section have been useful and what are the key human resource and skills gaps? · What is the likelihood of continued flow of benefits for beneficiaries after the project end? · What is the extant of programme influence on national level commitment for resource allocations and leveraging? · To what extent results achieved are sustainable at their respective levels (National, sub-national, communities) and if they can be replicated or integrated into policies, strategies and programmes? Performance Ratings The evaluation shall include an overall performance rating for each of the above five evaluation criteria, on the basis of the following scale: · Highly satisfactory (fully according to plan or better); · Satisfactory (on balance according to plan, positive aspects outweighing negative aspects); · Less than satisfactory (not sufficiently according to plan, taking account of the evolving context; a few positive aspects, but outweighed by negative aspects): · Highly unsatisfactory (seriously deficient, very few or no positive aspects). Each rating should be stated as part of the conclusions for each of the five criteria. 3. EDUCATION AND WORK EXPERIENCE · Advanced degree in water and sanitation related area preferable; · At least 5 years’ work experience in water and sanitation at national and sub-national levels; · Experience of sector policy advocacy, strategy development and funding arrangements; · Experience of WASH sector reviews and evaluations of WASH programmes; · Experience of risk assessment methodologies; · Knowledge of the Malawian water and sanitation sector desirable; · Knowledge of UNICEF internal procedures and programming an asset; · Excellent communication, analytical and writing skills, including writing reports, strategies and proposals in English; · Ability and flexibility to work to short deadlines and prioritise work, with minimal supervision; · Demonstrated ability to work in a multicultural environment and establish harmonious and effective working relationship both within and outside the organization. 4. DURATION 45 Days 5. METHOD OF APPLICATION Qualified candidates are requested to submit: - A cover letter, no longer than 2 pages, and curriculum vitae of the proposed members of the team (national team), showing how the consultant meets the required qualifications, experience and expertise. - A technical proposal, no longer than 3 pages, highlighting: How they have understood and interpreted the assignment and any suggestions to improve the outcomes

112

of the assignment. The proposed methodology that the consultant will utilize, including sampling frame, data collection methods and tools, processing, analysis and interpretation and reporting framework. Logistics and resources requirements, e.g. transport, perishables etc. Any ethic concerns that could affect people involved in the study Financial proposal, no longer than one page, including all eligible fees, (salaries, transportations and Daily subsistence allowance for data collection and fieldwork, communication costs. iv. Personal History Form (P-11 form) (which can be downloaded from our website at http://www.unicef.org/about/employ/index_53129.html) and attach copies of their certificates. Applications should be sent on or before 02 June 2015 to the following address

113

Annex 10: Pictures

Photo 1: WP with reasonable apron Photo: WP with cracks around the casing

Photo 3: Poor hygiene around the WP Photo 4: WP with Proper drainage

Photo 5: Properly built school latrines Photo 6: Dilapidated school latrine

114

Photo 7: School hand-washing facility Photo 8: HH Hand washing