Evaluation Report
Evaluation of the UNICEF Sanitation Programme at Scale in
Pakistan (SPSP) – Phase 1 (2013-14)
Final Version: Dec, 2014
[ii]
Disclaimer: The views and opinion expressed in this report are of the consultants and do not necessarily reflect the views or policies of UNICEF and/or other organizations involved in the programme.
[iii]
List of Acronyms and Abbreviations BCC Behaviour Change Communication
BHU Basic Health Unit
BRSP Baluchistan Rural Support Programme
CAP Community Action Plan
CATS Community Approach to Total Sanitation
CCD Climate Change Division
CEDAW Convention on the Elimination of All Forms of Discrimination against
Women
CLTS Community-Led Total Sanitation
CNA Capacity Needs Assessment
CRC Convention on the Rights of the Child
CRP Community Resource Person
DAC Development Assistance Committee
DFID Department for International Development
DRR Disaster Risk Reduction
DSI Development Solutions International
FATA Federal Administered Tribal Areas
FGD Focus Group Discussion
FRDP Fast Rural Development Program
GDP Gross Domestic Product
GoP Government of Pakistan
GSF Global Sanitation Fund
HACT Harmonised Approach to Cash Transfers
HCD Human Centred Design
HH Household
HRBA Human Rights Based Approach
IEC Information Education Communication
IP Implementing Partner
IRSP Integrated Regional Support Program
JMP Joint Monitoring Programme
KAP Knowledge Attitude and Practice
KP Khyber Pakhtunkhwa
LGRDD Local Government & Rural Development Department
LHW Lady Health Worker
MDG Millennium Development Goals
MEAL Monitoring, Evaluation, Accountability and Learning
MICS Multiple Indicator Cluster Surveys
MIRA Multi-cluster Initial Rapid Assessment
MoCC Ministry of Climate Change
MoRES Monitoring Results for Equity Systems
[iv]
MOU Memorandum of Understanding
NCBP Nations Capacity Building Programme
NDMA National Disaster Management Agency
NDWP National Drinking Water Policy
NFC National Finance Commission
NOC No Objection Certificate
NRSP National Rural Support Programme
OD Open Defecation
ODF Open Defecation Free
OECD Organisation for Economic Co-operation and Development
PATS Pakistan Approach to Total Sanitation
PCA Programme Cooperation Agreements
PDHS Pakistan Demographic and Health Survey
PHED Public Health Engineering Department
PMER Planning, Monitoring, Evaluation & Reporting
PRA Participatory Rural Appraisal
PSLM Pakistan Social and Living Standards Measurement Survey
RSPN Rural Support Programme Network
RUSFAD Rural Sanitation in Flood Affected Areas
SACOSAN South Asia Conferences on Sanitation
SAFWCO Sindh Agricultural and Forestry Workers Coordinating Organization
SLTS School-Led Total Sanitation
SMC School Management Committee
SO Social Organizer
SPSP Sanitation Programme at Scale in Pakistan
SPSS Statistical Package for the Social Sciences
SRSP Sarhad Rural Support Programme
TOC Theory of Change
TOR Terms of Reference
TPM Third Party Monitors
UC Union Council
UN United Nations
UNDAF United Nations Development Action Framework
UNEG United Nations Evaluation Group
UNICEF United Nations Children's Fund
VSC Village Sanitation Committee
WASH Water Sanitation and Hygiene
WDCO Women Development & Community Organization
WSP Water and Sanitation Program
WSSCC Water Supply and Sanitation Collaborative Council
[vi]
Acknowledgement The Consultants (AAN Associates) are extremely thankful to all those who contributed to the evaluation. Special thanks to Department for International Development (DFID), Government of United Kingdom, for funding the programme. We are thankful to the Implementing Partners (IPs) for coordinating field activities with communities and local stakeholders; and for sharing their own experiences with the evaluators. The consultants are thankful to the members of the ‘Evaluation Reference Group’, which included representatives from DFID, provincial and federal governments and UNICEF for their valued contributions through workshop on initial findings of evaluation. This is to acknowledge the continued support and guidance extended by UNICEF Pakistan and its various programme staff in Islamabad and the provincial offices. We would also like to thank UNICEF evaluation offices at the country and regional level for their consistent engagement, valuable feedback and advice. This may not have been possible without the hard work, resourcefulness and commitment of AAN’s team and the consultants engaged. Thank you, Wajahat Anwar, Salman Mahmood, Kashif Pervaiz, Zaki Ullah, Asmat Gill, Arsalan Kashfi and Arbab Qadir. We wish UNICEF and all other WASH stakeholders’ success with their work and hope to see a country where everyone has access to desirable and affordable WASH services. Nadeem Haider Team Lead/AAN Associates Islamabad Pakistan
[vii]
Table of Contents List of Tables ............................................................................................................ viii
List of Figures ............................................................................................................. ix
List of Annexes ........................................................................................................... x
1. EXECUTIVE SUMMARY .................................................................................... xi
2. OBJECT OF THE EVALUATION ........................................................................ 1
2.1. Programme Overview ....................................................................................... 1
2.2. Country Context ............................................................................................... 5
2.3. Theory of Change ............................................................................................. 8
3. EVALUATION PURPOSE, OBJECTIVES AND SCOPE .................................. 12
4. METHODOLOGY ............................................................................................. 15
4.1. Evaluation Conceptual Framework ................................................................. 15
4.2. Data Collection Methods ................................................................................ 15
4.2.1. Quantitative Evaluation Methods .......................................................... 16
4.2.2. Qualitative Evaluation Methods ................................................................ 17
4.3. Evaluation Constraints, Limitations and Mitigation ......................................... 19
4.4. Ethical Considerations .................................................................................... 22
4.5. Quality Assurance Mechanisms ..................................................................... 22
4.6. Evaluation Management and Stakeholder Participation ................................. 23
5. FINDINGS ........................................................................................................ 25
5.1. Relevance ...................................................................................................... 25
5.2. Effectiveness .................................................................................................. 32
5.2.1. Programme Design ............................................................................... 32
5.2.2. Programme Management & Coordination ............................................ 34
5.2.3. Programme Components & Results ..................................................... 35
Output 1: Demand for Sanitation ....................................................................... 35
Output 2: Supply Side Interventions .................................................................. 42
Output 3: Health and Hygiene Promotion (H&HP) ............................................ 50
Output 4: Linkages with Duty Bearers ............................................................... 53
Output 5: Attaining Total Sanitation ................................................................... 55
Output 6: Knowledge Management and Accountability ..................................... 57
5.2.4. Stakeholders Contribution .................................................................... 59
5.2.5. Gender Equality, Equity & Disability ..................................................... 61
5.3. Efficiency ........................................................................................................ 64
[viii]
5.4. Sustainability .................................................................................................. 68
5.5. Unforeseen Events ......................................................................................... 71
5.6. Human Rights Based Approach ..................................................................... 72
5.7. Partnership and Financial Management ......................................................... 77
5.8. Unexpected Findings ...................................................................................... 80
6. CONCLUSIONS & LESSONS LEARNED ........................................................ 81
7. RECOMMENDATIONS .................................................................................... 83
List of Tables Table 1: Programme Coverage in Terms of Beneficiaries per Area/Province. ........... 4
Table 2: SPSP Theory of Change ............................................................................ 11
Table 3: Evaluation Criteria and Core Questions ..................................................... 14
Table 4: Sampling Calculations for Post KAP Household Survey ............................ 17
Table 5: Summary Table for Instrument, Source and Limitations ............................. 19
Table 6: Evaluation Constraints, Limitations and Mitigation ..................................... 21
Table 7: Community Mobilisation Activities & Training ............................................. 36
Table 8: SLTS Training and Events ......................................................................... 36
Table 9: BCC Campaigns Outreach (Target versus Actual Number of People) ....... 37
Table 10: Supply Side Training Events for Masons and Entrepreneurs ................... 43
Table 11: Types of Low Cost Latrines ...................................................................... 44
Table 12: Number of Latrines Constructed (Target versus Actual) .......................... 45
Table 13: Outreach of Health & Hygiene Messages through Various Mediums ....... 50
Table 14: Capacity Building & Advocacy for Government ........................................ 53
Table 15: Villages Supported in attaining Total Sanitation Status ............................ 55
Table 16: Knowledge Management and Accountability Events and Activities .......... 57
Table 17: Cost Per ODF Declared Beneficiary ......................................................... 66
Table 18: Cost per ODF Declared and Verified Village ............................................ 67
Table 19: Evaluation Recommendations .................................................................. 88
[ix]
List of Figures Figure 1: SPSP Rural Districts and Agencies ............................................................. 4 Figure 2: Budgetary Trends for Water & Sanitation in Pakistan Source: Water Sector Report 2011 ............................................................................................................... 7 Figure 3: Evaluation Data Collection Methods ......................................................... 16 Figure 4: SPSP Contribution to Sanitation Target .................................................... 25 Figure 5: Province Based Access to Sanitation Facility ........................................... 29 Figure 6: Number of Sources of Health & Hygiene Messages ................................. 37
Figure 7: Health & Hygiene Awareness Messages .................................................. 38 Figure 8: Health and Hygiene Messages that the Respondents Remembered ........ 38 Figure 9: Respondents Able to Recall Three Key Hygiene Messages ..................... 38 Figure 10: Source of Health & Hygiene Messages in Schools ................................. 39 Figure 11: Topics Covered during Health & Hygiene Sessions in Schools .............. 39
Figure 12: Awareness of Hazards of Open Defecation ............................................ 40
Figure 13: Awareness of the Need for Achieving ODF Status .................................. 40 Figure 14: Proportion of Population Practicing Open Defecation ............................. 41
Figure 15: Willingness to Construct a Latrine (including methods of cost bearing) .. 41 Figure 16: WASH Facilities Constructed with External Support ............................... 46 Figure 17: Latrines Constructed By Respondent over Last Year and Type of Support Received .................................................................................................................. 46
Figure 18: Availability of Sanitation Supplies and Trained Masons .......................... 47 Figure 19: Population Using an Improved Drinking Water Source ........................... 47
Figure 20: School WASH Facilities .......................................................................... 48 Figure 21: Availability of WASH Facilities in Boys, Girls and Co-Education Schools 49 Figure 22: Household Water Treatment and Safe Water Storage ............................ 50
Figure 23: Proportion of Respondents who Treat Water .......................................... 50 Figure 24: Hand Washing Place & Soap Availability ................................................ 51
Figure 25: Critical Moments for Hand Washing ........................................................ 51
Figure 26: Knowledge of Water Related Diseases and Incidence of Diarrhea ......... 52
Figure 27: Government Expenditure on WASH ........................................................ 54 Figure 28: Households Using Underground Drainage .............................................. 55 Figure 29: Gender, Disability and Equity Based Access to Latrines ......................... 62 Figure 30: School Facilities Attributes ...................................................................... 75
[x]
List of Annexes Annex 1: Terms of Reference – Evaluation of SPSP Phase 1 Annex 2: Illustration of SPSP Programme Components Annex 3: SPSP Rural Results Framework Annex 4: Stakeholders & Roles Annex 5: Evaluation Matrix Annex 6: Key Indicators and Source of Information Annex 7: List of People Met for SPSP Evaluation 2014 Annex 8: List of Participants in Provincial Bottleneck Workshops Annex 9: List of Villages for Post KAP - SPSP Final Evaluation Annex 10: List of Documents Reviewed Annex 11: Evaluation Toolkit Qualitative Instruments Annex 12: Evaluation Toolkit Quantitative Instruments Annex 13: SPSP Outputs - Target versus Actual Annex 14: TORs for the Evaluation Reference Group Note: Consolidated version of all annexes is attached as Volume II.
[xi]
1. EXECUTIVE SUMMARY
Evaluation Object ‘Sanitation Programme at Scale in Pakistan - Rural’ (SPSP Rural)’ is a multiyear (2013-17) programme with first phase implemented through 2013-14. The programme targets rural parts and particularly the flood and security affected (insecure) areas. It builds on the experiences and learning drawn from UNICEF’s Post Floods Recovery (2010) Programme i.e. ‘Rural Sanitation in Flood Affected Districts’ (RuSFAD) implemented through 2010-12. The programme aims that; ‘By 2017, most vulnerable and marginalized children and women will be using safe drinking water, improved sanitation and hygiene practices’. The SPSP Phase I (the Year I) intended to reach out to 1.3 million people by implementing the Pakistan Approach to Total Sanitation (PATS)1. The PATS combines the School and Community Led Total Sanitation (SLTS & CLTS) approaches (referred to as Community Approaches to Total Sanitation (CATS) by UNICEF). The programme consists of six interlinked components, which include: i) Demand for sanitation; ii) Supply facilitation; iii) Health and hygiene promotion; iv) Linkages with duty bearers; v) Attaining total sanitation and; vi) Knowledge management. The programme design and delivery involved multiple stakeholders. UNICEF is the lead agency with contributions to programme design, resource mobilization, contracting the implementing partners (national and international non-profit organizations) and technical oversight. The field implementation has been carried out by non-profit partner agencies, and been referred to as ‘Implementing Partners (IPs)’. The programme has leveraged partnership with government agencies (mainly Public Health Engineering and Local Government Depts.) for some component. The field monitoring functions were performed by a ‘Third Party Monitors (TPM)’ in assistance to UNICEF. Moreover, the research and capacity building functions have been undertaken by the Development Solutions International (DSI - a consulting company) and Rural Support Programme Network. The programme activities have been carried out between February 2013 and March 2014. The Phase I had been rolled out with major funding of US $ 13.69 million, contributed by DFID UK. This phase was implemented in thirteen (13) districts and agencies i.e. 10 districts in Punjab, Sindh, Khyber Pakhtunkhwa, Balochistan and 3 Agencies from the Federally Administered Tribal Areas (FATA).
Evaluation Purpose, Objectives and Scope
The evaluation was commissioned by UNICEF Pakistan with the purpose to assess the programme achievements and to use the findings to inform the next phase. Objectives as stated in the TORs include, determining the relevance, effectiveness, efficiency and sustainability (as per UNEG standards), whilst also assessing the extent to which UNICEF core programme strategies and normative principles
1 Pakistan Approach to Total Sanitation (PATS) is a country specific model that evolved from Community Approaches to Total Sanitation (CATS), through a consultative process during 2008-09.
[xii]
(Human Rights Based Approach, Impact of Unforeseen Events and Financial Risk Management) were met. The evaluation has been carried for the activities undertaken in 13 districts and (FATA) agencies during Phase I (2013-14). The primary audiences or users of the evaluation include DFID, UNICEF, Implementing Partners (IPs) and government stakeholders. The secondary audiences include WASH sector partners and beneficiary communities.
Evaluation Methodology The evaluation methodology was evolved with a considered view of this being a formative evaluation and a clear focus on documenting key learning coupled with identifying areas for improvement. The intent was to have the next phase informed of the opportunities to improve both the design and delivery. The evaluators used mixed method approach for the evaluation. To map the extent of change for key quantitative output/result indicators, quasi-experimental research design was used to collect the information and hence draw comparison with baseline results. For this, a representative Post-KAP survey was administered in programme areas. The evaluation design however, excludes the creation of counterfactual; primarily for being formative evaluation rather an impact. The other methods used include secondary sources review, in-depth interviews, focus group discussions, bottleneck analysis (workshops) and observation checklists. For the evaluation design and implementation, the consultants followed UNICEF adapted United Nations Evaluation Guidelines (UNEG), accredited researching norms and ethics. An Evaluation Reference Group (ERG) with representatives from UNICEF, DFID, and government departments, which provided evaluation oversight and shared their own experiences with the evaluators. The report comprises of two volumes, one being the report itself and second includes all annexes.
Key Findings and Conclusions Relevance: The section carries findings and commentary on the level of coherence (consistency) of programme objectives and approaches to national and provincial policies and priorities, needs of men and women beneficiaries (including vulnerable) and total sanitation approaches. Also level of integration for UNICEF core strategies or crosscutting priorities i.e. gender equality, equity, and disability in programme design and implementation. By and large, the SPSP Phase 1 design was found coherent and consistent with the national policies and development commitments. The geographic prioritization seems appropriate, for the fact that most of regions picked up for programme implementation showed having lower sanitation coverage than the national average. The design carries evident departures from the conventional approaches like Community Approach to Total Sanitation (CATS), such as provision of subsidies (for latrine construction) following the ‘Disaster Response Approach’ within PATS which allows provision of Humanitarian assistance in post disaster/insecurity environment.
[xiii]
The design by and large, offers relevance to the UNICEF core strategies where it entails expressed focus on equity (subsidy to reach the extremely vulnerable and vulnerable families). Moreover, balancing gender consideration at all levels such as staffing, beneficiaries, forming men and women VSCs, hiring CRPs and in for some interventions positively discriminates to favour the marginalized like prioritising girls schools for rehabilitation of WASH services. In comparison to the equity and gender equability, the focus on disability was relatively less as evident from non-inclusive latrine design for rehabilitated WASH facilities (at schools and health outlets).
Effectiveness: The effectiveness of the programme has been assessed with respect to achieving outputs/results, actual application of the proposed theory of change. Each component has been assessed individually and all been assessed collectively also. Moreover, the section carries commentary on programme effectiveness with respect to implementing the integration of core strategies or cross cutting priorities i.e. gender equality, equity, & disability. The programme has performed well as the results indicate implementers being successful in achieving or in most exceeding the set (output) targets. The indirect implementation approach, where IPs were contracted, appears to have worked well to achieve targets in short time span. The limited engagement of public sector in programme delivery appears as a weak link. This has not helped much with preparing public agencies for PATS replication and scale-up. The partners and the partnerships for action research, third party monitoring and centralized training components, appear effective. This is evident form the fact that these partners been able to design and implement a parallel programme with different delivery approach, gather and document useful field learning (evaluators were told that the document is work in progress), facilitate timely information collection to inform management decision making and preparing and mentoring IPs staff respectively, to help achieve the set targets. The policy advocacy, for which UNICEF has been lead, has shown some positive results. These include selective roll-out of PATS in Punjab, WASH Master Plans, progress with respect to provincial sanitation policies, substantial sanitation budgetary increases for FATA and others. There was no evidence to suggest if it happened only for UNICEF advocacy efforts. There are multiple other WASH stakeholders involved in public advocacy, though with limited coordination and this could be attributed as a result of collective efforts. The strategies evolved for community mobilization Behaviour Change Communication (BCC) were found to be most effective in organizing, educating and sensitizing beneficiaries. For instance, 41% more people shared to have received health and hygiene messages than the baseline numbers. Similarly, proportions of those claiming to have adopted improved hygiene practices increased by 39% (compared to baseline). Receptiveness to the messages as well as ability to recall (including practice) has shown significant increase. In programme areas, 65% population now has access to sanitation facilities, significantly higher than the national average and two per cent short of the MDG sanitation targets.
[xiv]
Largely, the demand side activities found to be more effective than the supply side interventions. The programme may need to revisit the subsidy element (in supply side) to address equity and vulnerability. The parallel use of SLTS and CLTS helped in leveraging the complementarities and proved effective.
Efficiency: This section explores and shares evaluators’ views on how well the various activities have transformed the available resources into the intended results or outputs, including quantity, quality and timeliness. The programme implementation was found efficient especially where all (significant) performance targets have been met, despite delayed start. However, the fact that implementers were able to exceed most of the output indicators raises questions on whether realistic targets were set and corresponding resource calculations were appropriate. The section carries efficiency calculations mainly drawn through statistical analysis for cost per beneficiary and village for raising awareness and help achieving the ODF status. The analysis offers comparison with regional countries. Some inefficiencies were noted with contracting arrangements for hardware support in the programme i.e. rehabilitation of WASH facilities. The readers are advised to take caution while reading the efficiency analysis, as this has been carried out using budgeted figures, as consolidated expenditure statements were not available by the time of analysis.
Sustainability: The section carries findings and commentary around extent to which various stakeholders including the public sector and civil society are likely to sustain the positive outcomes of the project after the end of phase 1. The programme had envisaged using SLTS (school teachers and students) and CRPs as potential agents for sustaining the momentum. This however, found not to have been translated into an exit strategy or plan across sites and partners. The IPs found no or little time for post ODF monitoring and evaluators could take note of slippage signs. The linkages with duty bearers’ component could achieve little in terms of fostering ownership amongst public agencies to either takeover responsibility and/or scale-up the efforts. The community structures and mechanisms, such as CRPs and VSCs are likely to sustain but with limited success. The sustainability of sanitation mart owners as well as entrepreneurs created/promoted during the programme is also uncertain due to lack of access to loans/finance as well as demand issues. The evaluators are of the view that the SLTS component without continued support and engagement of district education offices is less likely to sustain either. The rehabilitated WASH facilities at schools and health outlets may not sustain without proper O&M mechanisms. The changes brought in the knowledge, attitudes and behaviours amongst men, women and children, are the most likely and significant results that may sustain.
[xv]
Gender Equality, Equity & Disability: This section carries commentary on how well the programme has implemented the integration of cross cutting priorities for gender equality, equity and disability. The design and PCA largely refer to balancing gender considerations in terms of boys/girls schools, men and women CRPs and VSCs, as well as IP field staff. The BCC campaigns found responsive to the information needs and preferred mediums of men, women and children. The programme also considered women and girls privacy and safety aspects (though the elevated WASH facility design has drawn criticism with respect to privacy). However, there were evident gaps around formal gender analysis before undertaking field interventions as to assess if interventions may not add to workload (burden) on the rural women and possibly children for additional requirement of water. This is significant in the context where gender roles place water fetching, cleaning of latrines as the domain of women and children. The programme demonstrated the equity focus by using PRA techniques to identify the vulnerable and extremely vulnerable and then extending household subsidy for 10% beneficiaries. However, the household subsidy component found to be challenging both in terms of design and implementation (creation of dependence in few cases and community friction at identification stage etc). It merits a serious review and revision. Addressing of disability, however, remained weaker both at community as well as school level in terms of sensitisation, infrastructure designs and sanitation supplies. Overall in terms of access to sanitation facilities, results of the post-KAP survey confirm an increase for women, as well as people in the lowest income quintile.
Unforeseen Events: PCAs in general seemed to lack any planning or comprehensive risk assessment of the programme areas and overall security environment. Fortunately, nothing significant happened (natural and human induced disaster of noticeable scale) during the course of implementation carrying potential to have had derailed the programme implementation. IPs knowledge of local communities and in most cases having local staff helped in ensuring smooth running of activities. However, given the environment within which the programme is implemented, there is a need to have a risk/disaster strategy in place with clear action plan, as part of the PCAs.
Partnership & Financial Risk: Largely, the partners’ selection was found to be appropriate with some exceptions (elaborated in detail in main report), However, TPM reports were able to identify operational and programme delivery challenges with the umbrella arrangement in Sindh. Financial risks were mitigated through some basic level capacity building. UNICEF did well to have conducted a baseline of partners’ capacities and developing plans, however it appears that this was not followed up to facilitate partners capacity development. Similarly, the programme seemed to have achieved limited results with building capacities of the relevant government departments. At provincial level, there was disconnect with duty bearers in terms of taking them on board, engaging them in planning to execution level activities and introducing rewards or incentives through the departments.
[xvi]
Lessons The key lessons learnt are outlined below; a) Access to Sanitation Facilities and Reduction in Violence Against Women. The evaluation has contributed to unravelling a critical link between open defecation and extreme cases of gender violence in rural Pakistan. A significant number of respondents shared that the availability of latrines inside the house, has significantly reduced the incidences of extreme gender violence such as Karo Kari (honour killing). To them, the construction of latrines have reduced women’s need to step out (as for defecation they were coming out at odd hours), thus the protection risk (or probability) of being accused of illicit relations (with men), and consequently being subjected to extreme violence. The lesson that is derived from this instance is that though an indirect result of WASH intervention, the benefit of contributing to a reduction in violence is a substantially important aspect and should inform the future programme design and delivery. b) Shame, Shock and Disgust versus Pride, Self-Respect and Dignity The two approaches generally are not used together for triggering sanitation, however, SPSP Rural I demonstrated that by using them intelligently (where shame, shock and disgust follows the other), better results could be achieved. c) Business Incentive and Opportunity Is a Larger Motive for Response. DSI’s action research suggests that manufacturers, entrepreneurs and mart owners (with business risk bearing attributes) see supply to the underserved areas as an opportunity to expand their business further. Given purely business incentives, they are more likely to sustain sanitation supply in order to meet the heightened demand and maintenance requirements in communities. Hence, it could be argued that the programme may benefit more, if it may focus on identifying and working with manufactures/entrepreneurs, with a clear intent to help them realise the real market potential and opportunity offered by these underserved areas for business continuation and expansion. This in the long run shall contribute to ensure technological innovation, continued and sustained supplies facilitation. d) Using Subsidy to Address Equity Can Lead to Dependency To a degree the subsidies, allowed in PATS in form of humanitarian assistance in post disaster/insecurity affected community, have served the equity purpose. However, the subsidy planning and delivery mechanisms have had some inherent challenges, which need deeper thinking and innovative solutions. The key problems that need to be addressed are;
a) Subsidies reducing the willingness of other poor families to put their own very limited resources into latrine facility construction, given the possibility of subsidy may become available to them, thus acting as a detrimental incentive.
b) By the end of the project, in few cases, those who have not received this subsidy, remain dependent and wishful for similar other interventions.
[xvii]
Main Recommendations Following are key recommendations across design, implementation and policy areas; 1) Future programming should provide adequate time for all stages of the
programme i.e. inception phase, PATS implementation phase and exit phase. This should include defining the duration it takes for a standard PATS implementation cycle.
2) The programme Theory of Change needs to be articulated with clear linkages, pre-requisites, risks and assumptions. Similarly a sustainability strategy, focusing on sustaining ODF status should be developed. PCAs should include specific IP exit plans, and stakeholders who will be ultimately responsible for sustainability should be clearly informed of their role along with their consent to delivery it (community as well as role expected from Government Departments).
3) Following aspects are recommended for the supply facilitation component, a) Sanitation Mart Owner/Entrepreneur Distinction between Sanitation Mart Owners and Entrepreneurs should be reassessed and if required dropped. The DSI action research points to the fact that market forces or business opportunity works well with sustained supply of materials, given, those in the business are made to realize that how CLTS and SLTS approaches can contribute to increasing demand for sanitation. This may need to be focused in the next cycles. b) Household Subsidy Modality and intended benefits of providing a household level subsidy should be reassessed, including possible detrimental effects on the community. Other mechanisms should be tested (such as providing collective incentives at the community level). For equity purposes and the needs of the vulnerable revolving credit/contributory fund can be considered as an option to be managed by the VSC.
4) The future programme design must clearly articulate and provide adequate implementation guidance for integration of gender equality, equity and vulnerability including addressing the needs of the disabled. UNICEF may need to take lead to provide guidance to the IPs as to how should these be incorporated into future PCAs. Also, provide adequate guidance for implementing those with the aim to achieve provision of inclusive WASH services.
5) UNICEF may need to provide greater guidance on engaging with public stakeholders (preferably at provincial levels) to create linkages, find sustainable exit (where responsibility to sustain the change shifted to relevant public authorities).
6) UNICEF may need to work more on harmonization of sector partner efforts and advocacy initiatives. This could be best delivered by providing strategic support to relevant public authorities (provincial level in particular) to help them realize the potential and leverage comparative strengths and resources of different WASH sector partners for shared goals.
7) In SPSP Phase II, UNICEF should consider piloting PATS implementation with public stakeholders in at least one province, receptive and willing to contribute to the idea.
[1]
2. OBJECT OF THE EVALUATION
2.1. Programme Overview ‘Sanitation Programme at Scale in Pakistan - Rural’ (SPSP)’ was launched by UNICEF in 2012. It is a multi-year national sanitation programme focusing on rural parts of flood and insecurity affected areas (2013-17). It builds on lessons and experiences of UNICEF’s ‘Rural Sanitation in Flood Affected Districts’ (RuSFAD) programme (2010-12). Purpose The purpose of the programme as stated in the project documents is; ‘By 2017, most vulnerable and marginalized children and women will be using safe drinking water, improved sanitation and hygiene practices’. The aim of the programme is to reach 1.3 million people by the end of year one or first phase2. The programme designed is built on the principles of Community Approaches to Total Sanitation (CATS), which with slight adaptations for Pakistan is referred to as Pakistan Approach to Total Sanitation (PATS). The PATS model advocates use of both School Led and Community Led Total Sanitation (SLTS & CLTS) approaches, and has been applied in parallel, for the programme under evaluation. Components and Indicators There are six components or pillars in the programme design. In the results matrix, these have been equated with outputs (with 18 output indicators) contributing to the programme result or outcome statement. The outcome has 3 indicators to be realized by 2017. The six components/outputs are as follows:
1. Demand for sanitation to achieve open defecation free status created in targeted communities
2. Sanitation demand sustained with supply side interventions 3. Improved Health and Hygiene behaviours promoted with the target
community 4. Linkages developed with duty bearers/public institutions to support PATS 5. Villages supported in attaining Total Sanitation status 6. PATS programme knowledge management and accountability Framework
implemented The outcome indicators as appear in the results framework are as below (result framework attached as Annex 3):
1. Proportion of the population using an improved drinking water source increases to 94% in rural areas.
2. Proportion of population practicing open defecation reduced to 35% in rural areas.
3. Government expenditure on WASH as a proportion of total government expenditure increases by 1%.
2. The terms SPSP Year 1 and SPSP Phase 1 have been used interchangeably, however both imply the implementation period February 2013 to March 2014. The evaluation covers the programme activities implemented through this period.
[2]
The overall programme outcome is to be achieved through phased implementation of yearlong or more implementation rounds. Hence, this is the evaluation of activities undertaken during year 1 (implemented during 2013-14) and excludes progress assessment on some of the impact/outcome level indicators. This remains a formative evaluation for UNICEF with the clear focus on learning documentation to inform the programme design and implementation for follow-up phases for the period of 2014-17. Stakeholders The programme design and delivery involved multiple stakeholders ranging from relevant public agencies to civil society organizations, research & monitoring organizations to leading capacity development networks (refer to Annex 4 for detailed breakdown of stakeholders and respective roles). Significant public agencies involved in the programme (first phase) include: Climate Change Division (CCD, formerly the Ministry of Climate Change) at the federal level, and public health engineering (PHED), health, education, and local government (LG) departments at provincial and district levels. The programme design offers a clear delineation of roles between stakeholders including coordination arrangements. For instance, the programme envisaged CCD to facilitate public sector coordination, oversight and provide policy guidance with respect to programme coherence with PATS principles and guidelines. The public stakeholders at provincial and district levels such as health, PHED, LG, education were expected to provide field implementation oversight including implementation of selective hardware/engineering components (identification of schools for schools sanitation interventions, planning and execution of water schemes rehabilitation), distribute and contribute to rewards and provide ODF certification. UNICEF contracted Development Solutions International (DSI) Pakistan, a marketing research and training enterprise to develop and implement (on pilot scale) a Sector Mechanism for Sanitation Marketing (SMSM). Moreover, DSI provided technical support for latrine designs. The Rural Support Programme Network (RSPN), a leading national network for technical advice and capacity development took lead in training and mentoring related capacity development of the Implementing Partners (IPs), Community Resource Persons (CRPs), government departments and officials on PRA, community mobilization methods, PATS/CATS approaches including IEC material. APEX, a consulting company was contracted for ‘Third Party Monitoring (also referred to as TPM) services. The role included developing a complete process and output driven M&E system, assisting UNICEF in undertaking field monitoring at all project locations and providing performance assessment reports for management decision making. Knowledge management was one of the key functions to be performed by TPM.
[3]
The non-profit organizations, contracted as implementing partners (IPs) took lead in field implementation3. These included national and international NGOs, with portfolio of WASH projects in designated geographic regions. UNICEF’s Role & Contribution The programme envisaged multiple roles for UNICEF mainly technical & operational oversight and backstopping, resource mobilization, stakeholders’ coordination, sub-contracting and contract management, advocacy and knowledge management. The following description elaborates on specific UNICEF contributions. Programmatic Redesign & Management of TPM: This included redesigning of the integrated sanitation model and programme implementation framework incorporating learning and recommendations from the RuSFAD programme. In addition to this UNICEF also contracted and managed an Independent third party monitoring system within the MoRES framework. Capacity Building: This involved building a cadre of PATS master trainers in all the provinces, while also liaising with and building capacity of government departments to implement and prioritize PATS. Policy Advocacy & Integration into Provincial WASH Master Plans: UNICEF provided policy oversight and contributed to decisions ensuring that the Provincial Government’s Master plans incorporated PATS programme implementation. Knowledge Management: Activities included preparing PATS training documentary, commissioning thematic issue papers, action research by DSI, Menstrual Hygiene Management (MHM) research and WASH Equity study, Development of PATS training manual for master trainers and guidance booklets for CRPs, SOs, School teachers, WASH clubs, Masons, entrepreneurs and Village Sanitation Committees (VSCs) Financial: This included capacity building of IPs and Government counterparts, as well as focusing on lowering third party independent monitoring costs to 3% of overall programme cost. Duration & Resources The evaluated programme is part of the UNICEF’s medium term rural sanitation programme planned for 2013-17. The phase evaluated commenced in February 2013 and activities were completed by 31 March 2014. The allocated budget for the first phase (2013-14) was US $ 13.69 million. Funds were provided by Department for International Development (DFID – UK) from two humanitarian funding streams; one for assisting flood affected areas while other targeted people in security (militancy) affected areas.
3 Muslim Aid, Qatar Charity, TVO (Trust for Voluntary Organization), BRSP (Baluchistan Rural Support Programme), WDCO (Women Development & Community Organization), NCBP (Nations Capacity Building Programme), NRSP (National Rural Support Programme), SAFWCO (Sindh Agricultural and Forestry Workers Coordinating Organization), Sabawoon (Sindh Agricultural and Forestry Workers Coordinating Organization), SEED (Social Efforts for Education & Development), IRSP (Integrated Regional Support Program), SRSP (Sarhad Rural Support Programme)
[4]
Implementation Status The activities for the first phase (2013-14) were completed on 31 March 2014. However, the government partners in KP province and FATA regions, mandated to restore/rehabilitate water schemes, were given a no-cost extension until end of July 2014 to facilitate complete utilization of funds. Coverage During Phase 1 of the programme, SPSP was implemented in 13 districts and agencies (10 districts in Punjab, Sindh, Khyber Pakhtunkhwa and 3 Federally Administered Tribal Area Agencies - FATA). Find below the details of programme coverage in terms of geographic spread with corresponding beneficiaries (intended) in different regions. Province/Area District/Agency Number of Beneficiaries
Flood Affected
Punjab Sindh Baluchistan
Dera Ghazi Khan, Rajanpur Jacobabad, Kashmore, Shikarpur Jaffarabad, Naseerabad
680,000
Security Affected
Khyber Pakhtunkhwa FATA
Kohat, Tank, DI Khan Kurram , Mohmand and Bajur
620,000
Table 1: Programme Coverage in Terms of Beneficiaries per Area/Province.
The following map (Figure 01) illustrates the geographic coverage of the programme.
Figure 1: SPSP Rural Districts and Agencies
[5]
2.2. Country Context Pakistan has an estimated population of close to 180 million, of which almost 1/3rd population lives below the poverty line. With a population growth rate of more than 2.9% a year, every year an additional 4.0 million people4 are added; needing additional clean water and sanitation facilities. This has long been established that lack of access to clean water and sanitation facilities carries wide variety of impacts. The data and evidence to establish the extent of the burden imposed on the people of Pakistan has been limited, which to a degree contributed to lower prioritization and consequently resource allocations to water and sanitation in public services delivery. Moreover factors such as insecurity leading to internal displacement, recurrent/annual floods and cyclic pattern of disasters have further aggravated both the access and quality issues. It is therefore not surprising that Pakistan lags behind its committed MDG (relating to sustainable access to safe drinking water and basic sanitation, MDG 7-C) target for sanitation outreach and access to 67% of population by 2015. The Joint Monitoring Programme (JMP) 2012 report indicates that only 48% people are using improved sanitation (72% urban and 34% rural), while 23% are still practicing open defecation. Moreover, poor sanitary conditions remain one of the major contributing factors to modest progress on key health indicators. Diarrhoea caused by these conditions is one of the leading causes of mortality for children under 5. Children also suffer disproportionately from sickness due to diarrheal diseases with almost 25 million cases (PSLM 2006-7) reported annually. UNICEF’s ‘Investing Wisely, Sanitation & Water, Saving Lives’ report5 indicates that between 60 to 75 million people are affected by diarrheal diseases annually, which costs the country 3.94% of gross domestic product (GDP) in terms of health expenditure. Another report (DFID 2013) states that, it is diarrhoea that is causing severe malnutrition (amongst 5% at least) and moderate malnutrition (amongst 20%) of the children under five, which places Pakistan behind Sub Saharan countries e.g. Ghana (DFID 2013). In addition to health concerns, lack of water and sanitation services is also associated with socio-economic disadvantages. A recent DFID study6 points out that evidently lack of availability of water, sanitation and latrine facilities can create gender gap at homes as well as significantly decrease girls’ attendance at schools. The social analysis of WASH illuminates gender and age disparities, limited community engagement in services delivery. For instance, within gendered domains, mostly women and sometimes children are tasked to fetch water from nearest water point. The RuSFAD programme evaluation raised concern around added workload for women for houses that constructed pour-flush latrines. Similarly, where open defecation is a norm, the standards for observing privacy are relatively strict for women. Hence, left with the option to go out in the dark, which again poses both physical and protection risks. Children and elderly people are often excused for age and mobility considerations. During field evaluation, some communities even
4 WSP: Economics Impacts of Inadequate Sanitation in Pakistan 5 Investing Wisely Sanitation and Water Saving Lives, Pakistan Sector Status Report 2012, UNICEF
6 Water, Sanitation and Hygiene, Evidence Paper, DFID 2013
[6]
Price for Poor Sanitation: 116,013 children (under five) die due to diarrhoea each year in Pakistan – approximately 13 deaths per hour - many of these deaths can be prevented by adequate sanitation, safe drinking water and improved hygiene (Pakistan Demographic Health Survey (PDHS) 2007-8). In Pakistan poor sanitation and hygiene accounts for an annual loss of PKR 343.7 billion. (Pakistan Sector Status Report, 2012). “Pakistan is one of the few fateful countries in the world that have witnessed large-scale outbreak of cholera and typhoid fever”. (Mukhtar et al as quoted in DFID 2013).
referred to men preferring open defecation despite having latrines at home, primarily due to the embarrassment they felt of sharing the same space with family women. In terms of involving users or communities in services such as design and management, very few examples are available where public agencies may have successfully engaged communities for design and management of water points/services. The policy initiatives undertaken in the past, have largely failed to translate into successful examples or models for limited technical capacities within public agencies for community mobilization, sustained engagement and unwilling users to contribute (financially) to the operations and maintenance of water services. Through the last decade, the prioritization of water and sanitation sector has led to steady increases in the budgetary allocations (refer Figure 2), a trend noted across all provinces. However, unfortunately increased financing has not translated into improved service delivery for multiple reasons. Some of those include under-utilization for delayed releases, budgetary cuts and diversions. The joint budgeting/allocation, as is practiced in most provinces, is skewed in favour of water in relation to sanitation, this is due to absence of need based resource allocation by the relevant government agencies, political influence and lack of credible data on district wise sanitation coverage. Moreover, the anecdotal accounts from the field indicate steady increase in latrine construction costs (on average between US$ 50-100), which in a way discourages people, more so the poor, to invest in constructing latrines. Those at risk to economic shocks (for natural disasters and insecurity) are also reluctant to invest in latrines. As for the groups who could afford, latrine construction rests at a much lower priority compared to food, clothing or entertainment. On a positive note, the last decade has seen some encouraging public policy shifts. For instance, formulation and approval of the National Sanitation Policy (2006) set the goal to reduce the number of people without sustainable access to improved sanitation by 2015, and provide improved sanitation to 100% population by 2025. The policy, while referring to strategic areas of intervention, mentions application of Community Based/Led Total Sanitation (CLTS) approaches to achieve desired results. Similarly Balochistan following on the line of the National Policy approved their Provincial Sanitation Policy in 2008. In order to contextualize communities' involvement in sanitation, a Pakistan-specific model of Community Approaches to Total Sanitation (CATS) was evolved, i.e., ‘Pakistan Approach to Total Sanitation (PATS), through a consultative process during 2008-09. The PATS, though an evolution of CLTS entails additional features than traditional CLTS approaches; for instance, introduction of sanitation marketing,
[7]
component sharing (public-private partnership) and others. PATS approach has a geographic focus on peri-urban and rural areas. It aims to end the practice of open defecation while transforming hygiene behaviour. PATS has also be endorsed in provincial sanitation policies by both Punjab and KP respectively. Another policy milestone achieved was the National Drinking Water Policy (NDWP 2009), which acknowledges water as a fundamental right of every citizen and aims ‘to improve the quality of peoples’ lives by reducing water-borne diseases with the provision of safe and sustainable drinking water by 2025 at affordable cost’. In last few years, the public sector spending on water and sanitation has seen a significant increase, which also reflects prioritization of water and sanitation on the public policy agenda (Pakistan Sector Status Report, 2012).
Figure 2: Budgetary Trends for Water & Sanitation in Pakistan Source: Water Sector Report 2011
The 18th Constitutional Amendment & 7th National Finance Commission Award (NFC) stand out as major developments, offering huge potential for devolved planning and management of civic services including water and sanitation. This shall give provinces greater access to resources and mandate to make their own specific policies and action plans. It offers the opportunity in terms of greater ownership at local level and drawing need based and context specific plans and interventions. However, the dividends of positive policy shifts are yet not visible on the ground in terms of improved service delivery. The dismal situation of sanitation and clean access to water (as highlighted above) was further exacerbated in Pakistan, due to the repeated floods during 2010, 2011 and 2012, which affected at least 20 million people in the country. Some have been affected repeatedly. According to the National Disaster Management Authority (NDMA) Pakistan, the heavy monsoon flooding in 2012, affected an estimated 4.8 million people in the provinces of Punjab, Baluchistan and Sindh. According to a Multi Cluster Rapid Needs Assessment (MIRA) an estimated 1.4 million children (of which 392,000 under five) were directly affected by the floods. These floods have resulted in contamination of wells and other sources of drinking water, further compromising the sanitation and hygiene environment. This has contributed to visible surge in water, sanitation and hygiene related diseases such as diarrhoea. Moreover, the insecure environment along the western borders and security operations in the Federally Administered Tribal Areas (FATA) has resulted in mass displacement across the region. Millions have been displaced since 2008. The authorities struggled to tend to the humanitarian needs of the displaced. The
[8]
displacement further strained the natural resource base and services delivery in host areas and communities. UNICEF conceptualised and rolled out SPSP to address the unmet sanitation needs in the rural parts of the insecurity and flood-affected areas.
2.3. Theory of Change The project documents shared for review did not include an explicitly defined ‘Theory of Change (TOC)’ for the programme7. However, as required the consultants in consultation with UNICEF and IPs have developed the TOC for the programme evaluated, derived from the SPSP flow chart of programme activities, results framework, details of project components and adaptation of the Global CATS TOC. The SPSP design is drawn on PATS framework with some adaptations. The SPSP design is drawn on a set of core assumptions that explain what changes should happen and why in order to lead to an ODF environment with improved health and hygiene practices and access to water. Moreover, there are assumptions, which explain how the envisaged changes happen and how they can be catalysed at each level (community, at local, provincial and national level). These constitute the building blocks of the overall envisaged/assumed change process. Based on the understanding of why, what and how changes may happen with respect to sanitation behaviours culminating in creation of Open Defecation Free (ODF) environment, the consultants have developed intervention logic. The logic includes strategies and activities that trigger the change process. The description below entails core assumptions and the two parts comprising a matrix and an illustration, which cumulatively explain the TOC. Core Assumptions The programme is based on the following core assumptions, with a logical flow between them from 1 to 5 (that is 2 logically follow 1, it does not represent hierarchy i.e. 1 is not more important than 2) describing the process of change: 1. Open defecation causes serious health and environmental concerns. It is an
individual and a collective community issue. Its practice should be abolished and a new social norm of ODF environment should be instituted.
2. The decision to end Open Defecation (OD) has to be collective, explicit and widespread, thus giving individual families the confidence that others are also abandoning the practice. When social pressure is transformed into collective commitment by the community to end the practice of open defecation, it can become self-sustaining. Once it reaches the tipping point, the change is more rapid and far-reaching.
3. Open defecation happens because of lack of awareness, resources and socio-cultural reasons. Even with awareness and resources, it may continue because of the socio-cultural acceptance within the community. This needs to be replaced and addressed through recognition of new behavioural expectations.
4. Due to socio-cultural and collective dimensions, the decision to end open defecation lies with the community. Ending open defecation is therefore a
7 UNICEF Pakistan WASH Security 2012 Proposal
UNICEF Pakistan WASH Flood 2012 Proposal
[9]
collective social change process led by informed and empowered communities that challenge and change pre-existing normative beliefs and attitudes.
5. A population wide shift to a new social norm of not continuing with open defecation requires an enabling environment. Firstly, a clear recognition of the hazards and consequences of open defecation to communities, who then consider it a practice not to be engaged in. Secondly, the enabling environment needs to provide for sustainability in terms of options (low cost options, drainage, supplies accessibility etc.) as well as connect with duty-bearers so that community is protected from future hazards of open defecation.
[10]
Activties
Outputs Outcomes Result (Impact)
Short Medium Long
- Formative research feeds into BCC .
- BCC campaigns launched. - SOs facilitate PATS in
communities and schools. - Wash Clubs formed and
strengthened. - Participatory Health and
Hygiene Promotion activities conducted.
- School WASH Facilities provided.
- CRPs trained and triggering CLTS.
- VSCs formed and their capacity built.
- Community Action Plans facilitated.
- School teachers and education officials trained on facilitating SLTS.
- Masons, Entrepreneurs, Sanitation Mart Owners Trained.
- Community informed on technical options.
- Demo Latrines installed - Water facilities rehabilitated and
built. - Linkages for accessing loans to
create household sanitation facilities.
- Capacity development and advocacy events with Duty Bearers.
- Advocacy for rewards by government for ODF communities.
- Advocacy with duty bearers to support Total Sanitation programmes.
- Mass Media BCC campaigns launched.
Demand for sanitation to achieve open defecation free status created in targeted communities.
Sanitation demand sustained with supply side interventions.
Improved Health and Hygiene behaviours promoted with the target community.
Linkages developed with duty bearers/public institutions to support PATS.
Villages supported in attaining Total Sanitation status.
PATS programme knowledge management and accountability Framework implemented.
At & Across Community Level
Community aware, mobilized and empowered to take action (through VSC), CRP & VSC formed, CAPs facilitated.
Community and School and engaged in BCC campaigns & attaining ODF status.
Low cost options, supplies and labour introduced.
ODF Certifications and Incentives awarded.
Demonstration of model ‘Total Sanitation’ villages.
Linkages with Duty Bearers.
Improvement in health condition and reduction in diarrheal diseases.
Safe water usage given households are treating water and storing safely.
At School Level
Health and Hygiene Promotion, formation of Wash Club, school taking responsibility to ensure ODF status in community.
Functional WASH facilities help maintain ODF status in school.
Schools Duty Bearers trained to take action.
Children, especially girls are regularly attending school.
At Provincial/District Level
Improved Provincial/District budget allocation to WASH specific basic social services.
PATS supported/implemented by the government.
Change in social norms. The decision to abandon open defecation has to be collective, explicit and widespread thus giving individual families the confidence that others are also abandoning the practice. When social pressure is transformed into collective commitment by the community to end the practice of open defecation it can become self-sustaining, once it reaches the tipping point, the change is more rapid and far-reaching and those who chose to practice OD are denounced. Social norm related to OD is addressed through recognition of new behavioural expectations as OD happens because of lack of awareness, resources and socio-cultural reasons. Even with awareness and resources it may continue because of the socio-cultural acceptance within the community. Due to socio-cultural and collective dimensions, the decision to end open defecation lies with the community. Ending open defecation is therefore a society level change in social norm led by informed and empowered communities.
A population wide shift to a new social norm of not practicing open defecation requires an enabling environment, firstly a clear recognition of the hazards and consequences of open defecation to communities, who then consider it a practice not to be engaged in. Secondly the enabling environment needs to be provided for sustainability in terms of options (low cost options, drainage, supplies accessibility etc.) as well as a connection with duty-bearers so that community is protected from future hazards of open defecation.
By 2017, most vulnerable and marginalized children and women are using safe drinking water, improved sanitation and hygiene practices
Open defecation causes serious health and environmental concerns. It is an individual as well as a collective community issue. Its practice is abolished and a new social norm of Open Defecation Free Environment is instituted. Access of targeted and neighbouring communities to safe drinking water, improved sanitation and hygienic practices, leading to conditions that do not cause diarrhoea and other water borne diseases. Sustainable ODF communities climbing up the sanitation ladder with duty bearers responding to infrastructure development needs through allocation of resources in an equitable manner.
SPSP Rural – Theory of Change – Part 1
[11]
Table 2: SPSP Theory of Change
Table 2: SPSP Theory of Change
SPSP Rural – Theory of Change – Part 2
Enabling Environment Assumptions and Risks
Changing Policy and Institutional Environment (18th Amendment, Local Government Elections, Provincial Policies and Programmes)
Proper Identification of Partner with capacity, outreach and ability to deliver with accountability.
Support at implementation level from the various departments of District and Sub-District Government.
Quality advice and facilitation, by UNICEF, IPs and involved experts.
Identification and effective use of natural leaders (whether in Government, Community (CRPs, VSC Member, Religious Clergy or Teachers etc.)
Proper sanctions (authority to work, NOCs, registration of organizations (IPs)) in place.
Key local community persons engaged through the process.
Availability of sound technical advice (UNICEF Senior Management, Subject Experts, 3rd Party Entities, Field level advice from IPs and input from Government (at provincial, district and sub-district level)).
Programme implementers with appropriate capacities to engage people in programme activities i.e. CRPs, Mason, Mart Owner, VSC Members, and Teachers etc.
Programmatically sound and locally viable interventions.
Effective financial controls and M&E arrangements in place (Financial management and reporting capacities of IPs, M&E arrangements for soft activities as well as technical activities).
Qualified staff and implementing partner’s identified/recruited/trained for the programme, having the requisite skills and knowledge.
Adequate availability and utilization of resources (both in terms of human resources as well as financial resources).
The programme implementation (its various phases community mobilization, triggering, BCC etc.) is not impeded by influential or other locals.
Community engages fully with the various programme interventions and stages.
Programme interventions are supported at collective as well as individual level, to ensure desired change occurs and necessary actions follow.
Programme based participatory techniques (whether through community, CRPs, VSC, government etc.) encourage and ensure individual and collective action.
Programme activities are not hindered by environmental and ecological constraints or issues, in which case alternate approach would have to be identified.
Resources for the programme remain accessible through the life time of the first phase, including financial as well as human resources.
Technical options and interventions are appropriate for the use and local context.
Security situation in few selected areas may delay achievement of results.
[12]
3. EVALUATION PURPOSE, OBJECTIVES AND SCOPE
Purpose This is formative evaluation with the purpose to assess programme design and delivery to inform the follow-up phases of SPSP. The evaluation is commissioned with the primary intent to accumulate learning, whereas accountability appears to be the secondary purpose. The evaluation TORs underlines assessment of programme achievements especially in terms of reaching out to the most vulnerable children and women. The assessment around choice to implement PATS with government and non-profit partners and integration of UNICEF programming priorities (normative principles) such as human rights and gender. The primary users or audiences of the evaluation include DFID, UNICEF, Implementing Partners (IPs) and government stakeholders. Each of these stakeholders has something to gain from the evaluation. For instance, the findings may facilitate UNICEF readjust the design and guide IPs with effective and efficient programme delivery. The government stakeholders may pick ideas of how PATS implementation could be adopted and up-scaled within public sector. WASH sector partners as secondary users could benefit in terms of picking up best practices for replication elsewhere. Objectives The evaluation TORs outline the specific objectives (refer to Annex 1 for complete TORs), which are reproduced below. These are:
1. Determining the relevance, effectiveness, efficiency and sustainability of UNICEF SPSP-Rural Phase I programme (as per UNEG standards).
2. Assessing the extent to which UNICEF core programme strategies were met. 3. Assessing how UNICEFs normative principles were reflected in the approach. 4. Identifying bottlenecks and success factors for programme implementation. 5. Commenting on the constraints and opportunities of programme
implementation. 6. Assessing effectiveness of the strategic partnership for process monitoring
and of the collaborative relationship for programme implementation (RSPN for Training).
7. Providing recommendations to strengthen PATS programme. 8. Assess the integration of UNICEF normative principles i.e. human rights
based approach, gender equity and environmental sustainability.
Scope The evaluation was commissioned by UNICEF Pakistan. The evaluation covered the SPSP implementation (activities undertaken) for the period of February 2013 – March 2014, hence could be referred to as SPSP Phase or Year 1 evaluation. As desired, the evaluation focused on determining the relevance, effectiveness, efficiency and sustainability of UNICEF SPSP-Rural Phase/Year I. The evaluation scope included assessment of all six key programme areas/outputs (explained in the introduction section) with more focus on areas where design had changed based on
[13]
learning from RuSFAD III evaluation (RuSFAD III preceded SPSP I roll-out and implemented in flood and polio affected regions). The evaluation scope included assessment of UNICEF’s choice for strategic partnership for process monitoring (TPM) including the collaborative relationship for programme implementation. Moreover, assessing the SPSP-Rural’ coherence (in design and implementation) to UNICEF core programme strategies i.e.
i) Capacity assessment and capacity development ii) Service delivery iii) Advocacy iv) Strategic partnership v) Knowledge management vi) Communication for development
The evaluation TORs prescribed the evaluation scope in terms of methods, which included:
i) Representative assessment of post KAPs (post KAP for communities and non-representative post KAP for schools) that helped assess the progress or extent of change on quantifiable indicators (thirteen outcome and output level indicators),
ii) Qualitative assessment of programme achievements and learning as per selected OECD-DAC criteria,
iii) Evaluation of capacity development of SPSP-rural non-governmental and governmental stakeholders,
iv) PATS related bottleneck analysis for each province. The geographic scope of the evaluation includes thirteen districts and agencies (where SPSP-Rural Phase I implemented) of Baluchistan, Punjab, Sindh, KP and FATA. The evaluators interacted with/consulted stakeholders at national, provincial district/Sub-district and community levels. Evaluation Framework The evaluation followed the UNICEF adapted United Nations Evaluation Guidelines (UNEG) norms, standards and ethical guidelines. The evaluation TORs outlined evaluation purpose, objectives, scope and criteria, broadly found consistent with OECD-DAC evaluation framework i.e. relevance, effectiveness, efficiency and sustainability. Assessment of impact was not a requirement given the short duration of project implementation during Phase 1 and the fact that the phase had recently ended. However, the evaluation included additional criteria including assessment of financial risks, integration of human rights and the impact of unforeseen events on programme delivery. Furthermore, evaluators assessed the gender equality and equity aspects, as crosscutting theme for the programme design and delivery across the other criteria. The element of financial risk assessment was added to evaluate UNICEF’s ability to enable IPs to demonstrate transparent and accountable management of programme funding. The aspect of human rights was assessed to give a measure of how design and delivery incorporated WASH as rights issue, given that Pakistan has ratified several international treaties, under which the provision of/access to safe drinking water and improved sanitation is guaranteed. It
[14]
was included to help assess the translation and application of HRBA approach in WASH services both by UNICEF and implementing partners8. Lastly, the impact of unforeseen events was assessed in view of repeated floods (in some areas) as well as the insecure environment. The evaluators are of the view that TORs could have been made more explicit in terms of defining evaluation questions and sub-questions. During inception phase of the evaluation, the consultants together with relevant UNICEF team members reviewed and revised the evaluation questions to enable more precision in terms of information needs for the evaluation. The evaluation matrix drawn describes the information needs, sources and methods for data collection (refer to Annex 5 – Evaluation Matrix for detail breakdown). The Annex 6 provides a complete list of key result indicators used for assessment of programme achievements (refer to Annex 6 – Key Indicators and Sources of Information). The following table outlines the core evaluation questions.
OECD-DAC Evaluation Criteria
Core Questions
Relevance To what extent the programme design and implementation was relevant to national/provincial priorities and needs of men and women beneficiaries in terms of achieving expected results across all socio-cultural groups including the vulnerable in the targeted communities?
Effectiveness To what extent were project implementation strategies effective and successful in achieving the planned outcomes/results for rich and poor men and women across all socio-cultural groups including the vulnerable in the targeted communities?
Efficiency To what extent have the outputs of the project been achieved (in terms of quality and quantity) with the allocated resources/inputs (such as funds, time, and procedures)?
Sustainability To what extent various stakeholders including the public sector and civil society are likely to sustain the behaviour change related to the goals of the programme after it has ended?
Note: Assessment of Impact was not required given the recent completion of project activities as well as the short duration of Phase 1.
Additional Evaluation Criteria
Core Questions
Unforeseen Events To what extent was the project implementation affected/influenced by unstable, insecurity or emergency situation?
Equity and Gender Equality
Though initially listed as separate criteria in the original TORs, these were included as cross cutting elements across the other evaluation criteria following UNICEF Team review and advice.
Human Rights Based Approach
To what extent did the project observe HRBA and/or contributed to observance of Human Rights in Pakistan as an implicit objective of the project?
Financial Risk Management
To what extent the project has observed and applied UNICEF’s standard financial management practices with Implementing Partners as well as with Government?
Table 3: Evaluation Criteria and Core Questions
8 These include the International Covenant on Economic, Social and Cultural Right (ICESCR), the Convention on the Elimination of All Forms of Discrimination against Women (CEDAW), and the Convention on the Rights of the Child (CRC).
[15]
The evaluation design took note of established research and evaluation principles and standards such as UNEG standards, norms and ethical guidelines. These standards have been complied with at varied levels such as design of evaluation framework, tools, training of staff, field data collection. For instance, requesting prior consent from respondents for research tools application, assuring data anonymity and others (more details in the methodology section).
4. METHODOLOGY
4.1. Evaluation Conceptual Framework The evaluation conceptual framework has been developed while keeping in view the formative nature of the evaluation, as outlined in the TORs. The evaluators used the quasi-experimental research design, where Pre and Post-KAP results have been compared to assess programme/approach effectiveness for quantitative result indicators. This was done with the view to assess effectiveness of PATS approach in terms of improved health and hygiene awareness, access to water and sanitation and use of improved/unimproved sanitation. However, keeping in view the formative nature of the evaluation, the quasi-experimental design was chosen with one exception i.e. exclusion of control group. Hence, the readers may not find any reference to creation of counterfactual for quantitative data analysis. The evaluation design is drawn on mixed method approach to leverage the complementarities of secondary and primary data. The primary information collection entailed use of both quantitative and qualitative methods. The quantitative information is gathered through representative household survey i.e. Post-KAP surveys and school sanitation surveys. The qualitative methods include in-depth interviews, focus group discussions, field observations and consultative workshops with key stakeholders. For data analysis, the evaluators used data triangulation techniques to further substantiate commentary and analysis. This facilitated in measured assessment of the extent and mapping the reasons for change. To understand the programme effectiveness in terms of inclusion/equity (as per UNICEF’s MoRES framework), the quantitative analysis offers useful segregation at income levels, gender and disability. In addition to gender and equity, it also examines vulnerability (including for the women, children, disabled and the elderly) and focuses on elements such as provision of subsidy and construction of demo latrines for the vulnerable, as well as on household and school wash facilities (taking into account the needs of the disabled). To map programmes contributions to larger context related changes (in comparison to the baseline), the evaluators have drawn province based comparative bottleneck analyses. For this consultative workshops have been organized at provincial level.
4.2. Data Collection Methods The data or information was gathered through both secondary and primary information sources. The secondary sources included project documents, national and regional research reports, published and unpublished data, web search and review of data-bases. For primary data collection, both quantitative and qualitative methods were used. The quantitative data was gathered through a representative
[16]
household level Post KAP survey. For assessment of school WASH facilities, a non-representative school sanitation survey was undertaken. The quantitative tools used for the SPSP Baseline were used for the evaluation, with minimal changes. The data entry was done on Census and Survey Processing System (CSPro), specialized software for data entry. The data was then exported to Statistical Package for Social Sciences (SPSS) and MS Excel, the software used for the data analysis. The qualitative tools included IDIs, FGDs, interactive workshops, field observations checklists, and photographs. The qualitative tools were applied to a wide variety of stakeholders (directly and indirectly involved in the programme) to get their reflections around programme design and delivery. The qualitative tools were used across all regions/provinces, where at least one site/community was picked up from each province/region. Although, there are no prescribed standards for sampling for application of qualitative tools, however, the evaluators are confident that the scale and breadth, at which quality tools have been applied, enabled to capture the depth and diversity of views in varied contexts. The evaluators have reached out to all key stakeholders (some even multiple times), which is quite reassuring. The figure below illustrates the range of research methods used for the evaluation.
Figure 3: Evaluation Data Collection Methods
4.2.1. Quantitative Evaluation Methods Post-KAP Household Survey Post-KAP survey was administered at the household level to collect representative quantitative data, which enabled to draw trends and patterns analysis with respect to changes in knowledge, attitudes and practices around water, sanitation, health and hygiene aspects. The analysis has been undertaken to capture trends for equity and gender aspects. The Post-KAP administered on 5185 households, where respondents were selected through multi-stage random sampling (refer Annex 9 for more details on sampling approach). Equal numbers of men and women respondents were interviewed in all districts, however in FATA only men were interviewed.
Evaluation
(UNEG, OEAD/DAC Criteria, UNICEF Core Strategies, Normative Principles)
(disaggregated by gender, income quintile and vulnerability status)
Quantitative Data
Post KAP Household
Survey
Post KAP School Survey
Qualitative Data
Secondary Data
Review
In-Depth Interviews
Focus Group
Discussions
Bottleneck Analysis
Workshops
School WASH FAcility
Observation
Checklists
[17]
Find below the sampling calculations, where detailed list of villages is attached as Annex 9.
Categories of
Respondents
No. of districts/
locations
Total Remarks
Male
13 + 3 (additional
FATA regions)
3189 In 3 FATA regions data
was collected only from
male respondents that’s
why male respondents
are calculated against
16 districts/locations and
female respondents
against 10 districts and
locations.
Female
13 – 3* = 10 (*FATA
regions)
1996
Total Sample 13 5,185
Table 4: Sampling Calculations for Post KAP Household Survey
Note: The above calculations were based on 13 districts/agencies considering, the overall margin of error for target sample as 1.35% at 95% confidence level.
Post-KAP School Survey Non-representative School Sanitation Survey was also administered in 78 schools, where at least 6 schools were selected (randomly) in each district/agency. Only those schools were selected, which had received assistance to rehabilitate the WASH services. The respondents comprised school head teachers or senior teachers. The inclusion criteria for schools included equal number of boys and girls (except FATA, where only boys schools were selected) schools at primary and secondary level. 4.2.2. Qualitative Evaluation Methods Secondary Data Review The evaluation involved extensive desk review of relevant documentation comprising Government Policies (PATS, NSP), UNICEF policies and programme documents (such as gender equality, HRBA, CATs, Child Friendly Schools etc.), sector reports, databases, TPM reports, national and regional survey reports etc. The extensive review helped understand the programme, larger environment in which it was implemented and the challenges. Also, it fed into the evaluation design especially tools development. The numeric data (secondary) was used to draw comparison with field survey findings and referenced as such (refer Annex 10 carrying details of the documents reviewed). In-Depth Interviews In-depth interviews (IDIs) were carried out with range of stakeholders primarily at national, provincial and district level public officials (MoCC, PHED, Health, Local Government, Education and others), managers of project partners (including UNICEF), non-profit organizations active in sanitation, political and community leaders, masons, third party monitors, mart owners and others. In total 36 IDIs were conducted where provincial and districts level public officials were covered multiple
[18]
times (refer Annex 7 – List of People Met for the SPSP Evaluation). The senior members of the evaluation team administered interviews to ensure consistency and quality. The instrument design took note of generating complementary information to help validate Post KAP results and triangulate information to deepen the analysis. The focus was to establish how far and in which ways the programme interventions brought changes in engaging public stakeholders and delivery of WASH services. The sample was evenly spread across the regions (across districts/agencies in each province/area) to capture regional disparities and reflect as such in the analysis and recommendations. Focus Group Discussions Like IDIs, the evaluators administered FGDs to draw complementary primary information to deepen commentary and analysis. Varied community or beneficiary groups comprising members of village sanitation committees (VSCs), members of school health club (children), community resource persons (CRPs), and School Management Committees (SMCs) participated in the FGDs. The discussions were carried out to understand the involvement, contributions and results of these efforts in changing knowledge, attitudes and practices around WASH. In total 40 FGDs were undertaken in five selected communities in as many districts/agencies i.e. one each in the provinces and an additional in the FATA agency. Please refer Annex 7, which carries the list of people who participated in the FGDs. Bottleneck Analysis Workshops The evaluators facilitated day-long bottleneck workshops, one each at provincial headquarters (four in total), in line with UNICEF MoRES framework to assess the extent change that SPSP I has been able to bring to the larger sanitation environment at provincial level. The participants included both public officials from relevant depts. and IPs from respective provinces. Find attached the reports that offer analysis as to the extent of change and causes in the larger sanitation sector (in relation to what transpired during baseline stage) including list of participants i.e. Annex 8.
School WASH Facility Observation Checklists To assess the quality and consistency with child friendly WASH facility standards, evaluators developed and administered WASH facility observation based checklists. The checklists were used to assess whether the facilities met the design specifications as provided in the manuals, and establish facility usability, standard of maintenance and likelihood to sustain/remain functional. The checklists were administered in selected schools.
[19]
Find below the table that offers summary of the methods/tools used, sources/respondents and limitations.
Instrument Source Limitation
Post-KAP Household Survey Men and Women at household level from target geographical locations
Absence of Control Group Some instances of Security Incidents and inaccessibility to areas, requiring a change in sample villages.
Post-KAP School Survey Teachers and School Children No Specific Limitation
Secondary Data Review Various Reports, Assessments and Policy Documents
Dependency on the existing body of literature, data and statistics. In some cases data desegregation is not available for comparison (geographically in the context of districts or in terms of gender).
In-Depth Interviews Government Departments (Health, Education, PHED, LGRDD at Provincial and District Level)
Availability of Government Officials in all project locations for interviews during the evaluation period. Frequent change of officials during the project life.
Focus Group Discussions Children (boys and girls), VSCs (Men and Women), Entrepreneurs and Mart Owners, CRPs (Men and Women)
Availability and sufficiency of numbers to allow for FGDs. In some instances due to limited attendance FGDs had to be managed as Interviews.
Bottleneck Analysis Workshops Public Sector Partners (Government Officials) drawn together at the provincial level.
Official deputed from relevant department may not have been involved during the project cycle.
School WASH Facility Observation Checklist
Actual visits to School WASH Facilities (Evaluation Team Observations)
Checklist is focused on assessing whether provided criteria and design specifications were observed or not. In some cases requires qualitative comments.
Table 5: Summary Table for Instrument, Source and Limitations
4.3. Evaluation Constraints, Limitations and Mitigation The evaluation went largely as per plan. As happens with most of the evaluations, the evaluators faced some constraints and challenges, for which appropriate remedial actions were adopted to not compromise quality. The key constraints and challenges are listed below together with mitigation measures undertaken by the evaluators.
[20]
Constraints/Limitations Mitigation Measures
Terms of Reference: Ambiguities around evaluation key questions and sub-questions.
Consultants sought clarifications and convened meetings with UNICEF team (including PMER section) to get clarity on key evaluation questions and sub-questions, which later were made part of the inception report.
Theory of Change: Shared documentation and programme proposals lacked an articulated Theory of Change.
The consultants in consultation with relevant UNICEF staff developed Theory of Change, and used as such for evaluation.
RBM Logic: The results framework found Inconsistent with RBM principles e.g. missing outcome statement.
This could partly be attributed to gaps in the donors’ results framework template; however evaluators designed comprehensive tools (post-KAP, IDIs and FGDs) to capture all results indicators desegregated across income quintiles, gender, and disability.
Creation of Counterfactual: The evaluation design excluded establishing counterfactual to assess the element of attribution.
The evaluation being formative in nature did not require establishing counterfactuals (as it mostly required for impact evaluations), hence excluded from the evaluation design. Moreover, it might have a challenge in itself to find a reliable control group, given the fact that communities (in project areas) are receiving humanitarian assistance from multiple stakeholders.
Security Situation Unstable security environment constrained accessing some project sites and caused delays.
A robust security management system established with senior resources on the ground taking lead in security assessment, liaison with local authorities and taking on the spot decision around withdrawal, rescheduling and changing field sites/villages.
Gender Context: Beliefs and cultural sensitivities especially in parts of FATA constrained access to women beneficiaries.
To overcome local sensitivities, locally recruited female enumerators and FGD facilitators were involved to administer household interviews and FGDs from female beneficiaries.. However, in view of the particular context in FATA agencies, the consultants in consultation with UNICEF agreed to interview male respondents only.
Methodological & Individual Biases: Methodological and individual (core team members and field staff) biases potentially affecting evaluation findings and analysis.
The evaluation took note to address this at all stage such as using both quantitative and qualitative methods, use of data triangulation techniques for analysis, instruments design with the intent to limit human biases, deploying experienced and trained resources for qualitative data collection. Moreover, extensive consultations within the core evaluation team, with UNICEF staff (including evaluation unit) and later the presentation to the Evaluation Reference Group (including the UNICEF Evaluation
[21]
Unit) helped keep continuous check to overcome methodological constraints and ward off the personal biases to keep into the analysis and recommendations.
Non-availability of Some Key Project Documents: Consultants did not receive the key project documents such as expenditure statements constraining desegregated cost benefit analysis.
The consolidated expenditures statements were not available hence evaluators used the budget figures gathered from PCAs.
Table 6: Evaluation Constraints, Limitations and Mitigation
[22]
4.4. Ethical Considerations The evaluators take note of and followed the UNICEF adapted United Nations Evaluation Guidelines (UNEG), standard research norms and ethics, for evaluation design, field data collection and reporting. Some of these include demonstration of impartiality, seeking prior consent from the respondents, ensuring data and informant anonymity. The evaluators sought prior permission to record conversations (digital recording) and for field photography. UNICEF focal points were kept informed of the progress and where required consulted during field data collection, analysis and reporting. The members of reference group were also kept informed mostly through UNICEF focal points.
4.5. Quality Assurance Mechanisms To ensure consistency to the accredited and agreed research and evaluation standards, the evaluators developed a set of quality assurance mechanisms for data collection, field work, data analysis and reporting. Some of those mechanisms are listed below; a. Where respondents agreed and allowed, the IDIs and FGDs were digitally
(audio) recorded and transcribed to avoid loss of field information;
b. To the extent possible, the evaluators hired the local youth/resources and
rigorously trained them before having them involved in field work. Female
local resources hired to overcome cultural sensitivities around interacting with
outsiders;
c. The field training of enumerators was conducted by senior evaluator, involved
in evaluation and instrument design. The training and contents and delivery
approach was standardized. Moreover, to ease work of field teams, field
guidelines were developed for field staff comprising District Coordinators,
Supervisors, Enumerators and Data Entry Operators;
d. The evaluators used the frequently used and reliable industry
software/packages for data entry (CSPro) and data analysis such as SPSS
and Excel;
e. 5% of the Post KAP forms were validated through field quality assurance
staff;
f. Distinguished sector professionals with complementary training and field
experience were engaged in the evaluation design, field data collection,
analysis and reporting;
g. Field application of IDIs and FGDs (the key quality tools) by senior evaluators
(some involved in design also) to ensure consistency and quality
administration;
h. Security protocols were developed and implemented throughout field data
collection. Regular contact was maintained within the team members, field
teams and local authorities (through respective IPs and UNICEF staff).
[23]
4.6. Evaluation Management and Stakeholder Participation
The evaluation was commissioned and managed by UNICEF Islamabad Office. The WASH Team was in the lead with technical oversight from the Evaluation Unit. The UNICEF Provincial Offices extended field level coordination and facilitated in organizing province based bottleneck analysis workshops. UNICEF teams at varied levels assisted in linking up the evaluation team with respective stakeholders (Government officials, IPs and community/beneficiary groups). A regular contact was maintained between UNICEF and evaluation team focal points. Significant time was taken during inception phase to clarify expectation, responsibilities, methodology, process and the output. The field activities were primarily facilitated by the IPs. This included scheduling interviews with local officials, FGDs with school children, VSCs, CRPs, SOs and School Teachers and application of school based observation checklist. By and large, the national and district level government officials made themselves available to discuss in detail and respond to evaluation questions comprehensively and candidly to help better inform the evaluators on their perspective of what worked well and what could be improved in the programme. At national level, UNICEF facilitated initial contacts whereas in the field IPs helped connecting to the local officials. Community was involved in various forms in the overall evaluation process, including making themselves available and giving time to the various evaluation processes and use of instruments (such as household members giving time and sharing their views and feedback in the household survey and similarly for children and school teachers during the School Survey). FGDs were organized with groups formed or supported during the project such as CRPS, VSCs, sanitation mart owners, entrepreneurs and masons. The IPs facilitated gathering community members at common spot (mostly guest room of local influential) to help evaluators interact with respondents. The female FGDs were largely performed inside the houses. It was encouraging to note that women and girls shared their views openly. An Evaluation Reference Group was formed (refer to Annex 14 for Evaluation Reference Group TORs) comprising representatives from UNICEF (WASH Team, Evaluation Unit), DFID, CCD and relevant provincial government departments. The evaluation team maintained regular contact and sought inputs both from the Reference Group and UNICEF’s Evaluation Management Committee (mandated for quality assurance) through PMER Section. A two days Reference Group consultation session was organized towards the end, where the evaluation team presented the key findings, analysis, conclusion and recommendations. Over the course of the consultation session, the recommendations were refined and translated into action plans for UNICEF to incorporate into the next cycle of SPSP programme. Third Party Monitors, Apex Consultants, on the advice of UNICEF monitored and reported on compliance to agreed standards. For instance, the TPM monitors
[24]
participated in the enumerators’ trainings, observed the implementation of the Post-KAP household and school surveys (by observing the conduct of field enumerators) and other field activities. The AAN consultants were responsible for evaluation design, field planning, organising field resources, training, logistics and security management. The firm was also responsible for conducting a professional evaluation, and producing high quality evaluation report.
[25]
5. FINDINGS
This section of the report offers key findings and analysis. The description is structured around key evaluation criteria and questions.
5.1. Relevance Evaluation Q1: To what extent the programme design and implementation was relevant to national/provincial priorities and needs of men and women beneficiaries in terms of achieving expected results across all socio-cultural groups including the vulnerable in the targeted communities? The evaluators have evaluated the relevance of the programme against a range of aspects including, relevance to international commitments, national and provincial policies and priorities, to the needs of men and women beneficiaries including the vulnerable, to total sanitation approaches and also to the integration of gender equality, equity and disability. Government’s International Commitments The programme objectives, selection of approach and geographic prioritization all point to relevance to Pakistan’s international commitments such as MDGs and SACOSAN. The relevant MDG sets the target to achieve 67% sanitation coverage by 2015, which at present stands at 50%. The programme was planned to increase sanitation coverage in the selected target villages and communities in rural parts of the country and successfully increased access to sanitation facility to 65% within the project areas thus also contributing to the overall MDG target (SPSP Evaluation Post KAP results - refer to Figure 4).
67%
50% 44%
65%
0%
10%
20%
30%
40%
50%
60%
70%
80%
MDG Target Current National Coverage
SPSP Baseline SPSP Evaluation
Proportion of Population having Access to Sanitation Facility
SPSP Contribution to Sanitation Target
Figure 4: SPSP Contribution to Sanitation Target
[26]
SPSP contributes to the following WASH related CCCs: Commitment 3: Children and women access toilets and washing facilities that are culturally appropriate, secure, sanitary, user-friendly and gender-appropriate; Commitment 4: Children and women receive critical WASH-related information to prevent child illness, especially diarrhoea; and Commitment 5: Children access safe water, sanitation and hygiene facilities in their learning environment and in child-friendly spaces.
Constitution of Pakistan does not recognise sanitation as a right yet, however, the national sanitation policy recognises that right to health cannot be achieved without sanitation. (http://www.washwatch.org/southern-asia/pakistan ) Following SACOSAN 4, Pakistan (including other SAARC Countries) adopted Colombo Declaration which commits governments to legislate and make the right to sanitation a part of the constitution.
The programme also conforms to the Kathmandu Declaration issued at the end of SACOSAN V, where Pakistan pledged to contribute to the ODF status for South Asia by 2023. The SPSP made a noticeable contribution by having successfully reached to 1.2 million people (men, women, boys and girls) with demand creation for sanitation messages and increasing the number of people living in ODF villages by 711,467. UNICEF Global and National Commitments Programme objectives also have close relevance to UNICEF’s Pakistan country programme commitments and to its global Core Commitments to Children (CCCs). The objective is to ensure children and women access to safe, sufficient and reliable WASH facilities. UNICEF’s work in Pakistan is guided by cooperation programme for 2013-2017, aimed at improving the lives of children and women in Pakistan. WASH is reflected as one of the priority areas for UNICEF and interventions are guided by UNICEF CCC and United Nation Development Assistance Framework i.e. UNDAF. Hence, it could be argued that SPSP I demonstrates coherence to organizational commitments and standards and larger UN mandate and priorities. National and Provincial Policies The programme has relevance to the National Sanitation Policy 2006 and draws on the Pakistan Approach to Total Sanitation (PATS) 2009. The programme design is found consistent with PATS approach, which is integral part of both national and provincial sanitation policies, even if available in draft form. The PATS approach places communities (beneficiaries) at the centre of driving and sustaining changed sanitation behaviours. The programme is strategic in terms of rolling out PATS and engaging the relevant public sector stakeholders selectively to work together with civil society to better understand PATS principles and components, and in the due course get prepared for complete ownership, replication and scale-up. Particular components and principles were found aligned to other relevant national level polices e.g. Environment Policy 2005, Drinking Water Policy 2009 and Climate Change Policy 2012.
[27]
In the wake of devolution of public functions and resources to provinces after 18th Constitutional Amendment and 7th National Finance Commission (NFC) respectively, provincial WASH policies and plans have either been approved or being finalized. Across all provinces, PATS remains the key policy priority for sanitation services, which again points to consistency with the evolving policies within provinces. Sector Coordination and Harmonization Activities The programme apparently has made limited success with sector coordination. UNICEF is recognized as key sector player and is involved in coordination, however that role transcends beyond programme. The focus remained largely on national level coordination with limited attention and resources for provincial and district levels sector coordination. For instance, UNICEF organized Sector Partners Workshop in June 2013, which brought the stakeholders together and event was used to showcase UNICEF work and findings of the RuSFAD III Evaluation and SPSP I Baseline. There are multiple thematic coordination forums, mostly at the national level where UNICEF is represented and participates regularly. UNICEF leverages its presence to share learning from SPSP and other programmes to inform sector discussions and policy decisions. The forums available are as under (some have become dormant also);
National Drinking Water & Sanitation Committee
National Sanitation Policy Implementation Committee
PACOSAN
CLTS Core Group under PATS (UNICEF is one of the members) As far as harmonization of efforts is concerned, neither federal government nor UNICEF is actively involved to facilitate harmonization of WASH stakeholders and activities. The situation is more or less the same in all provinces with exception of Punjab, where Planning & Development department has recently formed a Provincial Sanitation Committee to guide and coordinate the sanitation sector interventions. For programme interventions, the evaluators did not find evidence as to allocation of resources and active contributions by relevant district authorities such as contributions to rewards, though envisaged in the programme design. This may be the result of limited guidance by UNICEF provincial offices and lack of focus by IPs on mobilizing resources from concerned local officials. This may need a different delivery approach by IPs and considerate guidance and support from UNICEF provincial offices. The evaluators suggest that the programme roll-out in following years must prioritise and set guidance for formal coordination at national, and more importantly provincial and district levels. The relevant departments such as PHED or LGRDD (as varies in different provinces) must be given due support to take initiative for harmonization of efforts and resources of WASH stakeholders. The provincial level coordination and harmonization by lead public agencies may create enabling environment for IPs to work with district officials and cultivating greater sense of ownership amongst public agencies on ground.
[28]
CATS Essential Element 4: Subsidies – whether funds, hardware or other forms – should not be given directly to households. Community rewards, subsides and incentives are acceptable only where they encourage collective action in support of total sanitation and where they facilitate the sustainable use of sanitation facilities. CATS Essential Element 5: CATS support communities to determine for themselves what design and materials work best for sanitation infrastructure rather than imposing standards. External agencies provide guidance rather than regulation. Thus, households build toilets based on locally available materials using the skills of local technicians and artisans. UNICEF Field Notes – CATS
Total Sanitation Approaches The integration of SLTS and CLTS approaches where SLTS is seen to contribute to the continuity of WASH work in communities, found consistent with both PATS and CATS. The simultaneous application of SLTS and CLTS was found complementary and contributed to reinforcing the message. The provision of subsidy to poorest, though appear relevant on equity grounds, together with construction of demo-latrines, remain the major departures from the CATS/PATS essential elements. Moreover, the use of pride, respect and dignity as integral to triggering appear another departure from the conventional approaches. Overall, the use of SLTS and CLTS contributed to complementing project interventions and also reaching out to all family members, hence resulted in reinforcing the message at household level. FGDs and feedback from IPs highlights several challenges around provision of subsidies 3% and 7% subsidies to the vulnerable and extremely vulnerable respectively. Some relate to distinguishing between vulnerable and extremely vulnerable, which at few places caused frictions in the community. Some IPs referred to vulnerable being so poor (or in some case lack of interest or commitment) left the latrines incomplete (as IPs supported latrine construction except super-structure). Some even not used at all and most being unusable anymore for lack of cleaning. The future programme design may need to take more considered approach with this. Justification for use of subsidy was under ‘Disaster Response Approach’ within PATS which allows subsidy in form of humanitarian assistance in case of post disaster/insecurity affected communities. PATS includes human rights perspective in sanitation programming. Use of ‘Pride, Self-Respect and Dignity’ instead of conventional ‘Shame, Shock and Disgust’ approach remains another key departure. The evaluator found it relatively more consistent with rights based programming, which remains cardinal to UNICEF programme design. The post-KAP results point to greater dependency amongst respondents, no matter what economic strata they belong. This is evident from survey results where overall numbers for people willing to construct latrines have increased with a corresponding increase in people asking for external assistance/cost sharing. It is difficult to establish in concrete terms what may have contributed to this, however apparently it may have been for project subsidy/demo latrines and larger environment where non-profit organizations have been distributing free of cost latrines.
[29]
The consultants propose review of subsidy delivery mechanics with greater involvement of VSCs and communities to decide about subsidies and rewards. One option could be to transfer funds to these community forums to prioritize localized interventions, which may very well include latrine construction for poor and extremely poor households. Masons were trained, as part of supplies facilitation to help improve supply of trained and locally available masons. The evaluators noted departure from CATS application which emphasizes engagement of local skills (to support local livelihoods) whereas the programme required IPs to engage contractors (not necessarily local) for hardware components such as rehabilitation and construction of school and health facility latrines. Future programme design may need to take note to address this.
Community Priorities and Programme Area Selection The programme areas demonstrated greater deprivation in terms of poverty, insecurity and gaps with respect to sanitation coverage in comparison to urban areas. The programme areas found falling far below the national targets set for MDG Goal 7. Moreover, districts illustrated low scores on UNICEF’s Child- Wellbeing Index, which as per UNICEF contributed to their prioritization. In essence, programme coverage demonstrates coherence to larger programme goal/aim. The multiple information sources such as Joint Monitoring Programme (JMP 2012), Pakistan Social and Living Standards Measurement Survey (PSLM) and UNICEF Multiple Indicator Cluster Survey (MICS), all show the selected districts being low on WASH indicators. The Fig # 5, offers comparative analysis for MICS and SPSP pre-KAP (baseline) results, and illustrates the significant gaps that exist in terms of access to sanitation in urban and rural areas. The baseline results indicate these all largely being the poorest districts across Pakistan. The numbers indicate that by and large the programme prioritised districts, which had low sanitation figures including poor and security affected. Moreover, the interaction with both public officials and communities (varied groups) in programme districts point to sanitation being a priority need, particularly for women, children and poor.
The UNICEF team shared that as children focused organization, they selected districts based on child wellbeing index, which points to the larger deprivation of these districts.
96%
72% 77%
91% 93%
38%
64%
26% 29%
68%
33% 22%
50% 45%
61%
0%
20%
40%
60%
80%
100%
120%
Sindh Punjab FATA Balochistan KPK
MICS Urban MICS Rural SPSP Baseline
Figure 5: Province Based Access to Sanitation Facility
[30]
Gender Equality, Equity & Disability Integration The programme design and implementation demonstrate a clear focus on gender integration and targeting the most vulnerable i.e. equity. In comparison, the focus and the demonstrated integration for disability have been far less e.g. in BCC, latrine designs and others. The programme did demonstrate incorporation of gender equality at varied levels. These include gender being a key element in defining vulnerability, thus ensuring access to subsidized sanitation inputs. The women and girl centred BCC components supported by women staff and volunteers. Moreover, creation of men and women community groups, which demonstrates recognition that women are equally important to take this forward, as men. The collection of gender desegregated data during baseline, evaluation and during monitoring to track progress on gender equality. Women were part of the delivery apparatus, where field staff had healthy women representation. As described above, UNICEF justifies inclusion of subsidy (a departure from PATS and CATS) for equity consideration, to support and enable poorer segments to have access to sustainable sanitation. The poverty context in rural Pakistan and programme focus on rural poor, to a degree justifies the provision of subsidy, despite the fact that this at times creates frictions (in community) and element of dependency amongst communities (as results of post KAP illustrate) to seek outside assistance for construction of latrines. The programme design refers to activities being designed and implemented with due regard to addressing the needs of the disabled. The elements assessed such as BCC; triggering; and education and awareness were all found less disability informed. Similarly, neither prescribed latrine designs nor products available, appeared disability friendly. The design and delivery of future programme may need to demonstrate greater focus on disability integration particularly with respect to BCC campaign, latrine designs and products.
Theory of Change The programme documents did not have an articulated ‘Theory of Change (TOC)’. However, as part of the evaluation, a TOC was documented by holding repeated consultations with relevant UNICEF staff. The TOC found largely consistent with PATS and SPSP results framework.
The IPs and Third Party Monitors (TPM) shared that the programme documents may not have an expressed TOC; however, the essence of it was verbalized during the Inception Workshops while explaining the programme implementation plan. This helped to create a common understanding between UNICEF and the IPs on inputs, outputs, outcomes and possible challenges and risks. This to a degree helped delivery of programme in line with verbalized TOC, where some incidences of variations noted amongst IPs. Key variations noted around departure from the set sequence to roll-out BCC. UNICEF is advised to either incorporate the TOC in the programme concept note or emphasize incorporation of
[31]
TOC in future PCAs. This shall facilitate common understanding of results, approaches, interventions, inputs and risks and minimize implementation level variations, unless essentially required for context reasons. Overall, the programme design found largely relevant and consistent with national and international commitments and policy frameworks including those of UNICEF. The geographic prioritization found extremely relevant, as evident from the low progress vis a vis MDG 7 targets. By and large the design found consistent with PATS and CATS approaches, with evident departures from key conventions such as provision of subsidies, non-engagement of trained masons/skilled workers and others. Both the design and delivery merit re-consideration to achieve consistency with principles of CATS and PATS. The evaluators found an expressed focus on equity and gender equality, however with relatively less focus on disability integration. The programme documents lack an articulated TOC, which to a degree compensated by its verbalization during inception phase.
[32]
5.2. Effectiveness Evaluation Q2: To what extent were project implementation strategies effective and successful in achieving the planned outcomes/results for rich and poor men and women across all socio-cultural groups including the vulnerable in the targeted communities? The evaluators have assessed effectiveness of the programme across varied programme aspects, i.e. design, programme pillars/interventions and strategies (demand creation, supply facilitation, health and hygiene promotion, linkages with government, attainment of total sanitation, knowledge management and accountability) as well as effectiveness of integration of gender equality, equity & disability.
5.2.1. Programme Design The programme achievements as evident from comparative analysis of pre & post KAP results, point to effectiveness of PATS/CATS especially the parallel implementation of SLTS and CLTS. This helped leveraging complementarities and to a degree has planned to set foundations for continued community mobilization through schools. UNICEF used indirect implementation approach, whereby it contracted International and national non-profit organizations to lead implementation. Similarly, arrangements were made with relevant public agencies to work closely with these IPs. The implementation modalities did help with efficient delivery, however limited engagement of public agencies remains a concern (at both provincial and district levels). This may affect having willing and capacitated public sector for PATS replication and scale-up. UNICEF and other development partners such as WSP, WSSCC, WaterAid and others, are advocating PATS adoption at varied levels across the country. There are developments that point to the success of advocacy such as increased budgetary allocations for FATA, selective rollout of PATS in Punjab and others. UNICEF surfaces as a major player; however given the involvement of other stakeholders in advocacy attribution to UNICEF efforts alone is difficult. . The evaluators did not find reference to stakeholders leveraging their engagement and resources to piece together a collective advocacy plan to maximize impact. UNICEF may need to prioritize this for subsequent cycles of SPSP rural. UNICEF contracted specialized partners for action research, third party monitoring and centralized training, which largely proved effective. DSI representative, the partner contracted for action research shared that there are several key points that UNICEF can integrate in the next phase. For instance, broadening the scope of BCC materials by adding multi-dimensional focus e.g. emphasizing sanitation vis a vis health, privacy, religion and others. Moreover, DSI provided entrepreneurs with business and motivational training. APEX Consultants, the TPM worked as per the framework available providing real-time monitoring information to help UNICEF keep track of progress and make informed decisions. While centralised training by RSPN was effective in terms of standardization, deployment of dedicated teams for capacity building across different functions and layersthe RSPN staff and IPs both pointed to gaps in implementation mainly for overstretched RSPN staff. This may
[33]
have been because of under-estimation of human resources required on the part of RSPN. The training materials/manual produced will be available for future SPSP cycles. The programme design offers expressed focus on integration as to gender, equity and disability. For some elements, the inadequacies as to operational guidelines and standards (PCAs offer limited information) resulted in in-consistent implementation across sites and partners. For instance, a girl school visited did have elevated latrine including water points, where the students and teachers complained that as the boundary wall is low, they feel more exposed to people outside whenever they move up either to collect water or use latrines. This may have helped securing the WASH facilities from floods; however in absence of elevated walls, they feel that their privacy has been compromised. The future programme design and delivery may need to consider this and couple elevated WASH structures with raising boundary walls, particularly at girl schools. Similarly, for equity integration household subsidies were provided, however identification of vulnerable and extremely vulnerable individuals was done during the PRA exercises carried out before VSC formation, which caused friction at later stages. There are indications that the subsidy provisions are working to discourage communities (even the well-off) to construct latrines on their own, as evident from the survey results (shared above). For disability integration, the WASH manual (latrine designs) provides guidelines; however evaluators did not find much evidence suggesting the disability integration for facilities restored at schools, communities and health centres. The evaluators feel that ambitious targets were set especially with respect to time allocated. The partners were stretched further for delayed start, owing to PCA approvals and seeking field level authorization from district authorities. This left not more than 9 months (on average) for actual field implementation, making it even more ambitious. The field interaction with communities points to design and planning inadequacies around taking note of and adjusting activities to the community work cycle (e.g. harvesting activities, closure of schools during summer holidays), despite the fact that seasonal calendars were prepared through PRA. This contributed to IPs circumventing (to the extent of compromise) the laid processes to save time, especially capacity development, post ODF monitoring and others. This needs serious consideration from UNICEF to ensure that each IP gets at least 12 months of implementation time. For lack of clarity as to what it entails (from UNICEF) for the first programme component or pillar i.e. coordination with public stakeholders, the IPs used varied interpretations and implemented whole range of different activities. The evaluators did not get more clarity from UNICEF either as to what it entailed. If it meant more structured and systematic engagement (with respect to contributions to rewards, increased budgetary allocations etc.) with local and provincial public agencies, the results fall short of expectations. The evaluators noted gaps in communication chain (both at horizontal and vertical levels), again probably result of un-explained pillar. The provincial PHED which is the lead agency in Punjab also communicated that reporting needs to improve ensuring regular reporting and monitoring of field progress.
[34]
UNICEF appear to have consulted stakeholders particularly public agencies, however the extent to which these consultations contributed or informed programme design was difficult to ascertain. The other stakeholders like WSP and WaterAid did refer to being aware of the programme, however none referred to having been consulted for the design. UNICEF apparently guided and supported IPs for the design of PCAs, however how much have IPs contributed to the design is unknown. The evaluators view is that a programme like SPSP with a range of partners and of significant magnitude may need to evolve mechanisms to foster consultation and wider ownership by the stakeholders. This would surely have bearing on both the design and implementation.
5.2.2. Programme Management & Coordination Overall arrangements for programme management and coordination were found appropriate and relevant to the nature and scale the programme. The timely completion and achievement of results point to these being effective also. UNICEF WASH Unit based at Islamabad worked as the lead management and coordination point. Support was provided by the UNICEF Provincial Offices for coordination with public stakeholders, oversight of province level IPs, field monitoring and troubleshooting. Most of the decisions, in many instances were delayed for instance, delay in capacity building PCA resulting in a 3 month waiting period before RSPN could provide training to partners, delayed guidance on voucher system as well as delays caused by withdrawal/disengagement from loans and grants to entrepreneurs. The IPs shared that they were given sufficient flexibility, which they appreciated given the diversity of contexts. IPs referred to UNICEF being supportive through the implementation. Also, that they were able to communicate with confidence and freely, and received timely guidance and support with any issues or bottlenecks highlighted. For instance, an IP in Sindh shared that demo latrine cost revised upwards as were found to be under-budgeted in the original budget. The umbrella arrangements as used in Sindh added an extra layer with raised operating costs; however evaluators were unable to find a clear reference or evidence to the value addition of the umbrella partner. In absence of any significant value addition, UNICEF is advised to re-consider umbrella arrangements. More analysis around this is available in the partnerships section. The practice of monthly substantive monitoring reports helped keep track of progress, take note of deviations and constraints and take necessary management decisions for course correction. The evaluators were told of some of key challenges that programme faced, which are as below;
i) The TPM were contracted centrally by the Islamabad Office and relationship featuring direct reporting to WASH Unit Islamabad completely excluding the provincial UNICEF offices. The review of the monitoring framework (templates and tools available to TPM) indicates lesser flexibility and scope to highlight and incorporate the operational constraints. This IPs and UNICEF provincial offices referred to the process
[35]
being rigid and less sensitive to the field context they operated in. The UNICEF provincial offices raised concerns around being out of reporting loop for TPM, which UNICEF Islamabad office may need to look into. The consolidated versions could be shared with provincial offices for information purposes.
ii) UNICEF (both at Islamabad and provincial offices) could have played more effective role with public sector advocacy, had more structured process followed for public sector coordination at varied levels. The limited success with budgetary (provincial) increases, contributions by public sector into rewards and others, demonstrate UNICEF and in particular provincial offices may need to take more hands on approach for public sector coordination. This impacted IPs engagement with district level public stakeholders and level of ownership of the programme by PHED and LGRDD in districts.
iii) For lack of systematic and structured coordination, the approvals for implementation by the district authorities got delayed invariably for all partners. This delayed the project kick-off thus stretched the IPs resources further to deliver on committed results. One such example is the NOC for Muslim Aid in DG Khan, which was never resolved and left the project exposed to a range of risks. Moreover, the relationship and terms of engagement (including the focal agencies) between IPs and district governments kept on evolving. Some IPs were lucky to find receptive focal points, thus were able to mobilize public sector support, whereas for others it was struggle all along.
5.2.3. Programme Components & Results Note: The results framework lacked outcomes statements and indicators, hence evaluators used outputs indicators (which in strict results term could be equated to outcomes) to map the extent of change the programme implementation brought (by comparing pre and post KAP results).
Output 1: Demand for Sanitation SPSP used both CLTS and SLTS approaches, which helped to sensitize and educate communities and thus resulted in increased demand. The results are indicative of the approaches complementing each other, which was supported further by the cyclic BCC campaigns. The objective to reduce practice of open defecation, raise community’s awareness of the need to achieve ODF status in their villages and to create demand for sanitation facilities by the target population, all showed encouraging upward change towards the end of the year (end of Phase 1). Community Mobilisation The decision to engage RSPN (to help train SOs, CRPs etc.) and draw on RSPs experiences and approaches for community mobilization (in use for over three decades) was well-considered. However several aspects including sufficiency of training team in terms of numbers, as well as better scheduling could have led to greater effectiveness in mobilisation.
[36]
The results in Table 7 points to IPs exceeding targets mostly for community mobilization and sensitization related indicators.
Community Mobilization Activities & Training Target Actual
SOs Training for PATs (participatory tools, IEC material) 8 85
CRP Training on Conducting CLTS 46 67
PATS Facilitation Session by SOs in Villages 230 911
CRPs facilitation session on CLTS 1,385 3,471
VSCs Formed and Trained 1,385 2,994
Community Action Plan (CAP) Developed 1,385 1,878
Table 7: Community Mobilisation Activities & Training
Generally the feedback has been positive on the quality, duration and content of the RSPN Social Organisers (SOs) training. The SOs shared that the training helped them get prepared for the expected work, however highlighted that one extra day for field work (practical demonstration) may have added more value. Moreover, they shared that the scale and frequency (considered excessive) of mentoring activities and refreshers (to be undertaken by RSPN) affected SOs mobilising schedule. Most of the SOs were found satisfied with the workload, however for delayed start, they were left with no option but to execute their field plan, even if at times it clashed with community activity cycle (rural). The availability of both men and women SOs were appreciated to help overcome the gender divide. Again, the programme performance found encouraging where the IPs exceeded almost on all targets (refer Table 8).
School Led Total Sanitation Training and Events Target Actual School Teacher Training 23 84
WASH Clubs Formed in Schools 692 881
Events Conducted by School Wash Clubs 886 1,433
Boys and Girls reached with sanitation messages in Schools. 103,000 176,022
Table 8: SLTS Training and Events
Behaviour Change Communication (BCC) The post KAP results and interaction with stakeholders involved including communities point to the effectiveness of BCC strategies and interventions. The key factors that appear contributing to the success are i) use of a variety of mediums targeting varied groups (IEC material, theatres & mass media) and; ii) repetitive rounds of multiple activities. They helped targeting different intended population groups with varied messages and mediums (matching the information preferences and interests) and in a way re-assured that message gets across to everyone at least once. The involvement of opinion leaders such as Imam Masjid (prayer leaders through Friday sermons), health workers, and teachers further strengthened the communication. Refer table 9 for details. The reports indicate IPs being able to meet the targets for most of the activities.
[37]
BCC Campaigns Outreach Target Actual
Children (boys and girls) reached with demand creation for sanitation messages in Schools. 103,000 176,022
People (men, women, boys, girls) reached with demand creation for sanitation messages. 776,000 1,200,529
People reached with demand creation for sanitation messages through electronic/print media. 970,000 540,786
People reached with demand creation for sanitation messages through community events (theatres, puppet shows). 291,000 195,450
Table 9: BCC Campaigns Outreach (Target versus Actual Number of People)
The effectiveness of BCC strategy is evident from the number of people claiming to have received the health and hygiene messages in last one year. The KAP results indicate that this has increased from 18% (during baseline) to 59% (pre-post KAP results). This shows that within the span of 8-10 months, the number have risen from 1 to 3 (for every five) people having received at least one health and hygiene message. The effectiveness of the mixed medium and repetitive BCC campaigns contributed to a significant proportion of respondents claiming to have received information from one or more sources (36%, 37% and 27% claimed to have received information from one, two and three sources respectively). It means that those who received messages, 2 out of three had received from two or more sources (refer Figure 7).
The awareness and knowledge factor (in terms of being able to remember and recall messages) has also been effectively improved from 71% to 98%. The most significant is the change in behaviours, where 92% respondents claimed that they have adopted the practice (following receipt of message) compared to 53%, at baseline (refer to Figure 7).
One Source, 36 Two Sources, 37
Three Sources, 27
Number of Sources of Health & Hygiene Messages
One Source
Two Sources
Three Sources
Figure 6: Number of Sources of Health & Hygiene Messages
[38]
Some elements of the BCC campaigns have been appreciated and being practiced more than others e.g. 10 Steps of Hand Washing. The Figure 8 illustrates that 93% respondents were able to remember/recall health and hygiene specific messages compared to 67%. Similarly 66% and 40% (from 56% and 24% respectively) of the respondents were able to remember/recall messages about significance of clean and safe water and water borne diseases. .
The number of people who could recall three key hygiene messages doubled over the course of year (refer Figure 9). The Punjab province is in the lead followed by Sindh and Baluchistan.
18%
71% 53% 59%
98% 92%
0%
20%
40%
60%
80%
100%
Health & Hygiene Message Received
Remember H&H Messages Practice H&H Messages
Health & Hygiene Awareness Messages
Figure 7: Health & Hygiene Awareness Messages
67% 56%
24%
93%
66%
40%
0%
50%
100%
Personal Hygiene Clean And Safe Water Water Born Diseases
Health & Hygiene Messages that the Respondents Remembered
Health and Hygiene Messages that the Respondents Remembered
Baseline Evaluation
Figure 8: Health and Hygiene Messages that the Respondents Remembered
24% 43%
23% 26% 23% 27%
80% 64%
48% 46% 33%
51%
0%
50%
100%
Punjab Balochistan Sindh KPK FATA Total
Proportion of Population Recalling Three Key Hygiene Messages
Baseline Evaluation
Figure 9: Respondents Able to Recall Three Key Hygiene Messages
[39]
The BCC strategy has also been effective in delivering health and hygiene messages at schools, as noted in Figure 10. 77% schools confirmed having received health and hygiene sessions over the last six months, whereas 41% schools reported to have had sessions conducted in-house by the school teachers. Gender focus in terms of coverage of boys and girls schools also comes across as sufficiently balanced.
Health and hygiene topics covered through school sessions, where message around personal hygiene were appealed the most. This is evident from the fact that 73% school teachers were able to recalling delivery of this message, followed by environmental and domestic hygiene i.e. 69%.
In terms of effectiveness of BCC mediums, face to face community sessions by the SOs and the CRPs were referred to as being most effective in terms transfer of knowledge. The respondents shared that these were more useful as it provided an opportunity for them to interact with the facilitators, ask questions and get response for their queries. It was also noted by a range of stakeholders (SOs, children, VSC members and CRPs) that theatres, cricket matches, key events celebrations (including world water day, environment day and hand washing day) and use of mass media, was also effective though with varying degree of results and responsiveness. These elements need to be kept and further strengthen in the subsequent years.
77%
41% 47% 27% 27%
59%
9% 31%
0% 20% 40% 60% 80%
100%
Imp
lem
enti
ng
Par
tner
Sch
oo
ls
Bo
ys
Co
Ed
uca
tio
n
Gir
ls
Bo
ys
Co
Ed
uca
tio
n
Gir
ls
Implementing Partner School
Source of Health and Hygiene Messages in Schools
Figure 10: Source of Health & Hygiene Messages in Schools
73%
58%
36%
69%
42%
42%
0% 10% 20% 30% 40% 50% 60% 70% 80%
Personal Hygiene
Clean & Safe Water
Water Born Diseases
Environmental & Domestic Hygiene
Safe Disposal of Human Excreta
Solid Waste Disposal
Figure 11: Topics Covered during Health & Hygiene Sessions in Schools
[40]
The BCC campaigns proved effective in terms of making people understand and recall hazards of open defecation. Similarly, a significant increase is noted in terms of better awareness as to the types of hazards associated with open defecation. Figure 12 highlights the proportion of post-KAP respondents, who remembered and were able to recall various hazards related to open defecation.
The extent of change in terms of respondents being able to realize and communicate their desire to make their settlement/community ODF, points the success of BCC. Results indicate significant increase across all provinces (refer Figure 13).
Another measure of success is the overall decrease in number of people defecating in open, refer Figure 14. The numbers have come down from 68% to 47% overall, with Punjab and Sindh showing the most significant changes. The results are negative for FATA, which are due to limited data collection at baseline stage and need to be probed further. Limited data collection in Mohmand Agency of FATA was due to security reasons.
14%
34%
54%
24% 10%
54% 62%
77%
47%
18%
0%
20%
40%
60%
80%
100%
Awareness Message about
Hazards of Open Defecation
Dirty Surroundings Diseases Environmental Problems
Health Issues Related To Women
Hazards of Open Defecation
Awareness of Hazards of Open Defecation
Baseline Evaluation
Figure 12: Awareness of Hazards of Open Defecation
14% 14% 24% 25% 1% 10%
61% 73% 93% 84%
32% 39%
0%
50%
100%
Total Punjab Balochistan Sindh FATA KPK
% of targeted population aware of the need for achieving ODF status in their settlement
Proportion of population aware of the need for achieving ODF status in their settlement
Baseline Evaluation
Figure 13: Awareness of the Need for Achieving ODF Status
[41]
Another measure of success comes in the form of increase in numbers for respondents willing to construct a latrine due to the hazards associated with open defecation (refer Figure 15). The respondents willing to construct a latrine have increased from 61% (baseline) to 86%. Majority of those referred to cost-sharing arrangements as preferred option to construct latrines, whereas the numbers for self-construction have reduced from 18% to 10%.
As highlighted above, this trend could be for subsidy factor or for the provision of free latrines by other stakeholders (particularly after floods). However, this merits further exploration as to whether household subsidy is playing a positive role in increasing the sanitation coverage or creating a dependency factor. UNICEF is advised to explore alternative methods for incorporation of equity principles especially in view of the excessive poverty in rural Pakistan. The BCC campaign faced some constraints particularly around short time-frame (for implementation), timely production and delivery of IEC materials and sequenced implementation of BCC campaigns. The IPs underlined that delayed delivery of IEC
83%
56%
85%
47%
70% 68%
44% 39% 35% 34%
76%
47%
0%
20%
40%
60%
80%
100%
Punjab Balochistan Sindh KPK FATA Total
Proportion of Population that Practices Open Defecation
Baseline Evaluation
Figure 14: Proportion of Population Practicing Open Defecation
61% 28% 25% 23%
86%
10% 20%
62%
0% 20% 40% 60% 80%
100%
Yes Self-finance Loan Cost-Share Basis
Willing to Construct a Latrine Due to Hazards
of Open Defecation
Method of Bearing the Cost of Construction
Willingness to Construct a Latrine (Including Methods of Cost Bearing)
Baseline Evaluation
Figure 15: Willingness to Construct a Latrine (including methods of cost bearing)
[42]
materials shortened the time for complete BCC campaigning. This pushed to have additional per week time engagement with the communities (also affected by community’s unavailability during the harvesting season), and accelerated delivery to a degree lead to information overload and reduced absorption time for beneficiaries. Within the mediums used, theatres performances though liked by children and adult men, excluded women as in most cases these were organized at cluster (village) level, where participation required considerable travel – a constraining factor for women for limited mobility. The gender exclusion also constrained mixed gathering for theatres. The next phase may need to consider easier access and if possible more privacy for women (may also consider women exclusive theatre performances). The engagement of religious leaders and institutions proved useful in adding the religious element that appealed to the religious groups, a significant majority in the rural Pakistan. School health and hygiene sessions (plays, puppet shows and others) were found effective in terms of packaging messages in child friendly manner. The use of mass media was relatively less, which may need to be looked into for future.
Output 2: Supply Side Interventions Overall supply side interventions remained effective in terms of meeting the increased demand for supplies. Some of the interventions could have proven more effective given greater clarity and guidance such as involvement of trained masons for hardware (project supported) components, loans and grants for entrepreneurs and marts. The range of supply facilitation interventions included training of masons (for better design and low-cost sanitation solutions) training of entrepreneurs and sanitation mart owners (to encourage business activity and ensure sustained and subsidized supply of materials near community), and construction of demonstration latrines for both demonstrative and equity purposes (by providing subsidy or full costs for vulnerable and extremely vulnerable families). Moreover, it included rehabilitation of child friendly WASH facilities at selected schools and at health service centres. Training of Entrepreneurs, Sanitation Mart Owners and Masons The RSPN developed training contents and trained entrepreneurs and sanitation mart owners, through Master Trainers. The IPs and trainees both point to ineffectiveness of the training, mainly for weak training contents and limited facilitation capacities of Master Trainers - lack of understanding of marketing and supply chain issues. Realizing the gap, UNICEF contracted DSI to develop contents and train the trainees through refreshers (refer results in Table 10). These were apparently more effective in articulating expectations. The key learning from the experience include a full day training and training/sensitization of relevant IP staff to enable productive engagement (post training) with the entrepreneurs and marts.
[43]
In Kohat a local mart owner said that he was dealing in cement, sand and bricks. After the entrepreneurs training, he stocked supplies and was able to sell items for approximately 150 latrines through the voucher system. From this he was able to purchase more latrine supplies and sustain the sanitation business. However, he was of the view that availability of loans/grants would have been helpful, given village level vendors are generally cash-strapped and capital to invest in new products is not available. He also felt that IPs should have focused more on creating demand and publicising their sanitation business to the communities.
Training Events for Mason and Entrepreneurs Actual Target No. of masons trained on sanitation construction and using their learnt skills
692 1,250
No. of entrepreneurs/manufacturer trained and producing low-cost sanitation solutions 692 469
No. of facilitated sanitation marts doing profitable business 346 82
Table 10: Supply Side Training Events for Masons and Entrepreneurs
DSI also established a business helpline for entrepreneurs and sanitation mart owners for all 13 SPSP programme districts as part of active research and support. The helpline was launched to provide the entrepreneurs with the opportunity to seek business related advice from DSI SanMark expert and engineers. The records indicate that maximum numbers of calls were received in October and November 2013, mostly regarding the withdrawal of loan / grant by UNICEF and IPs, financing options, and supply and pricing issues. This service was considered by entrepreneurs and sanitation mart owners to have been effective in helping them with teething problems of setting up the marts. Masons trainings proved relatively less effective for largely being informed of the training contents, before taking up the sessions (as DSI research). The programme may have benefitted more in case it had employed the trained masons to rehabilitate the school and health WASH infrastructure. Financial Support for Entrepreneurs and Sanitation Marts The programme envisaged supporting entrepreneurs and marts with loans, which was not appreciated much. More interest was shown in grants, which ran against UNICEF rules prohibiting use of funds for setting up for profit businesses. In view of the resistance, the loan element was withdrawn, which impacted negatively given expectations had been raised that loan/grant facilities may be available. As the component was withdrawn during implementation, where some Provinces were found to have had advanced into implementation. For those, UNICEF created modality of vouchers and distributed to the most vulnerable/vulnerable groups to get supplies from the supported marts and entrepreneurs. This to a degree proved useful in terms of localized availability of supplies and at subsidized rates. With programme closure the voucher system has stopped, and this shall have affect on their businesses. The component needs deeper analysis and more innovative approach to facilitate both the users and the business involved. UNICEF could leverage the findings of DSI action research.
[44]
Design Options and Manuals for Low-Cost Sanitation Facilities UNICEF engaged DSI to help develop low cost latrine designs. DSI is a consulting company (for profit), contracted by UNICEF for the work. DSI used Human Centred Design (HCD) process to developed latrine design, which are as below (refer to Table 11) Type Description Cost Factor (maximum)
Latrine Type A Ventilated Brick-Lined Direct Pit Latrine
PKR 4,000
Latrine Type B Pour Flush Off-Set Lined Pit Latrine
PKR 7,000
Latrine Type C Pour Flush Latrine With Septic Tank
PKR 10,000
Table 11: Types of Low Cost Latrines
For field interventions, another manual i.e. Latrine Construction Manual, was developed for sub-engineers, masons and local facilitators in local (Urdu) language. This manual was used for field training and for project supported hardware interventions. The document was found comprehensive and incorporated disability and age considerations. The evaluators did technical assessment of the manual and deeply looked at constructed latrines, which are outlined below as key observations/comments.
The latrine construction manual is comprehensive, however it be more useful if additional information regarding latrine super-structure options as well as low cost options for pit lining (using local materials) could be added. Translation in Urdu contributed to improved delivery and understanding of trainees.
DSI’s report lacks design recommendations related to disabled/elderly persons’ needs. The manual however covers these elements comprehensively. It appears that the rehabilitation work overlooked the manual prescriptions around designs.
Communities (in hot zones) referred to latrines being small in size, low roof and low walls, hence lead to discouraging some for fear or perception of lack of privacy. Although designs prescribed local adaptation, which to a greater degree not been complied with. UNICEF and IPs are advised to adapt design with respect to roof height, larger ventilation points and higher side walls.
The prescribed pit size for the demo latrines highlighted being insufficient by the communities. There are reported cases where community tried to dig bigger pits, however the cover provided (by the project) found to be of smaller. This alludes to the fact that in some cases the process of agreeing a participatory design with community was not followed.
Life cycle cost analysis of the latrine options also seems not to have been done. Depending on the material used, design life of the latrine superstructure is expected to vary considerably.
SPSP aims to improve peoples’ access to safe drinking water and sanitation. It merits further researching as to what impact increased sanitation coverage has had on water quality and accessibility in target.
Design of hand wash facility needs further clarity and needs to be communicated to the IPs particularly in areas where water is being accessed
[45]
through hand pumps. It merits consideration whether to have separate hand wash facility or not, and if is not required then suggest what actions to take.
School WASH facilities were found not being disability friendly both due to stairs leading up to the hand wash facility, as well as the design of the hand wash facility itself, which in most cases requires children to squat and reach out for the tap.
The evaluators did use an observation checklist (in line with standard design guidelines and standards followed worldwide). Significant deviations were noted and in some cases communities and teachers shared that for limited school space, the compliance to the set standards (for instance the minimum or safer distance prescriptions) was not possible anyways. Local Context and Issues The evaluators found quite a few non-functional demo-latrines, which amounts to lack of investment (cost sharing required) from communities and in some case discouraging attitude from the landlords (in Sindh particular). Some of the community feedback articulating the reasons for non-functioning of demo-latrines is outlined below (all by and large point to non-compliance to prescribed participatory design process):
Some shared that even if the sides of the latrines are covered with cloth/provide some form of superstructure, the females were reluctant to use for privacy reasons for low height of boundary walls.
Some referred to land ownership issues where landlords (particularly in Sindh) did not allow the tenants to construct latrines on their lands by tenants/ workers. There was visible reluctance from the landlords to allow seasonal tenants to construct permanent latrines on their land.
The implications of the points highlighted above include women being dissuaded to use the latrine due to lack of privacy and a possibility of conflict with landlords. The larger implication is that, a facility that has been built is not used, but still counts to be contributing in efforts to achieve ODF targets. The IPs may need to be advised to integrate local context and follow the prescribed process to have people’s views before such interventions. This is likely to contribute to continued and effective use of the facilities or investments made. For latrine construction, outputs exceeded than the targets (see table 12).
Latrines Constructed under SPSP Target Actual
No. of latrines (other than demo latrines) constructed by masons trained under PATS programme
13,857 20,040
No. of complete demonstration latrines constructed for extremely vulnerable families as per agreed criteria with community (maximum 3%)
4,157 5,212
No. of demonstration latrines constructed through materials provision/vouchers for vulnerable families as per agreed criteria with community (max 7%)
9,700 8,581
Table 12: Number of Latrines Constructed (Target versus Actual)
For supply facilitation (as noted in Figure 16), the results indicate an overall 39% of the respondents found using latrine facilities constructed with external support in the form of subsidy, loan, materials or trained skilled labour (please note that for this indicator the baseline value was assumed to be 0%).
[46]
The results indicate increase in people claiming to have constructed latrine in last year. The numbers have gone up from 26% to 47%, which is significant. The respondents shared that supplies facilitation as major support is available for latrine construction (only for those who had constructed latrines last year). .
The key challenges include withdrawal of grant/loan for sanitation mart and entrepreneurs. This affected the supplies facilitation element negatively; however the positive results (around latrine construction and availability of materials) indicate that existing mart owners were able to meet up the increased demand. The increase for availability of materials and skilled labour both point to larger effectiveness of the interventions (see Figure 18).
The programme may need to evolve more effective and both user and entrepreneur friendly solutions for this. UNICEF may need to look into DSI’ work for this including work in the region.
39% 45%
17% 18%
53% 55%
0%
50%
100%
Total Balochistan FATA KPK Punjab Sindh
Proportion of Population Using WASH Facilities Constructed With External Support
Total Balochistan FATA KPK Punjab Sindh
Figure 16: WASH Facilities Constructed with External Support
26%
4% 2% 8%
47%
12% 6%
31%
0%
10%
20%
30%
40%
50%
Latrine Constructed in the Last One Year in Respondents House
Demo Latrine Cost Share Supply Facilitation
Type of Support Received by the Respondent for latrine construction
Figure 17: Latrines Constructed By Respondent over Last Year and Type of Support Received
[47]
Overall the proportion of people having access to and using improved drinking water source has improved from 76% to 82% (refer to Figure 19). This points to the success of programme with respect to increasing access to drinking water (excludes water quality).
School WASH Facilities Post-KAP results show a positive trend in all key aspects, including 97% access to latrine facilities and 79% access to water facilities (in some cases new water schemes were not installed for poor water). The results indicate a significant increase in availability of soap close to larine, which increased from 11% to 30%. This points to huge space/need and potential for improvement in this regard.
16%
53%
27%
60%
0%
10%
20%
30%
40%
50%
60%
70%
Availability of Sanitation Supplies in the Village Availability of Trained Skilled Mason/Labour in Village to Construct the Latrine
Figure 18: Availability of Sanitation Supplies and Trained Masons
76% 60%
82% 82%
53%
96% 82%
68% 86% 85%
66%
99%
0% 20% 40% 60% 80%
100% 120%
Total FATA KPK Punjab Balochistan Sindh
Proportion of the population using an improved drinking water source
Population Using an Improved Drinking Water Source
Baseline Evaluation
Figure 19: Population Using an Improved Drinking Water Source
[48]
The delivery mechanisms must have given consideration to the customary summer holidays, which apparently was not given much thought into while laying the plans. In line with the ‘Islamabad Commitment (2010)’, signed by all provinces for Child Friendly Inclusive Schools, the programme did effort and largely remained successful to improve school WASH facilities to make them child friendly9. The programme prioritised girls’ school for WASH facilities rehabilitation. It was found more relevant for the fact that culturally availability of safe and secure facilities at schools is relatively more significant for girls. The results indicate better performance in girls’ schools compared to boys such as availability of soap, though still insufficient. This merits greater attention from teachers and availability/use of SMC funds to ensure continued availability of soap, as well as proper functioning and cleaning of the facilities. Positive developments in this regards include Punjab government’s approval to use SMC funds for repair and maintenance, and the change in resourcing criteria for SMCs in Sindh from a fixed yearly amount to a per child allocation basis. Despite policy changes, the actual impact in terms of funds getting spent is quite limited. Often these remain unused or under spent for procedural reasons (possible audit objections, low engagement and capacities of SMCs, limited interest of school head teacher and others), hence the future programme design may need to prioritize addressing those procedural bottlenecks.
9 According to ‘Development of a framework and an action plan for school sanitation and hygiene education’ (SSHE) in Punjab, Analytical Report on Current
Status of School WES in Punjab School; Final Report December 2007. UNICEF Punjab, child friendly facilities have the following attributes,
easy and comfortable seating
proper ventilation and light arrangement
appropriate space inside the toilet
special non-slip ceramic tiles in pour flush latrines
provision for looking glass (mirror)
suitably designed hand washing points
appropriate height of door knobs/locks
appropriate height of hand washing basin
provision of shelf and hook to safely keep belongings while using toilet
provision for children with special needs
74% 53%
31% 11%
97% 79% 77%
30%
0% 20% 40% 60% 80%
100%
Schools with Latrine Facility
Schools with Drinking Water Points
Schools with Hand Washing Facilities
Schools with Soap Close to the Latrines
School WASH Facilities
Baseline Evaluation
Figure 20: School WASH Facilities
[49]
85% 97%
77%
23%
74%
95%
74%
26%
65%
100% 80%
45%
0%
20%
40%
60%
80%
100%
Availability of Drinking Water
Availability of Latrine Facility
Handwashing Facilities Availability of Soap
Boys Co-Education Girls
Figure 21: Availability of WASH Facilities in Boys, Girls and Co-Education Schools
[50]
Output 3: Health and Hygiene Promotion (H&HP) The results indicate largely positive performance for health and hygiene promotion component. However, for some indicators the results are not very encouraging.
The following table illustrates the mixed performance for some of the health and hygiene activities.
Description Target Actual
No. of people reached with WASH BCC messages through electronic/print media. 970,000 609,530
No. of people reached with WASH BCC messages through community events (theatres, puppet shows). 183,300 165,092
No. of school children (boys and girls) reached with messages through SLTS activities. 63,941 154,526
No. of people (men, women, boys, girls) reached with WASH BCC messages through CRPs. 291,000 846,444
Table 13: Outreach of Health & Hygiene Messages through Various Mediums
Overall the results are positive with respect to increasing knowledge and bringing positive attitudinal and behavioural changes. For instance, water treatment and storage practices show improved results where those treating water (through multiple tools) increased from 4% to 13% (a three-fold increase). Similarly, those storing water has increased from 74% to 91%. Balochistan province leads in terms of extent of change followed by Punjab (see Figure 22 & 23).
4% 13%
2%
41%
18%
0%
26%
74%
13% 23%
10%
48%
1% 5% 13%
91%
0%
20%
40%
60%
80%
100%
Water Treatment
Boil Add Bleach & Chlorine
Strain it through a
Cloth
Use a Water Filter
Solar Disinfection
Let it Stand & Settle
Water Storage Practice
Water Treatment Methods '
Figure 22: Household Water Treatment and Safe Water Storage
5% 5% 9%
3% 0% 8%
13% 18%
34%
10%
2%
11%
0%
10%
20%
30%
40%
Total Punjab Balochistan Sindh FATA KPK
% of respondents who treat drinking water
Baseline Evaluation
Figure 23: Proportion of Respondents who Treat Water
[51]
Another measure of success and effectiveness of H&HP is increase in the number of households reporting availability of soap in the house. Similarly, there is an increase in designated hand washing places (available). The increases indirectly points to heightened awareness around hand washing and linkages to diseases (refer Figure 24 for details). The availability inside or near latrines have not increased much though. This can be attributed to the fact that people in rural areas tend not to leave the soap near or inside the latrine for risk of dissolution (waste).
The encouraging increase in hand washing practice at critical moments, points to effectiveness of H&HP. The results are particularly more encouraging for post defecation (refer to Figure 25) compared to other elements such after urination, cleaning child etc.
19%
47%
27% 32%
60%
25%
0%
20%
40%
60%
80%
Separate Handwashing Place Anywhere in the dwelling Near/Inside Latrine
Soap
Hand Washing Place & Soap Availability
Baseline Evaluation
Figure 24: Hand Washing Place & Soap Availability
50%
29% 20%
15% 15% 10%
62% 68%
22% 14% 12%
7%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Before Eating After Defecation
After Urination Before Cooking After Cleaning a Child that has
Defecated
Before Breastfeeding a
Child
Respondents Practice of Hand Washing at Critical Moments
Hand Washing at Critical Moments
Baseline Evaluation
Figure 25: Critical Moments for Hand Washing
[52]
The effectiveness is evident from encouraging increase on awareness around water related diseases. The heightened awareness however has not contributed to any significant reduction of diarrhoea incidence though. This could have the seasonal element also; as numbers increase in summers anyways (refer Figure 26 for details).
52% 28% 27% 22% 27%
79%
44% 44% 33% 24%
0% 20% 40% 60% 80%
100%
Diarrhea Cholera Hepatitis Skin Disease Incidence of Diarrhoea in Last
Two Weeks
Knowledge of Diseases Caused by Using Unsafe Water '
Knowledge of Water Related Disease and Incidence of Diarrhoea
Figure 26: Knowledge of Water Related Diseases and Incidence of Diarrhea
[53]
Output 4: Linkages with Duty Bearers The level of interest and engagement of public sector entities varied significantly across provinces as well as IPs. However generally there was not much success with respect to fostering ownership of PATS and mobilizing resources for planned rewards/incentives. The district governments, however, helped with selection of villages, identification of schools and health facilities which needed rehabilitation, subsequently they were engaged at the ODF certification process, involving a range of government officials from the departments (LGRDD, PHED, Education and Health). Overall the interventions still contributed to raising knowledge and awareness of public officials both at the provincial and district levels though the extent of it merits further investigation. Bottlenecks identified through the provincial workshops as part of the baseline did not translate into activities to address them; this may need to be prioritized in future programming. This component aimed at developing linkages with duty bearers and public institutions to support PATS to positively influence budgetary increases (at provincial level) for WASH services, encourage PATS adoption in the public sector and finally leverage public funds as rewards for villages turned ODF. Multiple activities carried out for this and largely the results are encouraging (refer Table 14). Description Target Actual Percentage
Capacity Needs Assessments (CNA) 14 6 43%
Capacity Development Plans 14 6 43%
Capacity Development Events 14 15 107%
Number of Government officials and political leaders that participated in capacity development and key events of the programme.
185 230 124%
Advocacy initiatives undertaken with duty bearers/political leadership to support PATS
14 17 121%
Villages jointly selected for PATS programme by relevant Government duty bearers with IP
1,385 1,791 129%
ODF communities received rewards/incentives by government 39 62 159% Table 14: Capacity Building & Advocacy for Government
The level of interest and engagement of public sector entities i.e. Public Health Engineering Department (PHED), Local Government (LG), Health and Education departments, varied significantly across provinces. For limited guidance and standardization, the focal points in the public sector were found to be different for different IPs (for some it was DC office, others worked with either PHED or LGRRD). The IPs were found to be connected to bottom tiers of government officials. In some cases, where the (public sector) focal points were more sensitive and aware of potential NGO contributions, showed greater interest, ownership and leveraged the partnership for common objectives. The district governments helped with selection of villages and education and health authorities facilitated identification of schools and health facilities needing rehabilitation. The interventions did contribute to raising knowledge and awareness of public officials (particularly around PATS), both at the provincial and district levels. To what extent to which capacities been raised and contributing to improved public services is unknown and merits investigation.
[54]
The achievement of leveraging public resources both in terms of increased budgetary allocations (at provincial level) and mobilizing funds for rewards at districts are insignificant. This again reflects lack of planning and guidance for the intervention area (more details in Figure 27).
The frequent transfers of the public officials also did affect the efforts, as those who were sensitized and trained got posted elsewhere, leaving IPs to work with new officials with no/limited exposure to programme and areas where it needed their support. This element together with short time frame (requiring focus on delivering outputs) apparently work to discourage IPs to work more closely with local officials. Bottleneck analysis workshops conducted at provinces as part of the evaluation, which were attended by LGRDD, PHED and Education Departments. It was noted that not much progress been made with respect to actions identified in the bottlenecks carried out during baseline. This may need to prioritize for the coming years.
0.28% 0.39% 0.37%
0.12%
0.00%
0.50%
KPK Punjab Balochistan Sindh
Government expenditure on WASH as a proportion of total government expenditure
Figure 27: Government Expenditure on WASH
[55]
Output 5: Attaining Total Sanitation The programme appears effective with respect to achieving the targets for transforming the targeted communities or villages into ODF. This marks progression on the sanitation ladder. There are, however, concerns as to communities being able to sustain the ODF status. The table below presents the reported results with respect to the total sanitation component. The programme has achieved (overachieved) on all sub-indicators. Villages supported in attaining Total Sanitation status Target Actual
No. of households with new latrines constructed after start of the PATS programme 17,182 33,673
No. villages declared as ODF by community 831 1,436
No. of villages Verified as ODF by Implementing partner and independent monitors 831 1,250
No. of people (men, women, boys, girls) in communities declared as ODF 581,700 711,467
No. of people (men, women, boys, girls) in communities verified as ODF 581,700 679,967
No. of model villages with laid sewers, WW collection and treatment system through PATS programme 4 18
No. of advocacy initiatives undertaken to promote communal underground drainage system linked WW treatment system and communal SWM system 2 22
No. of ODF villages provided with underground drainage and WW treatment facility by relevant public sector stake holder/duty bearer 2 0
Table 15: Villages Supported in attaining Total Sanitation Status
This performance however comes with caveats around sustainability. In general, there is an understanding that the focus was on delivery rather sustainability and for shortage of time the IPs were left with no time to monitor and to sustain ODF status in the certified villages. The communities have started showing signs of slippage. The programme has also been able to demonstrate healthy increase overall in communities/households connected to under-ground drainage (details in Figure 28). The evaluators did however notice significant variations in terms of what constitutes drainage system.
27%
52%
0%
15%
35% 31% 44%
65%
33% 43%
58%
24%
0%
20%
40%
60%
80%
Total FATA KPK Punjab Balochistan Sindh
Households Using Underground Drainage
Households Using Underground Drainage
Figure 28: Households Using Underground Drainage
[56]
A CRP told the evaluation team that in Uthman Khel village a household latrine was being drained directly on to the street. Due to this the area was very dirty and no one was able to walk past that part of the street. After the PRA exercise the people contributed through VSC to construct drainage / sewer along the streets. Therefore, the streets are now clean.
The evaluators noted heightened
awareness evident from the identification
from the communities themselves that the
latrines with single pits designs; do not
seem to have taken note of the local
demographic context (where people living
in joint and extended family arrangements)
and the single pits may fill in very quickly.
Without adequate cleaning/disposal
arrangements, overflowing pits may cause
of pollution and spread diseases.
[57]
Output 6: Knowledge Management and Accountability The knowledge management component being an integral part to the programme design and delivery has contributed to critical learning around use of subsidy, sanitation marketing and low-cost sanitation options. The third party monitoring mechanism instituted for the programme proved to be particularly useful in real time monitoring and complemented decision making. The new learning resulting from this phase of the programme have already been fed into the design and modalities of the next phase of SPSP, and offers the opportunity and potential of informing a PATS review. Overall, the component contributed to improving the programme implementation and interventions by informing management and programmatic decisions through real time monitoring of programme processes, challenges and issues whilst helping in course correction. The component also focused on programme learning and documentation for future design, possible replication and scaling up through several lesson learning workshops, baseline and evaluation studies and action research. The achievements for the components are presented in the following table. Knowledge Management and Accountability Actual Target
No of Inception workshops conducted in the beginning of programme 12 12
No. of villages with baseline established in the beginning of programme by IP 1,385 7,448
No. of joint progress monitoring reports (IP and public institution) 80 39
No. of monthly substantive monitoring reviews’ visits conducted for confirming/measuring performance indicators in results framework 80 216
No. ‘monthly substantive monitoring review’ reports shared among partners 63 55
% of responsive progress reports submitted as per MEAL framework by IP - 33
No. of quarterly progress reviews conducted with IP and minutes documented 2 15
No. of mid-term review workshop report endorsed by UNICEF 1 4
No. of Human interest stories documented by IP and accepted as per UNICEF guidelines 80 78
No. of end line survey reports accepted by UNICEF 11 2
No. of Lessons sharing workshop conducted and feedback documented 1 1
Table 16: Knowledge Management and Accountability Events and Activities
Third Party Monitors (contracted for process improvement, verification and feeding into management decision making) work was a major factor in redesigning programme activities based on ground realities that emerged over the course of programme implementation. This monitoring process allowed UNICEF to monitor and oversee ground implementation in real time and push for more effective and efficient implementation through IPs (particularly those that were lagging). It also helped inform the management of the need for corrective measures where weak performance of IPs was spotted. Third Party Provincial Monitoring meetings (with IPs and UNICEF) also allowed for active and effective debate that helped keep the project on track. Monitoring was of four types, planned, verification purposes, spot checking and follow-up. Regular monthly reviews helped in making the programme more effective whilst ensuring efficiencies were achieved. Whilst most of the Third Party Monitors work proved effective, there was some feedback that at times monitoring took the form of policing. Moreover, there was confusion in respective roles of several firms contracted for the range of monitoring.
[58]
These aspects need to be clarified and addressed in the next phase and in future programmes. Given the delayed start of the project, tight timeline for activities and an ambitious monitoring schedule, more focus and emphasis remained on monitoring and less on knowledge management and documenting of key learning during the programme implementation period. However, the component did contribute to knowledge management by commissioning useful baseline and end of phase evaluation. The baseline findings and learning have been fed into the implementation of Phase 1 of SPSP, whilst the evaluation findings and lessons learnt have been used to inform the design and interventions of Phase 2 (during the presentation and workshop with the Evaluation Resource Group, UNICEF Staff and other Stakeholders). The component also made available data and evidence on the use of PATS in Pakistan, with a potential of using this information in a structured way to inform PATS and sector stakeholders of the experiences, challenges, and successes of SPSP implementation. Information is now available, however it needs to be consolidated, disseminated as well as used for a possible review of PATS based on the experiences, and lessons learnt through the implementation of SPSP Phase 1, particularly around low-cost sanitation options, use of subsidy for household latrines, and sanitation marketing. Moreover, the action research commissioned to DSI should also be proactively shared with stakeholders to help inform their interventions.
[59]
5.2.4. Stakeholders Contribution10 Multiple stakeholders contributed to the successful completion of year 1 activities. UNICEF has largely been successful in awarding roles (while taking note of partners strengths), creating synergies, complementarities and coordination mechanisms, and documenting the expectations and responsibilities between stakeholders. The review of the programme achievements indicate that most have been able to deliver on expectations. The description below captures the key contributions, successes and challenges (or opportunities) with respect to the key stakeholders involved. UNICEF (Islamabad and Provincial Offices) provided effective programmatic and management oversight to ensure that the programme was delivered according to the specific PCAs with IPs within the available resources. The number of actual beneficiaries as well as villages exceeds the planned targets. UNICEF also put in place a robust monitoring function for the overall programme which helped inform better decisions and course correction during implementation. Throughout the programme, UNICEF offered a range of capacity building interventions which helped develop the community’s capacity, build a cadre of PATS master trainers, increase capacity of non-profit entities as well as government departments (though the focus on government capacity building could have been greater) to implement sanitation responses. Though it is difficult to attribute all improvements and positive policy measures taken at the federal and provincial level to one stakeholder i.e. UNICEF, however there is evidence in the form of Provincial Government’s Master Plans that incorporate PATS model, as well as in some cases where government allocations to WASH sector have increased. However engagement with government needs more concerted effort from UNICEF Islamabad as well as Provincial Offices particularly in resource mobilisation for PATS. Other UNICEF contributions included a focused emphasis on Knowledge Creation including project and sector specific training manuals, IEC material, training documentary, producing thematic issue papers, focusing on issues such as Menstrual Hygiene Matters (MHM) research and WASH Equity study. Lastly, UNICEF’s contribution has been of raising sufficient funds for the two programme contexts i.e. follow up in flood affected areas as well as funding for WASH sector in the insecurity affected areas. UNICEF’s contribution had an impact across all the outputs of the programme. DSI’s specific contributions relate to sanitation marketing, design manual for latrine facilities and focusing on action research in SPSP pilot area. Sanitation marketing is an evolving process in WASH sector in Pakistan, and DSI demonstrated their flexibility in capacity building interventions during SPSP where circumstances necessitated that they provide recurring capacity building support. Moreover, offering a business advice phone service to the Mart Owners and Entrepreneurs specifically contributed to building the confidence as well as helping through the initial teething issues. DSI’s contribution resulted in building and sustaining demand through supply side interventions.
10 TORs required the evaluation to focus on a sample of partners; there was no requirement to conduct a comparative analysis of IPs.
[60]
RSPN’s capacity building activities (in particular social mobilisation training, use of PRA and mentoring activities) played a critical part in the delivery of results as it helped a range of non-profit organisations to deliver the SPSP programme across geographic regions using a uniform implementation methodology to carry out SLTS and CLTS. Some points have been highlighted above regarding changes that can be incorporated in the training, mentoring approach as well as social mobilisation to ensure it is more effective for future phases, however in the context of Phase 1, the activities were carried out effectively both in terms of number of activities and beneficiaries as well as in terms of delivery of results. RSPN’s contribution resulted in generating demand for sanitation. APEX contribution to the programme remained its regular monitoring reports and feedback which informed real time decision making to keep the programme on course both in terms of its delivery of outputs and achievement of results. Recommendations have been made to address issues that were highlighted in reference to monitoring i.e. rigidity of approach, which if addressed will further improve the role of third party monitoring in the next phases of the programme. APEX’s direct contribution is to the knowledge and accountability output, however due to their regular monitoring the results of their contribution have an across the programme impact for all output areas. Government Departments also played a contributory part although with substantial variation across geographic regions, i.e. in some cases close coordination and engagement with the project, but through lower tiers of the departments (e.g. Dera Ghazi Khan District), whilst in other areas limiting departmental engagement to only certification of ODF villages through senior district officials e.g. ADC in Jacobabad. However this allowed government departments to become familiarised with the PATS approach as well as to the issues on the ground in the villages. Across SPSP, in certain parts where government was more responsive, water supply schemes have been rehabilitated, whilst in other areas government allocation in its budget to WASH sector has increased, however direct attribution to engagement through SPSP is difficult to do.. Government’s direct contribution has been to the linkages output, with community-government, non-profit –government department links being formed to deliver PATS. IPs (range of non-profit organisations) contribution has been across the results framework in all output areas. The extent, degree as well as success varies across the IPs, however all have been able to create and sustain demand for sanitation, while also ensure supply facilitation, health and hygiene promotion and total sanitation results were also attained. Specific areas have been highlighted in the findings section as well as more specific detail is provided in the lessons and recommendations section to help improve IPs delivery and impact for the next programme phase. In summary the required number of activities and outputs were delivered, benefitting a larger number of villages and beneficiaries than targeted, with many result indicators showing positive trends i.e. increased access to sanitation facilities (for men, women, children as well as an improvement in the equity context). Similarly ODF declaration, verification and certification have also been achieved though slippage will need to be monitored and controlled.
[61]
Community stakeholders (CRPs, VSCs, Children’s School WASH Clubs, Teachers as well as masons, entrepreneurs and sanitation mart owners) have played the most vital part in helping achieve the results. The various community stakeholders stated have actively participated in the programme activities, given their valuable time to the interventions, facilitated IPs, APEX and UNICEF. Moreover, their willingness and acceptance of wanting to improve their and their community’s lives helped make this programme an effective one that also achieved its targets. Their contribution can be mapped across the range of the outputs.
5.2.5. Gender Equality, Equity & Disability In line with UNICEF policies, the programme design and interventions have incorporated principles relating to gender equality, equity and disability. In terms of gender, the programme planning and implementation demonstrates balanced focus on men, women, girls and boys in terms of having balanced field teams, creating of women/men VSCs, children clubs, and CRPs, gender and age focused BCC strategies and interventions, prioritization of girls’ schools for rehabilitation of WASH facilities. The focus on equity is evident from use of PRA for poverty ranking, and provision of subsidies to the most poor. However the integration of disability remained weak as evidenced in the construction and access aspects of WASH facilities. The selection of schools both for boys and girls, Social Organizers, Community Resource Persons (CRPs) and Village Sanitation Committees (VSCs) ensured participation of men and women in the programme. To address differential information needs, access to information sources and physical accessibility -- mainly for cultural seclusion between sexes -- the programme evolved responsive BCC strategies in terms of content, design and medium of transmission, e.g. female CRP sessions, and theatre performances for children. The programme was also in line with the needs identified by women, including children’s health due to the unhygienic conditions as well as hazards of open defecation, difficulties faced by women particularly due to privacy needs which force them to defecate either very early in the morning or to hold until it is evening causing health complications. It was also mentioned that going at night exposes them to a range of risks including being bitten by insects or attacked by animals, to even being harassed and physically attacked by men. The programme design therefore clearly identified gender considerations such as appropriate design of latrines for women and girls to ensure privacy and safety, appropriately designed water collection points as well as role in collecting water. Gender disaggregated data was effectively collected and used in the program. The programme design also effectively addressed equity considerations by including provision of low-cost environment friendly and culturally appropriate demonstration latrines for the poorest of the poor communities. The programme had a 3% (for the extremely vulnerable) and 7% (for the vulnerable) approach to address the sanitation needs, whereby the extremely vulnerable received subsidy for the whole latrine, whilst the vulnerable had to construct the super structure using their own resources. SPSP design also considered addressing the needs of the disabled; however, this approach lacked clarity in terms of expectation and implementation.
[62]
From a gender perspective the results of the post-KAP survey demonstrate that women’s access to latrines has increased by 30%. Moreover in numerous interviews and meetings with a range of stakeholders (both men and women), the evaluators were told about instances of violence as well as harassment towards women due to their vulnerable situation while going out at night for defecation. In the gravest of situations, false allegations of sexual indiscretion have even led to instances of Karo Kari.
In terms of equity, the lowest and the 2nd Quintile seems to have benefitted most in terms of access to latrines as intended by the programme. However, as highlighted above household subsidy as a method to respond to equity needs will need to be revised or replaced with innovative methods. Lastly in terms of the disabled, access to household latrine has increased to 66% during the evaluation as compared to 44% during the baseline; however whether these latrines cater to the special needs of the disabled is not confirmed, given this component remained weak during the design as well as implementation stages. Moreover, there is substantial evidence that suggests that the programme’s successes, as secondary impact, have also contributed to reinforcing a gender divide. For instance, the programme interventions have added to the workload of women and children being responsible for cleaning latrines and fetching additional water required due to the introduction of flush latrines. There has been some feedback on the DRR designs of the latrines as well as water facilities which have been designed to be elevated, and which have reduced the privacy factor for some. This is of particular issue where such facilities have been constructed without taking community into confidence about the design aspects of the facilities; otherwise an alternate site for the facility could have been facilitated by the community.
44% 34%
27% 32%
13% 7% 9%
66% 64%
40% 37%
12% 5% 6%
0% 10% 20% 30% 40% 50% 60% 70%
Disabled Having
Access to Latrine
Women Having
Access to Latrine
Lowest 2nd 3rd 4th Highest 5th
Income Quintile
Gender, Disability & Equity Based Access to Latrines
Baseline Evaluation
Figure 29: Gender, Disability and Equity Based Access to Latrines
[63]
Also there is however a strong perception by a range of stakeholders (CRPs, VSCs, Government Officials) that having access to latrines has reduced instances and opportunities for harassment and violence against women that was always a possibility when women were defecating in the open particularly given the privacy needs at night time.
[64]
5.3. Efficiency11
Evaluation Q3: To what extent have the outputs of the project been achieved (in terms of quality and quantity) with the allocated resources/inputs (such as funds, time, and procedures)? The evaluators have looked at the element of efficiency in terms of progress or performance against key results, adequacy of resources (time, human and financial) and coherence to the set/available quality standards. The evaluators did do value for money analysis with respect to cost per beneficiary (for varied results) and basic analysis around costs for partnership management. The programme has achieved efficiency in terms of performance against target indicators, Cost per beneficiary and villages achieving ODF declaration stage demonstrates efficiencies against the planned costs. However, some indicators underperformed, while concern around planning, target setting and best use of resources is raised on indicators that substantially overachieved their targets (this leads to the understanding that either there was a cost and resource implication or a quality impact). Project results indicate adequacy of human resources, however the results exceeded by a number of times pointing to gross miscalculation as to planned targets and resources required (financial and human). Instances such as contractors charging the same amount of funds for building new and rehabilitation old school wash facilities suggests areas of financial inefficiencies. Highly variable cost of latrines raises the question of affordability for the vulnerable across SPSP programme areas. Performance against Target Indicators The efficiency analysis with respect to performance against key indicators suggests that the IPs have been able to achieve several of major targets. The Programme Output Matrix (updated version available as of March 2014) shows that out of 60 output indicators, the programme exceeded targets for 41 indicators. Furthermore, within these there are 10 output indicators, where results exceeded by over 300%. These however, raise concerns around set targets being reasonable, over-budgeting and given these were achieved in shorter duration, the quality and sustainability of the results. However, for some indicators the performance has been below par. For example, reaching out to beneficiaries through media & theatres, provision of rewards and incentives (with the target to provide 158 critical community infrastructure schemes as incentives/rewards, only 2 were delivered). Furthermore, two programme components indicate little progress i.e. provision of loan and provision of underground drainage. This point to limited success for fostering partnerships (with relevant public authorities) and mobilizing public resources.
11 Note: The evaluators’ ability to conduct a detailed analysis was constrained by the fact that project expenditure sheets were unavailable (not compiled by the time evaluation was in progress). The calculations and analysis done below relies on budget estimates rather actual expenses, drawn either from PCAs or UNICEF submissions to DFID (UK).
[65]
Financial, Human & Time Resources The review of programme performance for key output indicators suggest satisfactory performance, and this at another level points to adequacy of resources to carry out tasks. However, the fact that the programme has exceeded most of the targets by 20% to 100% raises concerns around realistic costing by the IPs (refer Annex 13: Programme Target versus Achievements). The evaluators noted some costing oversights where they found that contractors (for rehabilitation of WASH services) charging the same amounts for building new facilities as for the rehabilitation of old ones at school and health outlets. Some IPs communicated that the umbrella arrangements contributed to raising operational expenses (where both umbrella and implementing partners charged operational expenses), which points to the inefficient use of resources. The staffing arrangements were found to be adequate by and large. The field staff especially SOs did raise concerns of the availability of time for quality work as mobilization and exit planning. To them, this contributed to pushing them work extra hours to meet the targets. Some IPs referred to this as key factor for high staff turnover. SPSP Phase 1 was designed for a period of one year. The field level implementation in some cases started with a delay of up to 4 months (PCAs with IPs were signed all through February to June 2013) which allowed some IPs implementation period of only 6-8 months against the planned one year. Despite the fact that IPs were able to achieve most of the set targets, there are concerns with respect to work planning & time availability (to IPs for implementation). Reportedly, there were oversights with respect to timing activities with community life-cycle (despite the fact that the information was available from village PRA exercises) and with those of public services outlets such as schools and health centres. This to a degree pushed IPs staff to use short-cuts to complete activities e.g. BCC, and left them with little time to plan exit and leave behind arrangements to sustain the achievements and take it further. WASH Facilities Cost Comparison Across IPs and Geographic Spread: The documents reviewed indicate significant cost variations for different interventions such as demo-latrines (varies between Rs. 9,500 - 20,000), hand pumps (from Rs. 18,000 - 69,590). Whilst a price differential is expected across various regions, it raises question on the definition of low-cost in terms of latrines. For instance, the costs range between Rs. 9,500 and Rs. 20,000 across Kashmore and Jaffarabad respectively, though the monthly household income of the lowest quintile across the two districts varies only by Rs. 292, thus questioning the appropriateness as well as cost efficiency for demonstration latrines for the vulnerable in Jaffarabad and likewise for other districts.
[66]
Cost Analysis This section offers cost/beneficiary analysis with respect to different output/outcome level results. The financial information used has been drawn from all the PCAs (Project Cooperation Agreements) signed between UNICEF and the Implementing Partners (IPs) for this phase of the SPSP Programme. Also, it offers comparative analysis with regional countries. Cost Per Beneficiary The budgeted figures indicate that for SPSP Phase 1, a budget of US$ 7.5 million was set aside for PCAs with IPs to reach 1.3 million people (men, women, boys, and girls) in 13 target districts/agencies. Reportedly (refer Indicator No. 2 in Annex 13), the project reached out to 1.2 million people with sanitation messages. Hence, the programme cost per beneficiary comes out to be US$ 6.2 (against a planned cost of US$ 5.8). Total Budget in PCAs US$ 7,500,724
Planned Beneficiaries in ODF villages 581,700
Actual Beneficiaries Reached 711,467
Per Capita Costs
Planned Beneficiaries Actual Beneficiaries
$ 13 $ 11 Table 17: Cost Per ODF Declared Beneficiary
The cost per beneficiary (Table 17) for all components of programmes in a village achieving ODF declaration for actual number of project beneficiaries is US$ 11, whereas the planned cost for the targeted beneficiaries was US$ 13, thus demonstrating efficiency gains of US$ 2 per beneficiary. However, the actual cost per capita for Social Mobilization12 is US$ 1.52 as compared to planned cost per capita of US$ 1.40 for the same, demonstrating reasonable control over planned costs during programme implementation. Cost Per Community/Village Another key result was for communities/villages to achieve ODF status. These include villages which declared (self-declared) ODF status and were verified by IPs and TPFMs.
12 Breakup costs for Social Mobilization Package comprising of salaries and training of SOs & PM, Incentive and training of CRPs, Training of VSC and Training of Govt Officials. This also includes proportionate direct and indirect programme support costs.
[67]
Total PCAs Budget 6. US$ 7,500,724
No. of Planned Target Villages* 1,385
No. of Actual Villages Reached* 1,878
No. of Planned ODF Villages 831 No. of Declared ODF Villages 1436 No. of Verified ODF Villages 1250
(Planned) Cost per village for total target no. of ODF villages
Cost per village to declare it as ODF village
Cost per village to Verify it as ODF Village
US$ 9,026 US$ 5,223 US$ 6,001 Table 18: Cost per ODF Declared and Verified Village
In this context the programme has overachieved both targets i.e. number of people living in ODF settlements, as well as the number of ODF declared villages. This has resulted in a substantial efficiency of 42% in the cost of having a village declared as ODF. In comparison to SPSP, the Total Sanitation and Sanitation Marketing (TSSM) programme in India which was implemented in two states Himachel Pradesh and Madya Pradesh13 with an actual budget of $1.9 million had a cost per person living in an ODF declared village of US$9 only. The programme documents describe the costs promoted for building a new latrine into three categories equalling $38 for category ‘A’ toilet, $70 for category ‘B’ toilet and $99 for category ‘C’ toilet. Using the average cost of these 3 categories, the average cost of constructing a new latrine equals $69. A study conducted by WaterAID for CLTS programmes across Bangladesh, Nepal and Nigeria suggests that the latrine (lined pit, concrete slab and concrete/ceramic pan, brick or concrete superstructure) costs $62-100 in the plain areas and US$150-187 in the hills. As such the three latrine designs are well within comparable costs. Overall, the evaluators feel that the time-lag in approving PCAs did affect the quality delivery of project activities especially for IPs whose contracts were approved late. The over-achievements by 100% or more for few indicators reflect under-targeting, as in most cases those results were achieved within the given time and budgets. Hence, UNICEF is advised to work more closely to set realistic activity targets and corresponding budgets.
13 Water, Sanitation and Hygiene Portfolio Review; Department for International Development (DFID); March 2012, available at https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67433/DFID_20WASH_20Portfolio_20Review.pdf
[68]
6.1. Sustainability Evaluation Q 4: To what extent various stakeholders including the public sector and civil society are likely to sustain the behaviour change related to the goals of the programme after it has ended? Programme design excludes any articulated and expressed exit plan or sustainability strategy. Knowledge, attitudinal and behavioural changes that this programme has been able to bring amongst men, women and children, are the most significant and likely results that may sustain. Whilst the community structures and mechanisms such as CRPs and VSCs are likely to sustain but with limited success. However, IPs found no/little time to prevent the process of slippage following ODF declaration and certification. Linkages with duty bearers component could achieve little in terms of fostering ownership amongst public agencies. The sustainability of sanitation mart owners as well as entrepreneurs created/promoted during the programme is also uncertain due to lack of access to loans/finance as well as low demand issues. Lack of having instituted an O&M mechanism for rehabilitated WASH facilities at schools and health outlets suggests a high probability for them to become defunct in time. According to the PATS approach achievement of an ODF environment (declared or certified) is not sufficient, it needs to be sustained. Moreover, there needs to be an enabling environment at local, provincial as well as national level to ensure long term sustainability. The evaluators have therefore assessed sustainability across a range of aspects i.e. sustainability as part of the design, level of commitment amongst stakeholders to sustain the ODF status in project areas, contribution of SLTS in sustaining ODF status in communities, partnerships established during programme implementation (with government, donors, civil society organizations, and other stakeholders) and lastly government commitment for PATS scale-up. The design excludes any articulated and expressed exit plan or sustainability strategy. Some references are made to sustainability in PCAs, though they appear more of a requirement, as none of the IPs, the evaluators interacted with, shared evolution of exit plan and its implementation. The exit strategy is neither defined nor possible stakeholders identified for sustainability, possible scale-up or even moving up the sanitation ladder. The implication of lack of an expressed sustainability plan or strategy is that there is no standard approach that can be followed and put in place for and by the various IPs. Moreover interventions and project aspects that could otherwise be brought to sustain may be affected by the omission of a focused and planned effort which would have been followed given a strategy/plan had been articulated, put in contract and monitored over the course of the programme. A further implication is leaving the community with unanswered questions e.g. who will continue the work? How other poor people will get access to a facility who did not receive it in the current phase? The programme has however contributed to development of training and IEC materials, latrine guides etc., which are available with partners. The applied
[69]
In response to a question on sustainability and continuity of
WASH Club work, one of the students from Dera Ghazi Khan said
‘ I am graduating to the 6th Standard and will be going to a new school but will continue to share my learning with the children over there
and also volunteer to be part of their Wash Club’.
In another instance, children imposed on their mother to wash her hands before she kneaded the
wheat dough for making roti.
knowledge acquired through practical application of PATS approaches; principles and standards will likely sustain and be used to further build on. In terms of stakeholders, several IPs have expressed interest to continue working in sanitation sector, though this is at their individual level and not attributed to efforts for sustainability through the programme. Moreover continuity and scale of interventions will be dependent on resources available to IPs. Most of the Programme Staff and SOs have also moved on to other programmes; however they will remain available in the market for other WASH programmes, drawing on their SPSP experience. CRPs are referred to as a cadre of human resource for sensitizing communities on the adoption of improved sanitation and safe hygiene practices by PATS. This cadre has been successfully created by the programme, and while all may not continue, feedback from most is that they will continue to raise awareness as it is a social good and in the benefit of their communities. With time this cadre of local resource persons may either themselves or through functional VSCs work on larger community development agenda, including water and sanitation. VSCs have developed CAPs and there is some sense of communal benefit for VSCs to continue beyond end of programme, their ability to sustain as VSCs/CBOs has not been strengthened and most of these VSCs are likely to either disband or become dormant. However the knowledge and experience gained by individual members will still be there and future interventions may be able to revive or strengthen them for developmental purposes. Sustainability of entrepreneurs and sanitary marts is not very likely, this is due to several reasons, first the way they were identified and mobilised (initial concept was to provide loans/grants which was dropped by UNICEF), secondly their innate nature to conduct business activities which don’t carry a possibility of risk, and lastly access to required finance. SLTS has played a complementary role to CLTS; however it is too early to say whether it will help in sustaining ODF status. However the change that has been evidenced in children’s behaviour, increased knowledge and practices will continue as well as be passed on from student to student, to household members and to community in general. Also as observed in the field they will convince their household members as well as community of the significance of health and hygiene as well as maintaining an ODF status. Longer lasting contribution will come through sustainability of Children’s WASH Clubs which depends on continued and proactive involvement of teachers (as WASH Club Supervisors). Therefore, it is important that teachers remain effective and
[70]
receive periodic refreshers. In the longer term the importance of health and hygiene needs to be incorporated into the curriculum, this can be raised by UNICEF and IPs as a policy need in the next phase. Sustainability of physical infrastructure (school latrines and wash facilities, household latrines and community water schemes) is going to have mixed results. School WASH facilities as well as community water schemes depend on continuous operation and maintenance (O&M) provisions without which they are likely to become defunct in time. At school level, O&M provision should have been institutionalised or linked with use of SMC funds through a mechanism; whilst a community based O&M system was necessary for sustainability of communal water supply schemes. Household latrine facilities are likely to sustain until their expected life span (depending on how soon the pit fills up and whether the household is able to afford another pit or cleaning of the first one). The behavioural changes amongst men, women and children, as also noted in health and hygiene section from the post KAP results, are likely not only to sustain but have a multiplier effect on the community that they interact with (including hand washing practices, construction of latrines for family (particularly women)). ODF status is likely to sustain but not at the current level, evaluators field observation as well as feedback from several FGDs suggests slippage (already visible in some villages), this is due to the fact that programme delays in the beginning left little margin for supporting the sustainability of ODF status. Most of the partnerships developed during the implementation phase are not going to sustain beyond project end, without a continued funding relationship. The implication of this will be that several of the IPs may end up not continuing their work on PATS and WASH sector unless they are able to secure funds from elsewhere, reducing the efforts to a projectized approach which lasted only one year, whereas the situation that SPSP is trying to address i.e. access to sanitation and achievement of ODF status is a much longer term effort as also reflected in the Results Framework, therefore it should be UNICEF’s effort to secure a minimum funding commitment for the entire period of the programme, moreover partnerships should continue with the IPs subject to continued funding. A further implication of operating this way is duplication of effort and costs of training IP staff each time a phase ends since the trained staff from previous phase is laid off by the IPs at the end of each phase of the programme given the uncertainty of partnership and funding. However key partnerships that were developed during SPSP Phase 1 include, DSI’s action research work that contributed to PATS learning for UNICEF, and RSPN’s role as master trainer which has been effective and may likely continue in the next phase, though by bringing on board some improvements. In terms of Government commitment to scale-up only Punjab as a province has shown commitment and has taken on the responsibility of sanitation sector with the intent of achieving ODF status. Therefore even in the absence of a stated sustainability strategy several critical areas are likely to sustain. The programme sustainability could have improved had there been more paced time to see through the post ODF status, also involvement of
[71]
school teachers should’ve been more and instead of SO’s providing the training in schools, teachers should have been used. As also highlighted in the PATS, CRPs could have a sustained role to play, particularly where they can pick up entrepreneurship or sanitation mart, given their own incentives, this area should be explored in the next phase.
6.2. Unforeseen Events Evaluation Q 5: To what extent was the project implementation affected and/or influenced by unstable, conflict or emergency situation? Programme design, TOC and PCAs lacked planning or comprehensive risk assessment of the security environment in the programme areas. Fortunately nothing significant happened that would have derailed the programme. IPs knowledge of local communities and recruitment of local staff helped in smooth running of activities. However there is a need to have a Risk/Disaster Strategy in place in future programme design/PCA. Although the programme was focusing on working in flood and security affected areas, the programme design, TOC as well as PCAs lacked any planning or comprehensive risk assessment of the programme areas and overall security environment. There was no detailed natural disaster risk assessment done, however, the latrine construction guidelines contain safety measures to counter impacts of natural disasters such as floods and there is a section detailing DRR in WASH. The PCAs also did not have a risk assessment or mitigation measurement in the event of security problem. Fortunately nothing significant happened during the course of implementation and the programme interventions were not derailed. There were a few instances reported where IPs staff was threatened or attacked e.g. one of the ‘Prepared’ staff member was shot at, in another incident a contractor in District Kohat was unable to complete the latrine construction job due to security situation, similarly programme activities were halted due to security threats in Tank and there was also an issue with Muslim Aid not being able to obtain NOC from the government. Other unforeseen issues noticed were the delays in supplies and release of funds. Generally, even without having any strategy in place the IPs managed to respond to the situation in a manner to reduce any significant impact on the programme implementation. Feedback from the IPs suggests that they used the knowledge of local communities to make decisions on day to day basis, and in most cases having local staff also helped in ensuring smooth running of activities. IPs who had prior presence in district were better able to deal with any such incidences. Considering the above mentioned issues there is a need to have a Risk/Disaster Strategy in place in the programme design/PCAs so that it could be used at the time of implementation. The strategy should have clear protocols to respond to situations. The strategy should be made clear to all and applied from field to office. Similarly, there should be a clear policy on IP staff security, possible insurance and clarity of responsibility and duty of care.
[72]
6.3. Human Rights Based Approach Evaluation Q 6: To what extent did the project observe HRBA and/or contributed to observance of Human Rights in Pakistan as an implicit objective of the project? Ethical Programme Design The design and implementation did not formally articulated ethical considerations though they were adhered to in practice, including protection of the dignity, rights, and welfare of human subjects, particularly children, and respect for the values of the beneficiary community. It is however recommended that ethical considerations be made a formal and written part of design documents as well as implementation arrangements including in PCA documents with partners for future programmes. The evaluation TORs did not specify conducting an analysis of how ethical the programme design was, rather the focus for evaluators was limited to ensuring that the evaluation was carried out under the UNEG ethical guidelines and Guidance Note. However for completeness of the evaluation process consultants have reviewed the programme in line with ethical considerations both in terms of design and implementation. The evaluation report mainstreams human rights and ethical considerations across the report, however following sections present a detailed and consolidated view. At the design stage there is clear focus on addressing the needs of the vulnerable and extremely vulnerable. The programme also ensures that the range of primary stakeholders i.e. the beneficiaries of the programme are reached and their needs addressed through specific mechanisms and approaches for instance a mix use of CLTS and SLTS focusing specifically on the needs of the community and directly on the needs of children through school led approach. Similarly interventions and operational methods were designed so that women, men and children were approached directly so that their views and input as well as their specific needs and concerns could be addressed directly. The programme also demonstrates focus on specific issues such as disability and MHM in design though less so during the implementation stage. Process of using PRA to ensure vulnerability assessment is carried out as well as approaching community through Men and Women Social Organisers is a good practice and considerate of local context and cultural sensitivities. Moreover the use of local resource in terms of CRPs, Mart Owners and Masons ensured the involvement and ownership of the community in the programme process. Children’s involvement is clear in the SLTS process and their views and participation in the process has been ensured. There was no particular mention of violations or breach in terms of ethical considerations during interaction with community, schools and children. However, some operational oversights have occurred particularly in the areas of constructing DRR specific elevated water supply facilities and latrines, in which case certain privacy aspects (particularly for girls in schools) have been infringed, this could be addressed in the future through better planning and involvement of respective stakeholders.
[73]
Having reviewed both the design and implementation, the evaluators argue that although not formally articulated ethical considerations were adhered to in practice, including protection of the dignity, rights, and welfare of human subjects, particularly children, and respect for the values of the beneficiary community. However it is recommended that ethical considerations be made a formal and written part of design documents as well as implementation arrangements including in PCA documents with partners for future programmes. Programme Design and Implementation UNICEF uses a human rights based approach (HRBA) and works in partnership with communities, especially women and children (in planning, implementing and maintaining water and sanitation systems). The design of SPSP does not explicitly state this, however the design elements comply mostly with the HRBA and this was made part of the programme following recommendations from the RuSFAD evaluation. The design and PCA refer to gender considerations as well as demonstrated equity focus At the contractual stage, all IPs were required to follow Millennium Declaration, Core Commitments for Children (CCCs), the Convention on the Rights of the Child (CRC) and the Convention on the Elimination of All Forms of Discrimination against Women (CEDAW). At the implementation stage, expected HRBA approaches and techniques are observed as strengths of the programme, including,
- Participatory rural appraisal. - Participatory situational assessment. - Community dialogues (within villages, and with district officials). - Village Sanitation Committees (allowing to build village consensus, promotion
of hygiene and responsibility of liaising with officials). - CAP (articulating community’s development needs and creating an action
plan to move towards these). - Joint activity with programme IPs for ODF declaration and certification. - Children’s WASH Clubs, helping them take charge of their health and hygiene
conditions in the school. As already noted above this was not done through an explicit statement of HRBA, therefore there are certain weak areas where the programme could have done more and better. The programme does not use a rights based language, therefore the articulation of water and sanitation as a right is not made, nor is the emphasis clear on the roles and responsibilities of the right holder as well as the duty bearer who is responsible for provision and upholding of the right. Also the programme could have done more in linking the VSCs with Duty Bearers and could have tried to access resources (water and sanitation) in an effort to demonstrate the HRBA. Instead the programme provides most of the water and sanitation facilities through IPs and school facilities through independent contractors, in many ways not in line with HRBA principles. Moreover the staff and IPs do not seem to be fully aware of HRBA as an approach and therefore areas which could have been stronger, come across as not being addressed. One such example is the lack of policy influencing and push on providing WASH facilities in government schools through the education department. Another, mobilizing communities to advocate and ask relevant agencies to rehabilitate
[74]
dysfunctional water schemes. The limited engagement of communities in programme monitoring, involvement in identification, bidding and rehabilitation of community schemes/infrastructure also points to limited understanding (in terms of translating HRBA into activities) of HRBA amongst partners. Monitoring Arrangements UNICEF’s approach in this programme was to mainstream HRBA across the programme design and implementation framework, including even in the context of monitoring. Third Party Monitors were therefore required not only to monitor the quality of processes but also on progress in activities and outputs under the framework including aspects such as proper identification of extremely vulnerable in the community. Similarly monitoring that the PRA process was followed as per the UNICEF checklist to ensure inclusion of marginalized in the planning process. The monitoring checklist included other HRBA related indicators, which was further strengthened through the Programme Results Framework which mainstreamed income quintile based desegregation as well as gender based desegregation. Other aspects including a monitoring focus on access for the disabled. In addition to this, UNICEF also conducted the WASH Equity study which helped informed UNICEF of the public spending in all districts of Pakistan on WASH as well as access to WASH facilities in terms of income quintiles, which could be compared with baseline and end of phase evaluation figures to demonstrate increased access to water and sanitation (declared as basic rights in the various treaties that Pakistan has ratified and is a signatory of). Contribution towards MDGs PATS is the formal strategy by Government of Pakistan towards attaining MDGs related to sanitation. The programme has been successful in increasing the proportion of population having access to sanitation facility to 65% from 44% at the time of baseline in the programme areas. Moreover this achievement is higher than the current national coverage of 50% (including both rural and urban areas), thus one can argue that there has been some appreciable increase in access to sanitation facility resulting from this programme even at the national level. Child Friendly Facilities In line with the Islamabad Commitment, 2010 which was signed by all provinces relating to Child Friendly Inclusive Schools, the programme has been successful in providing child friendly WASH facilities.
[75]
The programme has been able to provide 97% of the schools with latrine facilities, with 77% having access to water for latrines (including piped as well as through bucket). In addition to this SPSP was able to form WASH Clubs, train teachers, and reactivate SMCs which should help in addressing issues to ensure the school facilities remain child friendly. Disability Addressing of disability remained a weak aspect of the programme at both communities as well as school level in terms of sensitisation as well as actual infrastructure designs and sanitation supplies. At the proposal stage the programme clearly stipulates that all activities will be designed and carried out with due regard to disability. The baseline study also highlighted the need to collect desegregated data for the disabled, additionally Third Party Monitoring mechanisms also included monitoring of facilities including assessing their suitability for the disabled. Lastly, under the SLTS component the facilities were to be child friendly including accessibility to children who are disabled. However this emphasis has not been translated into action during programme implementation. None of the school or household latrines meets the requirements for being accessible to the disabled. The hand washing points are based on traditional design which means it can be limiting for a child having physical disability. Due to DRR concerns both water as well as Sanitation facilities have been designed to be elevated from the ground, however no arrangements were made to provide suitable ramps, instead both facilities have stairs. While visiting some hand pump facilities which did have ramps (although very steep), the evaluators were told that these ramps are provided so that community can bring wheel barrows/trolleys up to the hand pump so that it is easier to carry the water, failing again to highlight any possible link to disability.
95%
64%
42%
30%
73%
77%
97%
77%
77%
0% 20% 40% 60% 80% 100% 120%
Well Lit Classrooms
Ventilation in Classrooms
Dustbins in Classrooms
Availability of Soap
Is Latrine In Usable Condition
Water Availability in Latrines
Availability of Latrine Facility
Handwashing Facilities
Availability of Drinking Water Points
School Facilities Attributes
Figure 30: School Facilities Attributes
[76]
Duty Bearers Capacity Building The programme has carried out orientation and capacity building activities with duty bearers (including district budget seminar). However they have not been focused on HRBA principles and they do not formally and explicitly hold duty bearers responsible for respecting people’s rights, or identify delivery of basic services such as sanitation and access to water as a basic right for the people. Moreover, whilst the programme created community structures such as the VSC as well as the CRPs, there has been little effort and progress in connecting them to duty bearers or political individuals to ensure a statement of rights is placed in front of them, or resources and services are demanded of them. Formal and Informal Mechanisms In some cases IPs with the cooperation of district government were able to set up District Committees to oversee the progress, implementation of the SPSP programme, however in most cases the relationship remained informally structured with IPs picking and choosing who to interact and deal with given the responsiveness of the duty bearer (interaction varies significantly from District Commissioner down to Union Council Secretaries).
The programme however encourage Duty Bearers to interact and visit the communities through the orientation sessions, opportunity to visit and certify ODF Villages, engagement with IP field staff vis-a-vis Union Council Secretaries, BHU Staff, and Govt Field Extension Workers. At the community level, CRPs and VSCs (men and women) have been developed as the informal mechanisms for articulating community needs and rights. Similarly at the school level, Children’s WASH clubs (for boys and girls) help raise health and hygiene as well as WASH issues. However the process of articulating and voicing rights needs more capacity building to ensure these mechanisms remain effective as well as sustain. Similarly more work is required in building duty bearers understanding and ability to respect, protect as well as provide for people’s rights.
[77]
6.4. Partnership and Financial Management Evaluation Q 7: To what extent the project has observed and applied UNICEF’s standard financial management practices with Implementing Partners as well as with Government? UNICEF guidelines for partner selection were mostly complied with, though the process could have been made clearer, in terms of decisions and the process used to achieve them. The partner selection nevertheless was appropriate with partners having relevant WASH experience and geographical presence. The umbrella arrangement posed operational and implementation challenges and needs to be reassessed. Consultation with Government Departments and their involvement in the programme remained weak, while issues identified during the bottleneck analysis seem not to have been adequately addressed. One of the provincial team (Baluchistan) refers to self-defined scoring system under which points are awarded during CSP mapping exercise for the review and recommendations by the committee comprising of members from different thematic areas. Evaluators were not provided any approved policy document or decision from Islamabad to legitimate the scoring system and the factors defined under mapping exercise. Evaluators were interested to see how partners were selected for SPSP project and for this, a complete list of CSPs under consideration during mapping exercise for SPSP project was acquired with final scores from UNICEF team. It was found that out of three selected partners for SPSP in Baluchistan, two organizations (BRSP and NCBP) were not part of the list provided by provincial office. On further review of the mapping list provided, there were many organizations who had obtained more scores and had geographical presence with experience with UN and Government agencies and were not selected for partnership as compared to the organization (WDCO) which was finally selected. This indicates a requirement of making more informed and well documented decision making with a clear rational for selecting and not selecting the potential partners. The guidelines provide less guidance on how the committee should have made a decision, on what basis to shortlist and select partners particularly in the event of qualification of greater number of organizations than required. The proceedings of committee deciding on final partners were not made available for evaluation. Evaluators also noted some inconsistencies in scoring vis-a-vis final selection of partners. However information was not made available at the time of the evaluation to determine how these inconsistencies had affected the programme during the course of programme implementation. To a large extent appropriate partners were selected (though no specific capacities beyond an element of WASH experience and geographical presence/outreach in the area were assessed) while filtering for partners. Moreover, evaluators were able to identify an exception to geographical presence criteria in one province.
[78]
Before signing off the PCA with NRSP, micro assessment was conducted and NRSP was rated as low risk CSP. However as per joint proposal three consortium partners (Sub Implementing Partners) were selected which are FRDP, PREPARED and Sukkar Foundation. This umbrella PCA arrangement did not consider conducting micro assessment for these Sub Implementing Partners posing a considerable risk for the project. Even the funding amount is less than USD 100,000 for each sub IP however, as a matter of best practice; thorough assessment could have addressed any legal, financial and programme risks. Records were not available to confirm what type of comprehensive legal arrangements was made by NRSP to avoid any risks as well as how the capacities of these organizations were assessed before awarding them the sub IP status. Evaluators were able to identify through TPM reporting that there were operational and programme delivery challenges with the umbrella arrangement in Sindh (NRSP as lead agency with three sub-IPs) as well as in the partnership in Baluchistan (NCBP with WDCO). Working through sub IP is more costly as at least two organizations and their
resources are involved to deliver on the project. The process on umbrella PCA
arrangements needs to be defined clearly as under what circumstances, this
arrangement can be considered, how sub CSPs can be assessed and selected,
how legal risks can be addressed in case of poor or non-performances etc.
Micro assessment methodology comes across having its own limitation, however through it; UNICEF did identify the areas of improvement. Though it seems UNICEF ended up picking and choosing among the various risk areas for interventions and no record was available to explain this selective approach and the circumstances that contributed to it. PCAs contain sufficient clauses on anti-corruption and bind the CSPs to follow the UNICEF policies. CSPs are also contractually required to follow Millennium Declaration; Core Commitments for Children (CCCs), the Convention on the Rights of the Child (CRC) and the Convention on the Elimination of All Forms of Discrimination against Women (CEDAW). However evaluators do not have any mechanism to verify what orientation and alignment initiatives have been taken to ensure that CSPs and GPs are on board and fully understand these policies and comply with in their routine operations. This fact is not checked during spot checks. Some basic level capacity building was done which helped to mitigate the project
related financial risks (including training of the finance teams and introducing
independent internal audit functions). The evaluators also found that the
recommendations made by the EY during financial micro assessment were not
fully picked up and addressed under the Risk Mitigation plans and hence there is a
strong likelihood that the same issues will be raised during audit process. There
was no specific information available to confirm how provincial level assurance
committees have made a decision to pick and choose between different
recommendations as identified by CA firm.
[79]
Government Partners Coordination and Management Arrangements UNICEF provincial offices were not able to produce enough documents where coordination meetings at provincial and district levels and issues related thereof could have been reviewed. Evaluators had to rely on the interactions with UNICEF staff and Provincial Government department’s representatives. Provincial Government departments were not adequately consulted and were not taken on-board in the programme design even to the extent that details of roles and responsibilities expected from them had not been properly shared. For example, under the SLTS component, the education department at provincial level was not on board in terms of selection of schools, teachers training and formation of Wash clubs as well as details of operation and maintenance of latrines that were constructed. While orientation on PATS was provided in the beginning of the programme, neither the government departments were fully aware of key programme activities nor was an approach agreed with them to ensure the sustainability of the programme after exit phase. Evaluators found that situation at district level, was relatively better where IPs were more active in engaging with district management. The government departments however found to have little knowledge and involvement around incentives and rewards (including expectations from relevant departments to contribute). The engagement level at district level varied from district to district and was more dependent on personal and organizational reputation of the implementing CSP. Capacity development plans for provincial level departments in 2013 were not rolled out with allocation of budgets to carry out the activities nor the bottleneck analysis that were conducted with government departments in four provinces were fed into the programme or reflected fully during programme implementation. Financial Risk Management PCAs contain sufficient clauses on anti-corruption and bind the NGO partners to follow the UNICEF policies. However evaluators do not have any mechanism to verify what alignment initiatives have been taken to ensure that NGO partners and Government partners are on board and fully understand these policies and comply with them in their routine operations. This fact is also not checked during spot checks. Limited sharing of further information makes a conclusive assessment difficult. Procurement Procedures for Goods and Services The evaluators neither have the mandate nor access to key information on procurements by UNICEF and NGO partners as this comes under the financial audit domain. However as a process, supply management assessments were carried out for NGOs as mentioned on PCA-Review Committee checklist before signing off the PCA.
[80]
6.5. Unexpected Findings The unexpected findings resulting from the implementation of SPSP Phase include unravelling of the critical link between open defecation exposes women to accusations of illicit relations and can lead to being subjected to extreme violence. Access to Sanitation Facilities and Reduction in Violence against Women The evaluation findings have contributed to unravelling a critical link between open defecation and extreme cases of gender violence. A significant number of respondents (including Government Officials, CRPs and Women) across different geographical locations (Sindh and Punjab) shared that the construction of latrines has reduced the incidences of violence against women such as Karo Kari (honour killing). To them, the construction of latrines has reduced the need for women to step out, and hence reduced the probability of being accused of illicit relations and being subjected to violence. In a narrated instance a mother married off her daughter to a mentally handicapped person in order to save her daughter from Karo Kari. There is other emerging evidence for instance in May 2014, two girls were gang-raped and hanged in Uttar Pradesh, India where they had gone out to look for a place to defecate. A WaterAid study in the slums of Lagos in Nigeria also showed that a quarter of women, who lacked access to sanitation, had first or second-hand experience of harassment, threats of violence or actual assault linked to their lack of a safe, private toilet. Amnesty International has released similar studies from Kenya and the Solomon Islands. Results of the SPSP Phase 1 post-KAP survey suggest that 83% of the respondents feel it is unsafe for women to go out to defecate in the open. Though an indirect result, it is a significant aspect and therefore should be incorporated in a considered way into the evolution of BCC strategies and demand creation interventions. Shame, Shock and Disgust versus Pride, Self-Respect and Dignity The two approaches used for triggering sanitation demand, however normally have not been used together. SPSP Rural I, by using them together in certain order (where shame, shock and disgust follows the other), has demonstrated that if used together these two approaches could help deliver better results. Business Incentive and Opportunity Is a Larger Motive for Response. DSI’s action research suggests that manufacturers, entrepreneurs and mart owners (with business risk bearing attributes) see supply to the underserved areas as an opportunity to expand their business further. Given purely business incentives, they are more likely to sustain sanitation supply in order to meet the heightened demand and maintenance requirements in communities. Hence, it could be argued that the programme may benefit more, if it may focus on identifying and working with manufactures/entrepreneurs, with a clear intent to help them realise the real market potential and opportunity offered by these underserved areas for business continuation and expansion. This in the long run shall contribute to ensure technological innovation, continued and sustained supplies facilitation.
[81]
7. CONCLUSIONS & LESSONS LEARNED
The evaluation concludes that the SPSP Phase 1 was implemented successfully though with varied levels achievements in terms of programme components, interventions and results. This conclusion is made based on observations from field visits, discussions held with beneficiaries in visited villages, government staff and IP’s in the four provinces and FATA as well as from consultations at national level, desk reviews and comparison of the Pre-Post KAP. Under the programme, the poor and extremely poor communities in the rural areas benefitted from access to sanitation services, improved access to water, health and hygiene promotion and through supply side sanitation interventions. The programme maintained a gender and vulnerability focus which is demonstrated in operations and results. CLTS and SLTS were found to be complimentary to each other while community mobilisation training and capacity building proved effective in achieving ODF declaration status. The field interaction and involvement of government representatives during the ODF verification/certification process comes across as a strong point. However the programme could have developed stronger and better engagement with government departments with particular focus on drawing down public sector funds from provincial to district level and ensuring that communities received awards/ incentives following ODF verification/certification. The presence of a well-defined monitoring and evaluation system since the beginning of the Programme resulted in improved quality of progress reporting in terms of adequacy and consistency of information as well as informing management decisions and course correction during implementation. The programme also contributed in developing WASH training manuals, health and hygiene promotion material and conducting action research. Moreover, the baseline and evaluation studies commissioned during the programme have created evidence and provided data that can be used to inform future programme direction as well as contribute to review of PATS. Future programme would however need to focus more on modality of subsidy provision, methods of demonstration latrines, sanitation marketing (particularly developing sanitation marts and entrepreneurs), and availability of low-cost technologies for sanitation. The evaluation identifies lessons and proposes recommendations that the programme can take into consideration while designing the next phase or future programme. Overall the experience of SPSP Phase I confirms that PATS can be implemented on a large scale with key requisites such as increased capacity of and advocacy with government, sector stakeholders and communities to apply PATS approach effectively.
[82]
Lessons Learned Following four key lessons emanate from having implemented the programme, which may have broader implications and also help add to the general understanding and body of knowledge around these issues (both in the context of future WASH programming by UNICEF or others). These could also inform planned PATS review and UNICEF CATS approach. Access to Sanitation Facilities Has a Role in Reducing Violence against Women The evaluation has contributed to unravelling a critical link between open defecation and extreme cases of gender violence in rural Pakistan. A significant number of respondents shared that the availability of latrines inside the house has significantly reduced the incidences of extreme gender violence such as Karo Kari (honour killing). To them, the construction of latrines have reduced women need to step out (as for defecation they were coming out at odd hours), thus the protection risk (or probability) of being accused of illicit relations (with men), and consequently being subjected to extreme violence. The lesson that is derived from this instance is that though an indirect result of WASH intervention, the benefit of contributing to a reduction in violence is a substantially important aspect and should inform the future programme design and delivery. Shame, Shock and Disgust versus Pride, Self-Respect and Dignity The two approaches used for triggering sanitation demand, however normally not been used together. SPSP Rural I, by using them together in certain order (where shame, shock and disgust follows the other), has demonstrated that if used together these two approaches could help deliver better results. Business Incentive and Opportunity Is a Larger Motive for Response. DSI’s action research suggests that manufacturers, entrepreneurs and mart owners (with business risk bearing attributes) see supply to the underserved areas as an opportunity to expand their business further. Given purely business incentives, they are more likely to sustain sanitation supply in order to meet the heightened demand and maintenance requirements in communities. Hence, it could be argued that the programme may benefit more, if it may focus on identifying and working with manufactures/entrepreneurs, with a clear intent to help them realise the real market potential and opportunity offered by these underserved areas for business continuation and expansion. This in the long run shall contribute to ensure technological innovation, continued and sustained supplies facilitation. Using Subsidy to Address Equity Can Lead to Dependency To a degree the subsidies have served the equity purpose. However, the subsidy planning and delivery mechanisms faced challenges, which need deeper thinking and innovative solutions. The solutions may need to take note of the evaluation finding that such incentives contribute to reducing willingness of families to make financial contributions to construct latrines.
[83]
8. RECOMMENDATIONS
Following are key recommendations along with priority and identified stakeholder(s). The recommendations were drawn through a comprehensive and systematic process, including drawing on lessons learnt by the stakeholders (UNICEF, DSI, RSPN, Range of IPs, other development partners, government representatives as well as community representative) and their propositions that could improve the programme. Following this process the evaluators were able to consolidate key recommendations and present to the Evaluation Reference Group in a plenary session (including UNICEF representative from Regional Office, Islamabad and Provincial Offices, Government Officials as well as Donor Representative) during the two day workshop. The recommendations were then discussed within smaller groups, arguing their respective importance, clarifying their objective and then agreed through a mutual process with agreement of the Evaluation Reference Group.
No. Recommendations Priority Stakeholder
1. Informing PATS Policy
UNICEF has garnered PATS related experience through implementation of RuSFAD and SPSP (for instance whether to use Respect and Dignity or Shock, Shame and Disgust method), moreover lessons and findings are also coming out of the Action Research carried out under SPSP by DSI (suggestive that involving manufacturers with clearly identified incentives and opportunities goes a longer way in facilitating sanitation supply). Similarly other development partners such as WaterAID have tried other variations (for instance not offering household level subsidy) while interpreting and implementing the PATS approach. It is therefore recommended that UNICEF commission a joint research (involving relevant stakeholders including other development partners) to study the various PATS models being implemented in Pakistan and the adaptations that have been brought about to ensure ODF status. The research should also highlight the differences in approaches due to contextualising of cultural and geographical considerations and sensitivities such as found across the Provinces. This research should thus be used to inform and revise the PATS document and depending on scheduling be showcased in the next PACOSAN. Dissemination of the study would also help bring attention to PATS and the effort to make progress towards MDG targets.
Medium Term UNICEF, WASH Sector Development Partners, CCD and relevant Provincial Departments
[84]
No. Recommendations Priority Stakeholder
2. Knowledge Management
It is clear from the SPSP evaluation as well as from interacting with the various stakeholders that there is substantial and good work being done on PATS (e.g. WaterAID, plans of WSSCC as well as work being carried out by Punjab). This work is also being documented and adding to the existing knowledge base. It is therefore recommended that UNICEF takes a lead role (along with other development partners) to ensure that an online repository is created, which makes programme documents, evaluations, action research etc. relevant to PATS programmes (including those generated during SPSP Phase 1, as well as from RuSFAD) widely available. The repository should provide open access, be user-friendly and allows authorised people to upload documents.
Immediate UNICEF (along with WASH Sector Development Partners)
3. Harmonisation & Coordination
From the interaction with various stakeholders as well as based on the evaluation findings it is clear that the sanitation sector critically lags behind other sectors in terms of harmonisation as well as coordination mechanisms. It is recommended that UNICEF should use its stature and leverage its relations with other development partners, Federal and Provincial governments to ensure sector wide coordination forums at national and provincial levels are formed/made active for advocacy, policy making and resource mobilisation/allocation. Furthermore for coordination at project level, it is recommended that Provincial and District Coordination Committees be formed/notified through government with PHED/LGRDD as lead to leverage provincial and district level support and ownership, in cases where similar committees exist they may require reactivation and reconstitution to make them functional.
Medium Term UNICEF, WASH Sector Development Partners, CCD and Relevant Provincial Departments
4. Policy Influencing & Advocacy for Government Allocations
Based on the evaluation findings following things are evident, a) drawdown of government resources remained weak even though it is considered intrinsic to
PATS that communities are linked to Government Departments at district and sub-district
Immediate UNICEF
[85]
No. Recommendations Priority Stakeholder
level for continued work in achieving and maintaining ODF status, for which government resources should also be mobilised,
b) also the scale and magnitude of the sanitation challenge in Pakistan necessitates the increased involvement of Government along with the will to ensure scale-up and sustainability.
It is therefore recommended14 that UNICEF should use its position and influence and work top to bottom from province to ensure budgetary provision for awards, incentives and rewards to departments at the district level. A standardised approach and methodology should be adopted by UNICEF in Islamabad as well as the Provincial Office for policy influencing at provincial level and implementation at district levels (with variations to suit provincial or geographic context). This should be based on written MOU or agreement between UNICEF and Government Departments. In SPSP Phase II, UNICEF should pilot PATS implementation with Public Stakeholders in at least one Province (which is receptive and committed to addressing sanitation issues).
5. Integration with Other Sectors It is evident from the evaluation as well as through interaction at the ground level with Government Departments that to achieve sanitation targets, engagement and committed action on a multi-stakeholder basis will be required. There is a clear need to advocate mainstreaming of sanitation across other sectors (including Health, Nutrition, Education and Government Departments). This should include cross sector convergence on WASH policy and programmes, which can include advocating for hygiene to be led by Health Department, while being mainstreamed into the education curriculum. Moreover, it can look into options for improved usage of SMC funds while ensuring regular monitoring of Schools including monitoring of WASH facilities to identify and address defunct facilities. It is therefore recommended as a minimum that UNICEF engages with these departments both at the Provincial and District level, while ensuring that they come together (it is notified as part of SPSP
Medium Term UNICEF, WASH Sector Development Partners and Relevant Government Departments
14 The evaluators note that this is an ambitious recommendation, however it is considered necessary for scale-up and sustainability so that in the longer term progress can be made at the national level in meeting MDG Target.
[86]
No. Recommendations Priority Stakeholder
Phase 2 agreement with government) at the provincial level as a Coordination and Steering Committee headed preferably by PHED, and at district level as an Implementation Committee headed and reporting to the District Administrative head. This would also allow other stakeholders to use similar government structures for better coordination and implementation of their respective WASH programmes.
6. Project/Programme Duration The evaluation findings and the two-day workshop with the Evaluation Reference Group highlighted that there was no specific time framework (minimum incubation period) for PATS. Examples from different provinces as well as IPs ranged from as short as 3-4 months to 6-9 months. However based on substantial and in-depth discussion a clear recommendation for future programming was agreed which should take into account and provide adequate duration for the following phases of the programme,
- Start-up/inception phase (PCA signing, hiring of staff, IP inception workshops etc.) - Implementation phase of 12 months or more for CLTS and SLST (based on time required for
effective community mobilisation, as well as factoring in community life/work cycle aspects, school holidays etc.)
- Exit phase (should include exit planning, sustainability of interventions as per programme strategy, and post ODF monitoring).
Immediate UNICEF
7. Gender Equality, Equity, HRBA (Disability) While SPSP Phase 1 was clear in mandating and focusing on gender equality, equity and disability, evaluation findings suggest more clarity; working mechanisms and definitions need to be articulated, along with describing the process that will ensure these aspects are factored into the programming and implementation. It is therefore recommended that clear processes need to be articulated in the programme design, the TOC should explain the change process and results, while PCA should include sections detailing specific interventions, criteria and processes that IPs have to implement ensuring inclusiveness (such as for PRA, School WASH Facilities, more targeted BCC, disability integration in Latrines) in line with UNICEF guidelines to address gender equality, equity and vulnerability along with special needs of the disabled.
Immediate UNICEF, Relevant Government Departments at Provincial and District Levels.
[87]
No. Recommendations Priority Stakeholder
8. Exit Strategy & Sustainability
The evaluation highlights the lack of clarity about the mechanisms and time duration that is required to ensure a smooth exit and put in place sustainability mechanisms.
It is recommended that for SPSP Phase 1 implementing partners a minimal operational support for six months is provided in the districts of operation to ensure gains of the implementation phase (ODF Status - declaration and certification) are not lost to slippage. For the next phase and future programmes, it is then recommended that a Theory of Change needs to be articulated, and a Sustainability Strategy drawn in line with it, focusing on sustaining ODF status (for which a study can also be commissioned examining what needs to be done to sustain ODF status). PCAs should include specific IP exit plans, and those stakeholders who will be ultimately responsible for sustainability should be clearly informed of their role along with their consent to deliver it (CRPs, VSCs, Wash Clubs as well as role expected from Government Departments and their extension workers such as Community Development Officers, Union Council Secretaries, LHWs and School Teachers etc.).
Immediate UNICEF
9. Training & Mentoring
Findings suggest that using a central training organisation proved effective; though with some constraints which if addressed should help make it even more effective. It is therefore recommended that the programme should continue the use of a central training organisation, however the level of resource provision, staffing should be according to the magnitude of activities over the duration of the programme. Mentoring should be need based and resourced from the training organisation based on IPs need or as identified through progress monitoring. Training events should be announced in a timely manner to allow maximum attendance and should benefit all who are relevant, while taking into account aspects such as community work cycle and availability.
Immediate UNICEF and Training Partner Organisation
[88]
No. Recommendations Priority Stakeholder
10. Supply Facilitation Sanitation Mart Owner/Entrepreneur Distinction between Sanitation Mart Owners and Entrepreneurs should be reassessed and if required dropped. Innovative results emanating from Action Research carried out by DSI need to be examined to evolve an effective role for entrepreneur (including the element of risk taking, market forces of demand and supply, as well as issue of financing). Community Resource Persons (CRPs) The concept of service fee for Community Resource Persons (CRPs) worked and should continue for the period of mobilisation. Options should be considered to utilise these resources for sustained focus on sanitation by transforming them into small entrepreneurs and help expand the capacity of the market to supply services and products (this could be made possible by giving them a reward once they have achieved their agreed targets as an incentive). Household Subsidy Household subsidy could either be withdrawn and other mechanism such as support with material through voucher system in post humanitarian crisis situation and collective incentives can be tested. For equity purposes and the needs of the vulnerable revolving credit/contributory fund can be considered as an option to be considered managed by the VSC to ensure the sanitation requirements of vulnerable are also met, this can be done through options such as setting up revolving credit with the VSC.
Immediate to Medium Term
UNICEF (with advice of DSI, RSPN)
11. Low-Cost Sanitation and Drainage
More options for low-cost sanitation need to be explored. This should include options for low cost pits (possibly using low cost pit lining, increased use of local materials), as well as for superstructures (which should make use of local materials while ensuring usability of the facility by the target community). It is recommended that options for low cost end of pipe solutions should be identified (if required a study be conducted to inform options) and new ideas piloted in some locations given the demand for drainage articulated by community during SPSP Phase 1.
Continuous UNICEF
Table 19: Evaluation Recommendations
[89]
Annex 01 Terms of Reference
Technical Assistance for SPSP-Rural End of Programme Evaluation
1. Programme information:
Commissioning office: UNICEF Paksitan - WASH Programme, Islamabad Evaluation Title: UNICEF’s Sanitation Programme at Scale in Pakistan (SPSP) – Rural
Through Pakistan Approach to Total Sanitation (PATS) 2. Background of the Programme:
In Pakistan, diarrhoea is the leading cause of mortality for children under 5 (PDHS 2007-8) where 116,013 children under the age of 5 die due to diarrhoea each year, translating into the loss of life of 13 Pakistani children per hour (PDHS 2007-8). Children also suffer disproportionately from sickness due to diarrheal disease with almost 25 million case (PSLM 2006-7) reported annually. Major reasons for diarrhoea are lack of access to clean drinking water, poor hygiene practices and, poor sanitation. The already dismal situation of sanitation in Pakistan was further deteriorated by the devastating floods of 2010/2011/2012 which affected at least 20 million people across the country, some of them twice or thrice consecutively. The 2012 heavy monsoon flooding affected an estimated 4.8 million people in the provinces of Punjab, Balochistan and Sindh according to the National Disaster Management Authority (NDMA). Sectarian and tribal violence, violence by other non-state entities and military operations in the Federally Administered Tribal Areas (FATA) have caused displacement. There are about one million people registered as displaced who have still not returned to their area of origin. Most of the displaced people in flood affected districts have returned to their damaged or destroyed homes, infrastructure and livelihoods while most of the insecurity affected displaced families are still sheltering in relief camps or with relatives. UNICEF with the Government of Pakistan, through its partner organizations, launched a large scale Rural Sanitation Programme in 2010 - flood affected and high polio risk districts (RusFAD) to mitigate the grave needs of people related to sanitation. The program was implemented in three phases i.e. Phase I, II and III. In 2012, based on learning from the successful implementation of the RusFAD programme, UNICEF with the Government of Pakistan and its Implementing Partner (IP) organizations launched the ‘Sanitation Programme at Scale in Pakistan (SPSP)’ for 2012 - flood affected and high Polio risk and insecurity affected areas. The first phase of SPSP was implemented in fourteen districts to reach 1.3 million people of Punjab, Sindh, Khyber Pakhtoonkhwa, FATA and Balochistan aiming at by 2017, most vulnerable and marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The National Sanitation Policy of Pakistan, approved by the Federal Government in 2006, envisions the creation of an open defecation free environment with safe disposal
[90]
of liquid and solid waste and the promotion of health and hygiene practices in the country. Further to this policy, a holistic vision of sanitation was developed through an integrated Total Sanitation model resulting in the formulation of a Pakistan Approach to Total Sanitation (PATS) as a comprehensive strategy for promoting improved sanitation and hygiene outcomes in peri-urban and rural areas that focuses not just on ending the practice of open defecation and transforming hygiene behaviour, but also sanitation marketing, waste water collection, solid waste disposal and drainage (Government of Pakistan, Ministry of Environment, 2008). Operationalizing PATS, sanitation demand is created by applying Participatory Rural Appraisal (PRA), Community Led Total Sanitation (CLTS), and School Led Total Sanitation (SLTS) tools. It focuses on respect, dignity and pride as motivations for change rather than the conventional shock, shame and disgust methodology usually employed by CLTS projects. The programme also promotes the use of safe hygienic latrines and encouraged improved hygiene behaviour through Information, Education & Communications (IEC) campaigns while supporting the establishment of markets for low-cost sanitation goods and services. Trained masons demonstrate the construction of latrines to showcase low cost options. The strategy also includes pilot projects to demonstrate decentralized constructed wetlands for attaining 100 per cent drainage and wastewater treatment in select villages in all provinces. The conceptual framework of project and activities within the six pillars of PATS are illustrated below, which can be helpful to the evaluators in developing a Theory of Change for the evaluation mission.
Sanitation
Demand
Creation for
ODF
Communities
Technical
trainings of
masons
Mobilizing
communities
through school
teachers
/children and
social organizers
adapting a
cascading model
Launching
localized demand
creation C4D
campaign ( based
on KAP &
Formative
Research)
Construction of
low cost Demo
Latrines
(environment
friendly, gender
appropriate,
safe)
Facilitating
sanitation marts
and trainings of
sanitation
entrepreneurs
Develop
microfinance
options
Provide
incentives and
subsidies (7% +
3%)
Locally
customized C4D
material Packs
developed on key
health and
hygiene
messages with a
communication
strategy
Mass Media,
Community,
School children,
Religious and
Political leaders
led thematic
campaigns
Promotion of
low-cost
appropriate and
informed
sanitation
solutions
Ensuring adequate
drainage design with
the support of PHED
Advocacy for
wastewater
treatment through
Oxidation
ponds/Constructed
Wetlands
Advocacy for
underground
/covered drainage
system
Advocacy for waste
water disposal in a
hygienic way
Demonstrating
drainage and
wastewater
treatment in
selected villages
Sustaining
Demand
through
Supply Side
Interventions
Participatory
Health &
Hygiene
Promotion
Attaining
100% Adequate
Drainage and
Wastewater
Treatment
Linkages
development
with duty
bearers
Knowledge
Management and
Accountability
Capacity
development of
duty bearers to
enhance their
understanding
and ownership
of PATS
Advocacy with
duty bearers to
take policy
decisions in
support of
PATS
interventions
(e.g. selection
of intervention
areas, M&E,
provision of
rewards and
incentives)
Learning PATS:
Research, Inception &
Lessons sharing
Monitoring:
Baseline, Endline
Pre/Post KAP
UNEG Evaluation
Monthly Process
Monitoring,
Mid term review
workshop
C4D/Institutional
Learning:
Replication and
institutional memory
(Case studies, Human
interest stories ,
Technical papers,
Documentaries, online
PATS DMS)
Formation of
Village Sanitation
Committees
Development of
Community
Action Plans
(PRA, Transsec
Walk)
[91]
3. Purpose of Evaluation
The major purpose of the assignment is to conduct a final evaluation of the process outputs and outcomes of SPSP-Rural taking into account UNICEF guidelines for programme evaluation following OECD Development Aid Criteria. The final (end term) evaluation will examine the results of this programme with the purpose not just to to assess and review the outputs, outcomes and impact, but also to capture the lessons learnt in order to make use of them for recommending improvement in the evidence-based planning and decision-making in similar future initiatives. A large scale Rural Sanitation Programme in 2010 was launched in flood affected and high polio risk districts (RusFAD) to mitigate the grave needs of people related to sanitation. The programme was implemented in three phases i.e. Phase I, II and III. In 2012, based on learning from the successful implementation of the RusFAD programme, UNICEF with the Government of Pakistan and its Implementing Partner (IP) organizations launched the ‘Sanitation Programme at Scale in Pakistan (SPSP)’ for 2012 - flood affected and high Polio risk and insecurity affected areas. The first phase of SPSP was implemented in fourteen districts to reach 1.3 million people including most vulnerable and marginalized children and women. As the project is ending, now, is the most pertinent time to undertake an end of project evaluation in order to assess the outcomes for safe drinking water, improved sanitation and hygiene practices. The evaluation is expected to evaluate achievement of the planned results, whether processes were efficient and participatory and also assess UNICEF’s specific role and contribution and propose measures to increase relevance, effectiveness and efficiency of the revised PATS approach in order to strengthen the PATS strategy in the context of working in partnership with other stakeholders and draw lessons to direct future programme efforts. The primary audience of the evaluation are the DFID, UNICEF staff; IP staffs and government officials who are involved in programme re-design at next stage. The secondary audiences are the WASH sector partners and communities who can learn how areas of improvement in this programme can be adopted and best practices replicated elsewhere while taking sanitation programmes at scale. It is therefore very important to capture both analytical critique as well as best practices of the project. 4. Evaluation Objectives Specific evaluation objectives include:
I. Determining the relevance, effectiveness, efficiency, impact and sustainability of UNICEF SPSP-Rural Phase I programme (as per UNEG standards given under section 10);
II. Assessing the extent to which UNICEF core programme strategies; i) capacity, development, ii) service delivery, iv) advocacy, v) strategic partnership, vi) knowledge management and vii) communication for development were met;
III. Assessing how UNICEFs normative principles, specifically i) equity and human rights based approach, ii) gender equality mainstreaming and iii) environmental sustainability were reflected in the programme approach.
[92]
IV. Identifying bottlenecks and success factors for programme implementation including a detailed analysis of key determinants or factors leading to achievements, taking into account the framework conditions of the programme as part of Early Recovery following the 2012 floods (Humanitarian – Development Nexus);
V. Commenting on the constraints and opportunities of programme implementation under unstable and emergency situation;
VI. Assessing effectiveness of i) strategic partnership for process monitoring (Third Party Monitoring by Apex and other technical monitoring Contractors) and ii) collaborative relationship for programme implementation (RSPN for Training);
VII. Providing recommendations to strengthen PATS programme operational implementation to further scale up rural sanitation in Pakistan.
5. Scope of Evaluation
While reviewing the overall PATS programme approach, the following components relating to UNICEF’s efforts in improving its effectiveness and efficiency in implementing SPSP-Rural based on the learning of Phase III RuSFAD and its change in methodological approach shall be given priority:
‒ Improved institutional linkages & capacity development for governmental counterparts and implementing partner
‒ Campaigning for improved hygiene behaviour and polio eradication
‒ Attaining total sanitation
‒ Knowledge management
‒ Effectiveneness of SLTS
‒ Subsidy for the extremely vulnerable and vulnerable households
In the light of the above objectives, the evaluation shall comprise of a i) representative assessment of post KAPs prevailing in the targeted communities of SPSP-rural. In addition, the evaluation will also include establishing a ii) end line against outcome and output level indicators given in the ‘results framework of SPSP-Rural’ and a iii) capacity assessment of SPSP-rural non-governmental and governmental stakeholders, iv) PATS related bottleneck analysis for each province, using the OECD/DAC criteria. UNICEF intends to enter into an agreement for twelve months for 30 work days. Contractor would be required to start the assignment in mid-January 2014, The end of project evaluation will be limited to project areas covered by SPSP-Rural in its first phase including fourteen (14) districts of Balochistan, Punjab, Sindh, KP and FATA, targeting about 1.3 million beneficiaries in all selected districts of Sindh (Jacobabad, Kashmore and Shikarpur districts), Punjab (DG Khan, Rajanpur districts), Balochistan (Jaffarabad and Naseerabad districts) and Kyber Pakhtunkhwa (DI khan, Tank, Kohat, and Kohat) and FATA (Mohmand Agency, Kurram Agency and Bajaur Agency).
[93]
6. Evaluation Methodology
The evaluation will use both qualitative as well as quantitative approaches and inform itself by triangulation of data. Quantitative design will be used for statistically assessing the situation in SPSP-Rural targeted communities against outcome and output indicators in the results framework. After programme implementation, the qualitative design will give an in-depth understanding of the context and common perception, concerns and aspirations. The qualitative approach shall include focus group discussions and interviews with key implementing partners’ staff, community leaders, local government officials, group of men and women in villages to analyse existing information and comment on their capacity to implement SPSP-Rural in relation to base line findings. In order to uphold the principles of equity, purposive sampling shall be used to ensure participation from all sociocultural groups. Relevant data collected under this evaluation shall be disaggregated by gender and socio-economic (income quintile and vulnerability) status. The data shall be collected from a number of beneficiaries (to be determined later) which will enable a statistically representative sample, at household level, stratified by two (2) levels: male/female and districts (14 districts) to help make results statistically valid and reliable with confidence intervals around the means. For districts with small population size the sample should not be below 3%. The evaluation team will review already existing end of programme questionnaire. Additional questions shall be included to address the objectives of this evaluation, as outlined above, to gather relevant field data within the sampled households. The following strategies for data collection and analysis shall be applied:
i. Desk review of relevant existing information sources, project documents,
partner’s assessments and databases and monitoring reports.
ii. Discussion with WASH and Evaluation staff of UNICEF and implementing
partners
iii. Interview with key informants
iv. Field visits to targeted villages observations.
v. Workshop for capacity assessment guided and structured interviews and focus
group discussions with implementing partner’s staff, beneficiary communities
including women, men, children, district and provincial govt.
vi. Household survey questionnaire to gather quantitative data.
vii. School Survey questionnaire for students, school teacher and head
master/mistress to gather quantitative data.
viii. Workshop for validation of draft findings.
Following documents will be provided to the Evaluation Team:
[94]
i. Programme Results Framework, incl. baseline indicator for outcome and output of SPSP rural programme
ii. Work Plans of project/activities flow chart iii. Questionnaires for Pre-KAP study for SPSP rural; and post-KAP study for
RuSFAD iv. Final draft evaluation report RuSFAD Phase III v. Pre-KAP and Baseline study reports incl. results of bottleneck analysis and
capacity assessment vi. Methodology on how to conduct a bottleneck analysis vii. Copies of relevant sections from PCAs with IPs viii. UNICEF Evaluation Guidelines and supporting documents for evaluation ix. UNICEF branding template x. MoRES Approach xi. Relevant UNICEF policies
All interviews, field visits, focus group discussions, questionnaires and meetings are to be used to update information in relation to baseline conducted at the beginning of SPSP-rural including local sources of knowledge about contextual factors which might influence outcomes later. 7. Evaluation Criteria and Questions
The evaluation will be guided by OECD/DAC criteria and will strictly follow the UNICEF adapted UNEG standards covering the following questions related to the criteria of relevance, efficiency, effectiveness, impact and sustainability:
Efficiency The evaluation will:
Assess the efficiency of the project by comparing to the allocated resources with results obtained.
The evaluation should present a measured opinion on whether the results obtained justified the costs incurred.
Examine delivery of outputs in terms of quality and quantity. The evaluation will give a considered opinion of the total scope of work was as could be expected in view of the inputs and whether the initiatives have produced high quality results and outputs.
Effectiveness The evaluation will:
Assess the programme design as outlined in original documents and in view of the experience with implementation to consider whether the design from the outset could have offered more in terms of efficiency and effectiveness for the programme. The inception report will include a Theory of Change and will give a measured option as to how clear was the programme design in articulating Theory of Change (how change comes about) and how effectively was it used to achieve desired outcomes.
[95]
Assess the implementation as experienced in practical terms to see if the methods applied worked as intended. Special emphasis will be given to considering whether the programme outputs presented in the original plan/proposal were successfully and effectively produced.
Examine whether all parties involved in implementation had the necessary professional prerequisites, experience, expertise and attitudes to produce planned results in an effective and timely manner.
Relevance The evaluation will:
Consider the intended outputs and form an opinion on whether these have practical relevance to the overall goals as set out in the programme document. If so, specify what this relevance is and, if so, how this can be presented to a larger audience.
Consider what lessons could be drawn from the experience with this programme?
Evaluate the strategies used in this programme in light of national (GOP) and other policies and initiatives related to PATS.
Assess whether the GoP and its partners have a clear understanding as to how to harmonize support with the national development priorities as well as other organizations working on similar issues?
Sustainability The evaluation will:
Assess political will on part of GoP and other partners to sustain such an initiative without external assistance (ref. national policies and political discourse at the policy level).
Other related concerns:
Cross-cutting themes (Gender and Equity):
View the programme through a gender lens and consider whether all aspects of UNICEF policy relating to gender issues in development projects have been adequately handled in this initiative or sub-project related to this programme.
Determine whether programme used equity approach and how far it was successful in achieving results related to UNICEF’s equity goals for children and women across all regions and sociocultural groups.
Consider whether the prgramme observed HRBA or contributed to observance of Human Rights in Paksitan as an implicit objective of programme
Impact of unforeseen events/circumstances on programme:
View the influence of unstable and emergency situation on programme implementation.
Financial and Risk Management:
[96]
Measure and analyze influence of unstable and emergency situation on programme implementation
Give a measured opinion on whether the project(s) has dealt efficiently and effectively with all matters related to financial risk management.
Consider whether all matters related to financial management, including procurement procedures for goods and services, have been appropriately handled in light of the UNICEF’s zero tolerance to corruption policy.
Consider all aspects related to the programme’s cost effectiveness and evaluate whether alternative implementing methods could have been applied to increase the cost effectiveness.
Give an opinion on possible strengths and weaknesses in UNICEF/GOP management routines and administrative procedures for implementing the project effectively.
Consider whether UNICEF/GOP have networked and consulted sufficiently with other relevant agencies in the process of programme formulation and implementation.
The consultant is advised to take into account the following aspects in refining and devising evaluation questions in consultation with UNICEF and other key stakeholders during the inception workshop with a view:
To provide all parties/stake-holders with an understanding of the current achievements, successes and areas for future improvements.
To gain an understanding as to whether the programme was optimally implemented with regards to efficient and effective use of available resources.
To assess the programme’s contribution towards creating and maintaining gender equity among the implementing partners and the target beneficiaries;
To assess the implementing partners’ capacity, monitoring mechanism and orientation on applying a result-based approach as well as ability to documenting results;
UNICEF role in supporting the initiative and suggest recommendations for future better support mechanisms.
8. Major Tasks and Deliverables
No. Major Task Deliverables
1 Inception workshop Inception report (D1)
2 Statistically representative assessment of post KAPs, by sex and districts, prevailing in the target communities of SPSP-rural.
Revised post KAP Tools (D2) Populated SPSS database (D3) Post KAP report for statistically relevant number of beneficiaries (D4)
3 Data on SPSP-rural outcome and output level indicators
Output and outcome report for statistically representative sample at household level including interviews and verification of data submissions by IP (D5)
4 Capacity assessment of SPSP-rural non- government and government
Workshop with provincial governmental stakeholders to identify improved capacity
[97]
stakeholders based on a capacity assessment tool.
to deliver basic WASH services in rural areas (D6)
5 PATS related bottleneck analysis for each province.
Bottleneck analysis conducted and documented during same workshop in each provinces (D7)
6 Evaluation as per UNICEF/UNEG criteria
Included in final Evaluation Report (D8)
7 Assessment of effectiveness of i) strategic partnership for process monitoring and ii) collaborative relationship for programme implementation
Included in final Evaluation Report (D9)
8 Reporting Draft report submitted for comments
Final End of Programme Evaluation
Report using UNICEFs branding
template for reports (D10) while giving
consultancy which conducted the
evaluation sufficient visibility (logo)
ppt in UNICEF with government logo
branding template summarizing main
findings (max. 15 slides) (D11)
9. Reporting Standards
The Final Evaluation report outline MUST be based on UNICEF evaluation report standards. Inception Report The evaluation team will prepare an inception report before going into full-fledged evaluation exercise in the field. The inception report will follow the guidelines and proposed table of contents provided by UNICEF (PMER) mainly entailing the following:
‒ Contractor’s understanding on SPSP-Rural operationalizing PATS,
‒ Outline of evaluation methodology, proposed sources of data and data collection
methods/tools etc.
‒ A Gantt chart to define timeline on tasks planned for the assignment
‒ A proposed list of the team members and team leader, their qualification, roles
and responsibilities.
‒ Tools for end of programme evaluation
Draft and Final evaluation report and presentation
‒ The consultant will prepare a draft evaluation report presenting all Key
Deliverables as outlined above which shall then be presented and discussed in
consultative session organized by UNICEF/Implementing partners/Evaluation
Reference Group
[98]
‒ Consultant will incorporate changes agreed in consultative session or through
email for UNICEF comments.
‒ Consultant will incorporate comments by UNICEF and Reference Group on draft
report and submit a final report to be agreed by UNICEF and handover organized
evaluation data (soft and hard copies to UNICEF).
10. Evaluation Ethics:
Evaluation team will adhere to UNICEF ethical principles as outlined in UNEG ‘Ethical Guidelines for Evaluations (to be shared by PMER, UNIECF). The procedures to safeguard the rights of and confidentiality of information providers, e.g., legal issues pertaining to issues such as provisions to collect and report data will be observed, particularly, permission needed to interview or obtain information about children and young persons, storing and maintaining security of collected information and protocols to ensure anonymity and confidentiality. 11. Evaluation Work plan
Evaluation team activities Time
Desk review of existing project documents (as detailed above) 2 day
Inception meeting 1 day
Finalize evaluation design, tools, work plan, database, submission of inception report
4 days
Bottleneck analysis and interviews with IPs 2 days
Capacity needs assessment 3 days
Key informant field interviews, FGDs with partners/ community, HH and School Survey
10 days
Data entry and analysis 4 day
Draft Report/debriefing/Finalization of report 4 day
Total 30 days
i. Management Arrangements:
To facilitate the process, UNICEF will assist in linking up the evaluation team with respective stakeholders. UNICEF will provide any coordination support needed and ensure participatory study process. UNICEF will suggest key partners for interview and consultations with the team. However, UNICEFs involvement will not influence the process in anyway as the team is expected to retain their full integrity and flexibility to determine the best approach to collection and analyzing data for the evaluation.
Evaluation Reference Group (RG): UNICEF, MoCC and provincial governmental departments will form evaluation Reference Group. The programme Focal Person would ensure seeking nominations of members from the key stakeholders and subsequent involvement of the members of the RG in delivering a high quality
[99]
evaluation. UNICEF’s ToRs for RG for evaluations would guide and steer the work of RG. The RG will provide inputs on relevant and strategic issues. The evaluation team will liaise with the RG and provide regular evaluation progress updates to them against the agreed work plan during different phases of evaluation process. The Evaluation Team will be liaising with RG and UNICEF’s Evaluation Management Committee (mandated for quality assurance) through PMER Section and responds to queries and incorporate feedback as necessary through the focal person designated by Chief, WASH.
The final draft report will be submitted to the RG for review and comments for onward transmission to PMER (UNICEF) and other relevant stakeholders.
Specific Responsibilities:
UNICEF, including field teams in provinces:
- Facilitate meetings and interactions with stakeholders
- Comment on the draft report.
- Approve final report
- Follow up on recommendations/ management response
Implementing partners/ stakeholders:
- Participate in meetings/ consultations.
- Participate in a workshop to share study findings.
- Provided needed information and documents related to project outcome
- Comment on draft report.
Government counterparts:
- Serve on Evaluation Reference Group
- Participate in meetings/consultations
- Participate in a workshop to share study findings
- Provided needed information and documents related to project outcome
- Comment on draft report.
[100]
ii. Evaluation team criteria:
All UNICEF HR business rules and workflow processes would apply in selection of the consultants’ team. The evaluation team (with a desired gender balance) shall be composed of individuals with most appropriate experience and knowledge on conducting evaluations and technical skills in the areas being evaluated including adequate knowledge and experience on gender, human rights and equity issues, given as annex 1.
Prepared by:
Kamran Naeem Signature:_______________________ WASH Specialist, Islamabad Date:
Reviewed by
Signature ________________________ Chief WASH, Islamabad Date:
Cleared by:
Signature:_____________________ Chief of Supply & Logistics Date:
Approved by:
Signature:_____________________ Deputy Representative Date:
[101]
Annex. 1. Qualification Requirement for Evaluation Team Team Leader assigned to evaluation should meet the following requirements and qualification:
‒ At least 8-10 years experience in development sector of which at least 5 years experience of evaluations and research.
‒ Experience of providing leadership in similar assignments. ‒ Other attributes e.g. analytical skills, communicating and negotiation skills ‒ Experience with KAP studies and end line evaluations is must ‒ Experience with managing/leading implementation/evaluation of PATS
programmes is an added advantage Key individuals assigned to evaluation should meet the following requirements and qualification:
‒ At least 5-10 years’ experience in community health research including gender and equity issues.
‒ Experience in report writing and analytical skills ‒ Experience in the use of statistical software e.g. SPSS ‒ Other attributes e.g. analytical skills, communicating and negotiation skills ‒ Experience with KAP studies and end line evaluations is must ‒ Experience with implementation/evaluation of PATS programmes is an added
advantage
[103]
10. Annex 03
Programme Results Framework SPSP-Rural
Programme Result
Project Outputs
Project Activities Indicators Baseline Target MoVs Frequency of reporting and data
collection
By 2017, most vulnerable and marginalized children and women are using safe drinking water, improved sanitation and hygiene practices
1. Government expenditure on WASH as a proportion of total government expenditure.
4%, 28 Billion PKR, 2010/2011 ADP Punjab = Balochistan Ratio Water/Sanitation: 3/1
5% (1% increase)
- WASH Sector study report -PSLM/JMP -PSDP/ADP -MICS
Once a year/project duration
2. Proportion of the population using an improved drinking water source (disaggregated by residence and wealth
National :91% RURAL:90% Wealth Quintile: Poorest, 2nd, 3rd, 4th, richest RURAL: HH: 6% 14% 24% 31% 38% 6.50% Other improved: 81%, 74%, 68%, 65% , 60%, 0.70% Total: 0.87%, 0.88%, 0.92%, 0.96%, 0.98%, 0.072%
National :95% National RURAL: 94%
WASH Sector study report -PSLM/JMP -PSDP/ADP -MICS
Once a year/project duration
[104]
3. Proportion of population that practices open defecation (disaggregated by residence and wealth quintile).
23% RURAL: 38.7% Wealth Quintile: Poorest, 2nd, 3rd, 4th, richest RURAL: OD: 91%, 64%, 40%, 14%, 2%, 0.2%
22% reduction in OD at national level RURAL: 35.0%
WASH Sector study report -PSLM/JMP -PSDP/ADP -MICS
Once a year/project duration
1. Demand for sanitation to achieve open defecation free status created in targeted communities
1. Proportion of targeted population that practices open defecation (disaggregated by sex, residence and wealth quintile).
2. % of targeted population willing to self-finance construction of a latrine (disaggregated by sex, residence and wealth quintile).
3. # of CBO’s and/or WUA (Water User Associations) in the targeted areas that are functioning and utilized to claim equitable access to WASH services
4. % of targeted population aware of the need for achieving ODF status in their settlement (disaggregated by sex residence and wealth quintile).
HH survey Once a year/project duration
2. Sanitation demand sustained with supply side interventions
1. Proportion of the targeted population using an improved drinking water source (disaggregated by sex, residence and wealth quintile)
2. # of latrines in the targeted constructed
[105]
with external support in form of subsidy, loan, materials or trained skilled labour.
3. % of population using WASH facilities constructed with external support in form of subsidy, loan, materials or trained skilled labour source (disaggregated by sex, residence and wealth quintile)
4. % of school children in the targeted areas with access to child friendly WASH facilities in their schools and child-friendly spaces(disaggregated by sex and age)
5. # of basic social service delivery offices that use innovative technologies to improve service delivery.
3. Improved Health and Hygiene behaviors promoted with the target community
1. % of households in the targeted areas practicing household water treatment and safe water storage/using improved drinking water who use an appropriate treatment method
2. % of households in the targeted areas with soap anywhere in the dwelling
3. % of households in
[106]
the targeted areas using and maintaining hand washing facilities with soap close to their latrines
4. % of schools in the targeted areas using and maintaining hand washing with soap facilities close to their latrines
5. % of the targeted population aware of at least one design option for constructing low cost latrine (disaggregated by sex, residence and wealth quintile)
6. % of the targeted population recalling three key hygiene messages (disaggregated by sex and wealth quintile)
4. Linkages developed with duty bearers/public institutions to support PATS
1. % of provincial budgets allocated to WASH specific social services, with separated budget lines
2. # of ODF villages through PATS programmes implemented by the government institutions
5. Villages supported in attaining Total Sanitation status
1. % of population in the targeted areas using underground drainage and waste water treatment system (disaggregated by
[107]
residence and wealth quintile)
6. PATS programme knowledge management and accountability Framework implemented
1. Demand for sanitation to achieve open defecation free status created in targeted communities
1.1 Conduct formative research to develop/customize BCC (if required)
1.1.1 Formative research to guide BCC conducted with clear guidelines to develop communication materials in the start.
- Formative research report - Payment to consultant/Partner
- One time activity
1.2 Develop/customize standard BCC package based on formative research findings and results of KAP baseline survey (if required)
1.2.1 No. of items in standard BCC package materials (IEC, electronic, print, community shows) developed/customized in local context
- Standardization of BCC package report - Payment for printing of IEC materials etc. - Standardization workshop/meetings minutes
- One time activity - Payments and meetings as per required need
1.3 Launch BCC campaigns in targeted communities and schools to create demand for sanitation.
1.3.1 No. of children (boys and girls) reached with demand creation for sanitation messages in Schools. 1.3.2 No. of people (men, women, boys, girls) reached with demand creation for sanitation messages 1.3.3 No. of people reached with demand creation for sanitation messages through electronic/print media. 1.3.4 No. of people reached with demand creation for sanitation messages through community events (theatres, puppet shows)
-Monthly progress reports - Monthly substantive review reports - CRP follow up reports - Schools events reports - VSC meeting minutes - VSC register - Media reports - Community event reports, photographs and videos - CRP reports - SLTS monitoring reports - Third party monitoring reports - Human interest stories
- One event for demand creation conducted in each school of targeted community (boys and girls) in the beginning of project - One event at each selected religious worshiping places of community in the beginning of project - One community demand creation Event in each selected community in the beginning of project - One door to door demand creation campaign for housewives in the beginning of project - One 15 days cycle of publishing advertisement in local newspapers observed for demand creation
[108]
- At least one TV/Radio promo/spot aired in the beginning of project for demand creation
1.4 Train school teachers and education officials on facilitating SLTS
1.4.1. No. of training events conducted for school teachers (men, women) on facilitating SLTS
- Training attendance sheets -Photos of training events -Monthly progress reports -SLTS training reports -Monthly substantive review reports
- One training batch for 20 selected teachers of schools in communities conducted as per size of targeted area in the beginning of project
1.5 Form and strengthen WASH clubs in schools
1.5.1 No. of WASH clubs formed in schools and strengthened with planned follow up capacity development sessions 1.5.2 No. of events conducted by school wash clubs (boys and girls) as per plan
- SLTS events reports - Monthly Progress reports - SLTS follow-up plans - Monthly substantive review reports -WASH club activity register, photographs
-One WASH club will be formed in all selected schools for SLTS in the beginning of project - Each school will have a plan for events agreed with SLTS facilitator SO throughout the project cycle
1.6 Train social organizers on facilitation of PATS in communities and schools
1.6.1 No. of training events conducted for social organizers on participatory tools and IEC material for facilitating PATS 1.6.2 No. of social organizers successful using participatory tools and IEC material for facilitating PATS in villages according to developed criteria
- Master trainers and social organizers training reports - Training attendance sheets -Training evaluation reports -Reports on capacity development events -Monthly substantive monitoring reviews - Monthly progress reports -Third party monitoring reports -MIS/GIS Database - Partner vouchers on training event costs
- One training batch for trainers/Social organizers in the beginning of projectOne training batch for CRPs/Activists in the beginning of project -One monthly substantive review will be conducted in one selected district in the programme area every month
1.7 Train CRPs on facilitation of CLTS in communities
1.7.1 No. of training events conducted for CRPs on conducting CLTS in villages 1.7.2 No. of CRPs successfully facilitating CLTS in villages according to developed criteria
- CRPs and Activists training reports -Training evaluation reports -CRPs follow up reports -criteria for evaluating CRP facilitation
[109]
1.8 Identify existing/form new and develop capacity of VSCs
1.8.1 No. of VSCs existing/formed and trained
- PRA reports - ToPs for formation of VSCs - VSC register - Knowledge management - Database for PATS - VSC's CNA reports - VSC training reports, attendance sheets, evaluations - VSC meeting minutes - Reports on other capacity development events - Documented CAPs - Photographs - Monthly progress reports - Monthly substantive monitoring review reports - Monthly Progress reports - MIS/GIS Database
- One VSC formed for - One village will receive two trainings in the beginning of project and other as refresher - PRAs will be conducted in the beginning of project - VSC meetings are recorded in VSC register as an ongoing activity -One time activity with one community/village/cluster
1.9 Develop Community Action Plans (CAPs) in targeted villages
1.9.1 No. of villages/clusters with developed community action plan 1.9.2 No. of CAPs with prioritized need for action by community on Hygiene Behaviors
- Documented CAPs One time activity with one community/village/cluster
2. Sanitation demand sustained with supply side interventions
2.1 Conduct technical trainings for masons in low-cost sanitation construction
2.1.1 No. of masons trained on sanitation construction and using their learnt skills 2.1.2 No. of latrines (other than demo latrines) constructed by masons trained under PATS programme at the end of project
- Masons training reports - Training attendance sheets -Training evaluation reports -Monthly substantive monitoring reviews - Monthly progress reports - Implementing partners vouchers on training event costs - Photographs -Masons follow-up FGD
-Training of masons is one time activity in the early stages of project, however several training events can be planned depending upon No. of masons being trained and typology of latrines which might vary from area to area.
2.2 Construct low-cost, environment friendly, gender appropriate and secure demonstration latrines
2.2.1 No. of complete demonstration latrines constructed for extremely vulnerable families as per agreed criteria with community (maximum 3%) 2.2.2 No. of demonstration latrines constructed through materials provision for vulnerable families as
- Photographs - Monthly progress reports - Community Action Plans - MIS-GIS database - BOQs and drawings of demonstration latrines - Demo latrines beneficiary data
Monthly reporting and data
[110]
per agreed criteria with community (maximum 7%)
2.3 Train entrepreneurs for establishing sanitation marts and sanitation enterprises
2.3.1 No. of entrepreneurs/manufacturer trained and producing low-cost sanitation solutions 2.3.2 % of trained entrepreneurs doing sanitation related business
- Entrepreneur’s training reports - Training attendance sheets -Training evaluation reports -Monthly substantive monitoring reviews - Monthly Progress reports - Human interest stories - Trained entrepreneurs/manufacturer follow up FGDs
One time activity
2.4 Provision of support for establishing sanitation marts and sanitation enterprises as profitable businesses
2.4.1 No. of sanitation marts provided with financial support by VSC revolving sanitation fund 2.4.1 No. of facilitated sanitation marts doing profitable business
-ToPs signed with entrepreneurs -Business plan for sanitation marts in place -Sales record of sanitation marts - Monthly progress reports -Third party monitoring reports - Case studies / human interest stories - MIS – GIS database - photographs and postal addresses of Sanitation Marts
Establishment of sanitation mart in a target area is one time activity but monitoring for profitable business is an ongoing activity and needs continuous reporting.
2.5 Develop linkages to facilitate community in accessing loan for construction of sanitation facilities
2.5.1 No. of lending institutions successfully approached to give loans for sanitation facilities 2.5.2 No. of households received sanitation loan from some lending institution facilitated by PATS programme
- Lending agreement - Monthly Progress reports - Monthly Substantive monitoring reviews
Quarterly
2.6 Provide communal/collective incentives to communities achieving ODF status
2.6.1 No. of communal water facilities constructed as incentives/rewards 2.6.2 No. of communal sanitation facilities constructed as incentives/rewards 2.6.3 No. of house sanitation facilities constructed as incentives/rewards 2.6.4 No. of house water facilities constructed as incentives/rewards 2.6.5 No. of critical community infrastructure constructed as
- ToPs for construction of communal facilities - BoQs and Drawings of communal facilities -Taking over and O&M agreement for communal facilities with relevant government department or community - Photographs of communal facilities -GIS coordinates - Monthly progress reports - Case studies/human interest stories - ODF certification of community receiving incentives
Monthly
[111]
incentives/rewards 2.6.6 No. of health and hygiene packages given as incentives/rewards
3. Improved Health and Hygiene behaviors promoted with the target community
3.1 Customize communication strategy for BCC standard package
3.1.1 WASH BCC strategy in place for PATS programme.
- Communication strategy document - Comments by C4D section on communication strategy
One time activity in the beginning of project
3.2 Launch mass media BCC campaigns
3.2.1 No. of people reached with WASH BCC messages through electronic/print media. 3.2.2 No. of people reached with WASH BCC messages through community events (theatres, puppet shows)
-Recorded TV Promos and radio spots/talk shows - Media rating -Pre and Post KAP in community and schools - Monthly progress reports - Community event reports - CRP/social organizer report
Ongoing activity as per communication Plan
3.3 Launch BCC campaigns on promotion of low-cost, appropriate and informed sanitation solutions
3.3.1 No. of school children (boys and girls) reached with messages through SLTS activities. 3.3.2 No. of people (men, women, boys, girls) reached with WASH BCC messages through CRPs
- Monthly progress reports - Community event reports -SLTS events reports - CRP/social organizer report - Human interest stories -Monthly Substantive monitoring visit reports - Water quality testing reports
Ongoing as per communication strategy/plan
4. Linkages developed with duty bearers/public institutions to support PATS
4.1 Conduct capacity development events to enhance ownership of PATS by government officials and political leaders
4.1.1 No. of capacity needs assessments (CNA) conducted for relevant public sector stake holders 4.1.2 No. of capacity development plans developed for relevant public sector stake holders by Implementing Partner 4.1.3 No. of capacity development events undertaken for relevant public sector stake holders by Implementing Partner 4.1.4 No. of government officials and political leaders participated in capacity development and key events of the project. 4.1.5 No. of advocacy initiatives undertaken with duty bearers/political
-Stake holder analysis report -CNA report -Stake holder capacity development plans -Minutes of meetings with the government and political leaders - Duty bearers training attendance sheets -Training evaluation reports -Notifications issue by government -Minutes of meeting confirming agreement on advocacy agendas -Reports on capacity development events -Copies of MoUs signed by government departments - Strategy/Plan developed by IP for each stake holder as part of stake holder analysis
- Monthly - Event specific
[112]
leadership to support PATS 4.1.6 No. of MoUs signed by implementing partner with relevant public institutions responsible for sanitation 4.1.7 No. of villages jointly selected for PATS programme by relevant Government duty bearers with IP
4.2 undertake advocacy to facilitate rewards by government for ODF communities
4.2. 1 No. of ODF communities received rewards/incentives by government
-Village assessment and selection reports notified/endorsed by government -Annual Development Plans of government reflecting funding for PATS project and Rewards for ODF -Political leadership/Government official’s public/media announcements -end of project evaluation -monthly progress reports -Quarterly review reports -end of project report -Joint progress monitoring reports generated
5. Villages supported in attaining Total Sanitation status
5.1 Mobilize communities for ODF status through SLTS, CLTS or other Community Approaches to Total Sanitation (CATS)
5.1.1 No. of households with new latrines constructed after start of the PATS programme 5.1.2 No. villages declared as ODF by community 5.1.3 No. of villages Verified as ODF by Implementing partner and independent monitors 5.1.4 No. of people (men, women, boys, girls) in communities declared as ODF 5.1.5 No. of people (men, women, boys, girls) in communities verified as ODF
- CRP reports -monthly progress reports -midterm review report -end of project report -PCRWR reports -Photos, BOQs and drawings -IP sitreps on ODF status -ODF verification reports by IP -ODF certificates -ODF notifications by government
Monthly Quarterly Annual/project duration bound
5.2 Demonstrate total sanitation by
5.2.1 No. of model villages with laid sewers, WW
-Photos, BOQs and drawings of WW collection and treatment
[113]
constructing WASH infrastructure in selected ODF certified villages, as model villages
collection and treatment system through PATS programme
facilities -Meeting Minutes with government -IP sitreps on ODF status -monthly progress reports end of project report
5.3 Undertake advocacy with duty bearers to support total sanitation projects
5.3.1 No. of advocacy initiatives undertaken to promote communal underground drainage system linked WW treatment system and communal SWM system 5.3.2. No. of ODF villages provided with underground drainage and WW treatment facility by relevant public sector stake holder/duty bearer.
- Monthly progress reports -Quarterly review reports -End of project report -Annual Development Plans of government reflecting funding for PATS project and Rewards for ODF villages -Political leadership/Government official’s public/media announcements -Joint progress monitoring reports
Monthly
6. PATS programme knowledge management and Framework implemented
6.1 Launch PATS project inception workshop
6.1.1 Inception workshop conducted in the beginning of project
- inception/launching workshop report/proceedings
One time activity
6.2 Establish baseline against indicators in results framework
6.2.1 Baseline established in the beginning of project
-Baseline documented -Baseline dissemination/inception/launching workshop
One time activity
6.3 Conduct regular joint ‘Monthly Substantive Monitoring Reviews’ with implementing partners in the field
6.3.1 No. of joint progress monitoring reports (IP and public institution) 6.3.2 No. of monthly substantive monitoring reviews’ visits conducted for confirming/measuring performance indicators in results framework 6.3.3 No. ‘monthly substantive monitoring review’ reports shared among partners
- Substantive monitoring visit reports -Knowledge management data base on PATS -Progress reports -Minutes of Quarterly Review meetings -Mid-term review reports
Monthly
6.4 ensure all reports in M,E&L framework are timely received and are responsive to agreed formats and quality
6.4.1 % of responsive reports submitted as per M,E&L framework
- Progress reports -Minutes of Quarterly Review meetings
As per reporting schedule
6.5 Conduct structured quarterly review meetings with field staff
6.5.1 No. of quarterly progress reviews conducted with IP and minutes documented
- Minutes of Quarterly Review meetings - third party monitoring reports
Quarterly
6.6 Conduct internal 6.6.1 mid-term review -Mid-term review report Six monthly
[114]
midterm review workshop with IPs, field staff, CRPs and community representatives/activists
workshop report endorsed by M&E section of UNICEF
- Comments by M&E section
6.7 Document best practices for communication and replication
6.7.1 No. of Human interest stories documented and accepted as per UNICEF guidelines and disseminated
-Knowledge management data base on PATS -Progress reports -Minutes of Quarterly Review meetings -Mid-term review reports -Human interest stories published on UNICEF website, Annual WASH reports -End of project lessons sharing workshop report -End of project report by IP -practices/documentaries
Monthly Quarterly Six monthly End of project
6.8 Conduct end line survey
6.8.1 accepted end line survey report
- End line survey report - Lessons sharing workshop report
One time activity at the end of project
6.9 Conduct end of project lessons sharing workshop with all stakeholders
6.9.1 Lessons sharing workshop conducted and feedback documented
- Lessons sharing workshop report -List of participants in lessons sharing workshop - Photos - Payments made for lessons sharing workshop
One time activity at the end of project
6.10 Provide data to knowledge management databases on developed formats
6.10.1 Up to date knowledge database available in UNICEF for access to all stake holders at the end of project 6.10.2 Up to date GIS database available with government for access to all stake holders at the end of project
- Database up to date in government custody - on-line link to database
One time activity at the end of project
[115]
11. Annex 04 Stakeholders & Roles Following key stakeholders were involved during the implementation of SPSP-Rural. The role and geographic coverage of each stakeholder is highlighted in the table below.
Stakeholder Role
National Level
- Climate Change Division, The Cabinet Secretariat
- Policy guidance & formulation, monitoring of programmatic approaches
- Water and Sanitation Programme (World Bank), Water Supply and Sanitation Collaborative Council (WSSCC), Global Sanitation Fund (GSF), Plan International, Water Aid (UK) Plan Pakistan
- To coordinate WASH/PATS efforts/plans, share UNICEF learning, Advocate PATS implementation, organize learning events
- DSI Pakistan - Technical advice on sanitation designs and sanitation marketing
- RSPN - Capacity development of IPs and government on PATS/CATS approaches, PRA, community mobilization
- APEX Consulting - Third party monitors for field monitoring
Provincial Level
- Provincial Government Departments (PHED, Health, Education and LGRDD)
- Provincial policy advocacy/development, capacity development,
District Level
- Government Departments at local level (PHED, Health, Education and LGRDD)
- Planning and Implementation of selective hardware components (identification of schools for schools sanitation interventions, planning and execution of rural water schemes rehabilitation, rewards, field monitoring and ODF certification,.
Implementing Partners
- Muslim Aid - Qatar Charity - TVO (Trust for Voluntary
Organization) - BRSP (Baluchistan Rural Support
Programme) - WDCO (Women Development &
Community Organization) - NCBP (Nations Capacity Building
Programme) - NRSP (National Rural Support
Programme) - SAFWCO (Sindh Agricultural and
Forestry Workers Coordinating Organization)
- Sabawoon (Sindh Agricultural
- D. G. Khan Punjab - RajanPur Punjab - RajanPur Punjab
- Jaffarabad Balochistan
- Jaffarabad Balochistan
Naseerabad Balochistan
- Jacobabad & ShikarPur Sindh
- Kashmore Sindh
- Kohat KPK
[116]
Stakeholder Role
and Forestry Workers Coordinating Organization)
- SEED (Social Efforts for Education & Development)
- IRSP (Integrated Regional Support Program)
- SRSP (Sarhad Rural Support Programme)
- D. I. Khan KPK
- Mohmand, Bajour (FATA)
- Tank (KPK) – Kurram (FATA)
[117]
13. Annex 05 13.1. Evaluation Matrix for Inception Report - SPSP Evaluation
1. Relevance
Evaluation Q1: To what extent the programme design and implementation was relevant to national/provincial priorities and needs of men and women beneficiaries in terms of achieving expected results across all sociocultural groups including the vulnerable in the targeted communities?
Sub-questions What to Check (Indicators)
Data sources
Methods of Data Collection
To what extent were the project objectives and strategies aligned with government priorities, commitments and harmonisation efforts and also to PATS? To what extent the analysis of then-exiting WASH needs of community (if conducted) informed programme design and TOC? How far programme objectives were consistent with priorities and needs identified by men and women beneficiaries across different sociocultural groups? How aligned were programme objectives toward UNICEF’s equity focus for its integration into systems/processes and project deliverables? To what extent was the project aligned with UNICEF’s CATS’ principles?
- What lessons can be drawn from the introduction of CLTS triggering methods later in programme cycle, particularly of potential stigma for those who could not construct toilets vis-à-vis the ethical and human rights principles? To what extent use of PATS master trainers was relevant and beneficial to training cadre of human resources in sanitation sector?
Evidence of relevance at the various tiers of operation (provincial, district and community levels) as well as relevance to geographic areas, including views of key stakeholders on targeting Evidence of contextualising and aligning with national/provincial strategies as well as with the PATS approach. Evidence of alignment of the goals and results with identified needs of the targeted community. Evidence of using UNICEF’s equity focused approach in the project design. Evidence of contextualising and aligning with CATS approach. Evidence of stigma with late/no toilet construction for non-adherent families. Evidence of CLTS in harmony with ‘do no harm’ principle of ethics. Evidence of PATS master trainers delivering standard trainings and maintaining knowledge base for sector
- Pre-KAP & Post KAP Comparison
- SPSP Results Framework Review
- UNICEF - IPs - MoCC - PHED - Local
Government
- Evaluation Consultants
- Pre-Post KAP Results
- WSP/WB (CR)WAP
- PIEDAR
- Baseline
study –
context
analysis
report
- PCAs with
IPs and
RSPN
- Training
reports
- Literature Review - Survey (HHS,
SSS)
- Interview - UNICEF
- Interview – IPs - Interview –MoCC - Interview –
PHED
- Interview – Local Government
- Interviews with
Donors/INGOs
(WSP/WB,
RSPN)
- Interview
Delegation Lead
SACOSAN
Member of
National
parliament
- Interview RSPN management and PATS master trainers
[118]
2. Effectiveness
Evaluation Q2: To what extent were project implementation strategies effective and successful in achieving the planned outcomes/results for rich and poor men and women across all sociocultural groups including the vulnerable in the targeted communities?
Sub-questions What to Check (Indicators) Data sources Methods of Data Collection
What were the most effective activities/outputs within overall programme which contributed to behaviour change against OD, and why? How effectively activities and outputs were designed and conducted from the outset to involve men, women from various sociocultural and age groups? How the implementation strategies could have been improved/altered for them to have been more effective in achieving the results? To what extent the equity principles were integrated in the project implementation (social mobilization, BCC, supplies facilitation, and others) and substantiated by project results (e.g. targeting of poorest of poor for demo latrines, change in access to WASH services and facilities for the lowest quintile)? How was the gender analysis carried out within the communities and used to inform programme implementation? Can the results be gender disaggregated?
To what extent has the project delivered on its outputs and achieved any early outcomes? How do the results compare across the project? What factors helping/impeded the delivery of the outputs and achievement of early outcomes and whether were used for improvement?
Evidence of what worked effectively / identified best practice in terms of approaches/outputs linked to behaviour change against OD. Evidence of a logical and robust results framework with a clear attribution and vertical logic. Evidence of the validity of the theory of change including any changes over the implementation period (Including views of stakeholders). Evidence of what worked effectively in terms of approaches and as per guidelines - including departure/variation from project design during implementation. Evidence of use of gender analysis and use of Gender Equality Policy for WASH. Evidence of operationalizing the Equity Approach in the various project work streams and identifying benefits. Evidence of outputs delivered (NOTE: Analysis will be done using Output Indicators presented in Annex 6 for all outputs. Results will be further disaggregated on the basis of gender, geographical location, wealth quintiles and disability. Including pre-post KAP scenarios. Evidence of learning lessons and their use.
- UNICEF Project Documents
- UNICEF - Implementing
Partners - Post KAP results
- Pre-KAP & Post KAP Comparison
- VSC (men,
women), SOs,
CRPs, Masons,
Marts/Entreprene
urs, SMCs,
Teachers,
Students, Wash
Club
- Review of
partnership
guidelines, micro
assessments and
baseline partners’
capacity
assessment &
development
report.
- Men and women beneficiaries’ feed back
- UNICEF Staff
- Literature Review - Survey
(Household, School)
- Interview with UNICEF
- Interview with Implementing Partner
- FGDs with VSCs, SOs, CRPs, Masons, Marts/Entrepreneurs, SMCs, Teachers, Students, Wash Club
[119]
How has the project changed the perception of improved services and who has benefitted the most (men, women, children, or others including the disabled) and in what ways?
To what extent did the combination of SLTS along with CLTS resulted in enhancing effectiveness of the programme design in achieving the desired results? To what extent SPSP-Rural Phase I flow of activities was more effective in mobilizing people to construct toilets as compared to CLTS programmes in similar settings?
UNICEF and IPs views on project achievements, factors helping or hindering project outcomes. Views of CRPs, VSC, School Teachers, Children etc. to check change in perception in relation to availability of services. Evidence of the complementarity of SLTS and CLTS approach and its overall relevance within the PATS model. Comparison scenarios between SPSP and CLTS activities
[120]
3. Efficiency
EQ3: To what extent have the outputs of the project been achieved (in terms of quality and quantity) with the allocated resources/inputs (such as funds, time, and procedures)?
Evaluation Questions and Sub-questions
What to Check (Indicators) Data sources Methods of Data Collection
Were the resources allocated adequate in terms of quantity to help achieve project outputs?
- To what extent were the outputs/approaches delivered in line with planned quality standards?
- Was the project duration sufficient/adequate for implementation of PATS in delivering the expected outputs?
- How does the project implementation of PATS compare in terms of efficiency with other institutions within country?
- What was the project
investment for each toilet constructed?
- How much clarity was offered by UNICEF in partner selection guidelines about the capacities required for project implementation? To what extent were these adhered to? Whether these helped in achieving results effectively and in time?
- What areas can be improved for partners’ selection in future programmes?
Evidence of resource
efficiency – comparison of planned vs. actual expenditures for varied outputs.
Evidence of minimum
quality standards (such as for inputs, outputs and strategies, and their
application. Evidence of efficient use of time (in terms of project duration) and timely delivery of outputs (in terms of activities). Evidence of efficiency with
other national Sanitation
programmes.
Cost incurred on project against each toilet constructed in project by IPs or community. IPs and UNICEF Staff views on effectiveness of partnership guidelines and identification of weaknesses or need for change. IPs and UNICEF Staff views on application of partnership guidelines in view of PATS project requirements
- Project Documents – project proposal & reports, budget, expenditure tracking sheets,
- Post KAP Results
- 3rd Party Reports
- Evaluation Consultant
- BoQs of latrines
-
- Review of Project Documents
- Review of 3rd Party Reports
- Survey (Household, School)
- Literature Review
- Interview with UNICEF and IPs
- FGDs - IDIs - Checklists
[121]
4. Sustainability
Evaluation Q 4: To what extent various stakeholders including the public sector and civil society are likely to sustain the behaviour change related to the goals of the programme after it has ended?
3.
Evaluation Questions and Sub-questions
What to Check (Indicators) Data sources Methods of Data Collection
Was sustainability aspect of the programme a part of the design from the outset?
What is the level of commitment amongst stakeholders to sustain the ODF status in project areas? Are there any challenges? How far SLTS has contributed to sustaining ODF status in communities? To what extent have partnerships (with government, donors, civil society organizations, and other stakeholders) been established to foster sustainability of effects? To what extent political will and government commitment exist for PATS scale-up (in terms of increased budgetary allocation, organisational restructuring, improved harmonisation and coordination)?
Evidence of sustainability initiatives, strategy and approach in the project design. Stakeholders views on the sustainability of strategies used under the joint programme. Evidence of challenges to sustainability. Evidence SLTS ensuring ODF in communities. Evidence of strengthened involvement/buy in/ownership of the project in the relevant stakeholders. Evidence of Government interest and priority of PATS (possibility of Government ownership, leadership and scale-up).
- IPs - MoCC Officials - PHED - Local
Government
- MNAs/MPAs - Communities - IP Reports
- Third party
reports
- Interview with MoCC Officials
- Interview with PHED (Province & District)
- Interview with Local Government (Province & District)
- Interview with Political Representatives (District Level)
- Interview with IPs
- FGDs with VSCs - FGDs with
WASH clubs
5. Impact of Unforeseen Events on Project
Evaluation Q 5: To what extent was the project implementation affected/influenced by unstable, conflict or emergency situation?
Sub-questions What to Check (Indicators) Data sources Methods of Data Collection
- Did the project (including IP PCAs) have a Risk Assessment and undertook mitigation measures (both for natural disaster and conflict)?
- Was the project implementation affected at any time by security, conflict or any disaster/emergency situation? If so, what aspects were affected and what measures were taken to
Evidence of Risk Assessment and Mitigation Strategy and measures in design and at operational level. Views of UNICEF Staff and IPs on any incidents (security, conflict, disaster etc) that affected project delivery and how it was dealt with including evidence of lesson learning.
- UNICEF Staff - Implementing
Partner
- IP and Third party reports
-
- Interview with UNICEF
- Interview with Implementing Partner
- Literature review
-
[122]
mitigate/respond to the situation?
- What are the lessons learnt to plan for unforeseen in future?
6. Cross Cutting Theme: HRBA
Evaluation Q 6: To what extent did the project observe HRBA and/or contributed to observance of Human Rights in Pakistan as an implicit objective of the project?
Evaluation Questions and Sub-questions
What to Check (Indicators) Data sources Methods of Data Collection
- Did programme made efforts to developing capacity of duty bearers to take responsibility and empowering communities and contributing to addressing immediate needs.
- To what extent did the project support Government through the use of PATS approach in making progress on the MDG 7 (Target 10): (Halving, by 2015, the proportion of people without sustainable access to safe drinking water and basic sanitation)
- To what extent has the project ensured that all schools in the target area have adequate child friendly water and sanitation facilities and hygiene education project?
- To what extent was there a common understanding of HRBA in UNICEF, in context of Right to Water and Right to sanitation? With Staff? With Government? With Implementing Partners? And among sanitation sector stakeholders? If not, why?
- What formal and informal
mechanisms were put in place/existed for duty bearers and rights holders to participate/have a voice in the various activities/interventions of the project in terms of
Evidence of integrating and using HRBA in the project. Evidence of focused attention to child related issues. Stakeholders views on rights based approach used in the project. Stakeholder views on right holder/duty bearers linkage.
- MoCC Officials - UNICEF Staff - PHED
(Provincial level)
- Local Government (Provincial level)
- Education Department
- Health Department
- Implementing
Partners
- Children
- School
Teaches/SMC
- Interview with MoCC Official
- Interview with UNICEF
- Interview with PHED
- Interview with Local Government
- Interview with Implementing Partners
- Interview with Children
- Interview with Teachers/SMC FGD
- Interview with UNICEF Staff
- Interview with IPs
[123]
programming planning, implementation, monitoring and evaluation?
7. Financial Risk Management
Evaluation Q 7: To what extent the project has observed and applied UNICEF’s standard financial management practices with Implementing Partners as well as with Government?
Evaluation Questions and Sub-questions
What to Check (Indicators) Data sources Methods of Data Collection
What was the financial impact of security, conflict or any disaster/emergency situation during the implementation of the project? (including cost extension, slow implementation rate, associated security costs etc)
Was UNICEF’s zero tolerance to corruption policy applied in the project?
To what extent alternative implementing methods could have been applied to increase project’s cost effectiveness?
To what extent micro assessment process remained significant during the implementation of the project in terms of managing financial risks and did these assessments help in identifying key issues at the beginning of relationships with IPs and Sub IPs? What have been the areas of improvement from UNICEF and IP perspective?
To what extent the project (IPs and Sub-IPs) remained compliant with UNICEF Policies? What organizational arrangements were introduced for transparent procurements?
To what extent value for money was a consideration during implementation of project outputs and how do the results relate to this?
Evidence of security/conflict/disaster/ Emergency impact on the project. Evidence of application of UNICEF’s zero tolerance corruption policy. Evidence of cost effective methods. Evidence of significance and usefulness of conducting micro assessments process. Views of UNICEF and IP on improvements in financial risk management. Evidence of compliance with procurement policies and procedures. Note: Adherence to procurement policies & procedures is an auditing domain. Evidence of cost effectiveness and application of value for money principles. Evidence of cost effectiveness in terms of comparative approach and projects across partners and government.
- Project documents and others resources
- MOCC, PHED and LG Depts.
- Development partners (WSP, RSPN)
- UNICEF Staff - Implementing
Partner
- Review of previous Audit Reports, Spot checks, micro assessments and baseline partners capacity assessment & development report
- UNICEF Bottleneck Analysis and Minutes of coordination and other meetings between UNICEF and GOP
- Literature Review (project reports and expenditure statements)
- Interview with UNICEF
- Interview with Implementing Partner
- Documents Review as mentioned in Data sources
- Interview with MOCC, PHED, LG
- Interview with development partners (WSP and RSPN)
[124]
To what extent the implementation modality of working through CSOs and public sector partners prove to be cost efficient and effective? If not, why not?
What were the UNICEF/GOP coordination, management and administrative arrangements for this project? What were the key strengths and areas of improvement in the operational and administrative procedures/arrangements?
Views of GoP and UNICEF Staff on coordination, management and administrative arrangement.
[125]
15. Annex 06 Key indicators for the evaluation include the output, outcome and goal indicators which are drawn from the SPSP-Rural results framework. A baseline exists against them and a post-KAP will be carried out during the evaluation which will form the basis
Results Hierarchy
Operational Indicators Survey Instrument
Data Source/ Respondent
By 2017, most vulnerable and marginalized children and women are using safe drinking water, improved sanitation and hygiene practices
4. Government expenditure on WASH as a proportion of total government expenditure.
Secondary Sources Review
Secondary Sources
5. Proportion of the population using an improved drinking water source (disaggregated by residence and wealth Quintile)
HHS HH Respondent
6. Proportion of population that practices open defecation (disaggregated by residence and wealth quintile).
HHS HH Respondent
1. Demand for sanitation to achieve open defecation free status created in targeted communities
5. Proportion of targeted population that practices open defecation (disaggregated by sex, residence and wealth quintile).
6. % of targeted population willing to self-finance construction of a latrine (disaggregated by sex, residence and wealth quintile).
7. # of CBO’s and/or WUA (Water User Associations) in the targeted areas that are functioning and utilized to claim equitable access to WASH services
8. % of targeted population aware of the need for achieving ODF status in their settlement (disaggregated by sex residence and wealth quintile).
HHS HHS HHS HHS
HH Respondent HH Respondent HH Respondent
HH Respondent
2. Sanitation demand sustained with supply side interventions
6. Proportion of the targeted population using an improved drinking water source (disaggregated by sex, residence and wealth quintile)
7. # of latrines in the targeted constructed with external support in form of subsidy, loan, materials or trained skilled labour.
8. % of population using WASH facilities constructed with external support in form of subsidy, loan, materials or trained skilled labour source (disaggregated by sex, residence and wealth quintile)
9. % of school children in the targeted areas with access to child friendly WASH facilities in their schools and child-friendly spaces(disaggregated by sex and age)
10. # of basic social service delivery offices that use innovative technologies to improve service delivery.
HHS HHS HHS SSS KII District Officials
HH Respondent HH Respondent HH Respondent Head/Senior Teacher PHED/LG&RD District Officials
[126]
3. Improved Health and Hygiene behaviours promoted with the target community
7. % of households in the targeted areas practicing household water treatment and safe water storage/using improved drinking water who use an appropriate treatment method
8. % of households in the targeted areas with soap anywhere in the dwelling
9. % of households in the targeted areas using and maintaining hand washing facilities with soap close to their latrines
10. % of schools in the targeted areas using and maintaining hand washing with soap facilities close to their latrines
11. % of the targeted population aware of at least one design option for constructing low cost latrine (disaggregated by sex, residence and wealth quintile)
12. % of the targeted population recalling three key hygiene messages (disaggregated by sex and wealth quintile)
HHS HHS HHS SSS HHS HHS
HH Respondent HH Respondent HH Respondent Teacher HH Respondents HH Respondents
4. Linkages developed with duty bearers/public institutions to support PATS
3. % of provincial budgets allocated to WASH specific social services, with separated budget lines
4. # of ODF villages through PATS programmes implemented by the government institutions
Secondary Sources Review HHS
Context Analysis HH Respondents
5. Villages supported in attaining Total Sanitation status
2. % of population in the targeted areas using underground drainage and waste water treatment system (disaggregated by residence and wealth quintile)
HHS HH Respondent
[127]
16. Annex 07 16.1. List of People Met For SPSP (Rural) Final Evaluation
Summary Location Interviews/KIIs FGDs
Islamabad 8
Sindh 8 9
Balochistan 5 10
Khyber Pakhtunkhwa 8 3
Punjab 14 7
Total 43 29
Islamabad Interviews/KIIs S #
Name Designation/Organization Interview District/Village
1 Imtinan Ahmed Development Solutions International Interview Islamabad
2 Tania Khan WSSCC (Water Supply and Sanitation Collaborative Council)
Interview Islamabad
3 Shandana Khan CEO RSPN Interview Islamabad
4 Bashir ud Din RSPN Interview Islamabad
5 Irfan Tariq Ministry of Climate Change Interview Islamabad
6 Farhan Sami WSP (Water and Sanitation Program) Interview Islamabad
7 Muhammad Hafeez
WaterAid Islamabad
8 Tahir Jelani Apex Interview Islamabad
Kamran Naeem WASH Specialist, UNICEF, Islamabad Islamabad
Total # of Interviews = 8
Sindh – FGDs FGD – Community Resource Persons -
Male Ahmedabad, Misripur, Jacobabad FGD – Children (Girls) - Dao Syed
Bukhari Ghari Khero, Jacobabad
1 Ahmed Ali 1 Aisha
2 Zulfiqar Ali Jakhrani 2 Fareeda
3 Muhammad Raheem 3 Haleeman
4 Karimdad 4 Hira
5 Ali Murad 5 Asma
6 Shabir Ahmed 6 Tabasum
7 Hafiz Abdul Hameed 7 Hameeda
8 Ghaus Bux 8 Kainat
9 Nazar Muhammad 9 Shumaila
10 Naghma
FGD – Teachers Female – Dao Syed Bukhari Ghari Khero, Jacobabad
FGD – Community – New Goth Jakhro Ghari Khero, Jacobabad
1 Zahida Perveen 1 Sadiq Ali
2 Jameela 2 Muhammad Bux Mirali
3 Rani 3 Hussain Bux Malgani
4 Ghaus Bux Maachi
5 Abdul Wajid Jakhro
FGD – Village Sanitation Committee - Men New Goth Village – Alan Pur - Jacobabad
FGD – Social Organizers – (Male/Female) Jacobabad
1 Tahir Ali 1 Ghazala SO SUKAAR
2 Abdul Waheed 2 Safdar Abbas SO SUKAAR
3 Muhammad Ishaque Brohi 3 Naseebullah SO PREPARED
[128]
4 Muhammad Aslam 4 Taj Muhammand SO PREPARED
5 Shaban Ali Katohan
6 Abdul Wahab Jakhero
7 Muhammad Anwar
8 Abdul Wahab Rind
FGD – Teachers (Male) – GPS - QadirPur City - Jacobabad
FGD – Children (Male) – GBPS Jacobabad
1 M Nawaz Khoso 1 Mohkamdin Mugheri
2 Syed Bashir Ahmed 2 Sajjad Ali Khoso
3 Mehboob Ali Khoso 3 Ayaz Hussain Bhand
4 Muhammad Jan Bhand 4 Shaib Ahmed L ashari
5 Qurban Ali Maghsi 5 Liaqat Ali Khoso
6 Ahmed Ali Bhand 6 Irfan Ali Khoso
7 Nisar Ahmed Chohan 7 Sajid Ali Chamdani
8 Barkat Ali Brohi 8 Bilal Ahmed Brohi
9 Ali Jan Khoso
FGD – Village Sanitation Committee - Female - Jacobabad
1 Kaz Bano
2 Noor Bano
3 Anwari
4 Wazeeran
5 Shehzadi
6 Noor Jehan
7 Rani
8 Mai Birai
9 Azeeman
10 Shahnaz
[129]
Sindh – Interviews/KIIs
S #
Name Designation/Organization Interview District/Village
1 Dr. Abdul Aziz Hambi
Health Officer – District Health Office Interview Jacobabad
2 Sabir Ali Sheikh XEN PHED (JCB Division) Interview Jacobabad
3 Azizullah Odho DEO Education – Education Department Interview Jacobabad
4 Jamil Ahmed Rakhsani
Mason – New Goth Jakhro Ghulab Mari
Interview Jacobabad
5 Iftikhar Ahmed Mart Owner – Gari Khero – Meran Pur Interview Jacobabad
6 Syed Shahnawaz Mart Owner – Gari Khero – Alan Pur Interview Jacobabad
7 Syed Bashir Ahmed Shah
Political Activist Interview Jacobabad
8 Gul Wali IP Staff Interview Jacobabad
Total # of Interviews
Total Interviews = 8
Total # of FGDs & Groups
Total FGDs = 9
Balochistan – FGDs FGD – Teachers - Female –
Murad Ali - Jaffarabad FGD - School Children (Girls),
SohbatPur – GGS, Jaffarabad
1 Rukiya 1 Saleema
2 Azra 2 Resham
3 Nasreen 3 Yasmeen
4 Rizwana 4 Saima
5 Raheela 5 Afshan
6 Abida Perveen 6 Hira
7 Benazir Parveen CRP 7 Naima
8 Zubeda 8 Faryal
9 Rukiya Bibi 9 Raesa
10 10 Nageena
FGD - School Teachers (Male), Govt Boys Primary School, Goth Mehar Ali Khosa – SohbatPur Jaffarabad
FGD - School Children (Boys), Meher Ali Khoso - Jaffarabad
1 Ghulam Haider 1 Muhammad Asif
2 Nabi Bux Khoso HM 2 Sabir Ali
3 Abdul Sattar Brohi 3 Din Muhammad
4 Attaullah Brohi 4 Kareem Bux
5 M Anwar Brohi 5 Sajid Ali
6 Ghulam Fareed
7 Bakhtiar
8 Zahoor Ahmed
9 Hafeezullah
10 Izhar Hussain
FGD – Village Sanitation Committee (Male/Female) – Azam Khan - Jaffarabad
FGD – Village Sanitation Committee (Male) – Meher Ali Khoso - Jaffarabad
1 Muhammad Panah 1 M Naseeb Meher Ali
2 Rahim Bux 2 Sardar Khan Khoso
3 Shah Nawaz 3 M Anwar Pechoho
4 Zubeda Khatoon 4 Raza Muhammad
5 Muhammad Siddique 5 Shahad Khan Khoso
6 Naseeban (Female) 6 Allah Bux Sarki
7 Muhammad Hassan 7 Ali Nawaz Sarki
8 Fida Hussain Sarki
FGD – Masons – Jhat Pat Town – WDCO Office – Jaffarabad
FGD – Community Resource Person – (Male) – Meher Ali Khoso – Jaffarabad
1 Muhammad Bux 1 Ali Hassan
[130]
2 Ghulam Muhammad 2 Zafrullah
3 Saddam Hussain 3 Mehar Khan
4 Abdul Rashid 4 Barkat Ali
5 Allah Bux 5 Ali Murad
6 Inayatullah 6 Muhammad Umer
7 Muhammad Kabeer 7 Rahib Khan
FGD – Social Organizers – (Male/Female) BRSP Office - Jaffarabad
FGD – Community Resource Person – (Female) – Azam Khan – Jaffarabad
1 Haider Ali 1 Asia Bano
2 Jam Muhammad 2 Bhagul
3 Tahira (F) 3 Benezir
4 Amina (F) 4 Shareefan
5 Arbela (F) 5 Gull Bano
6 Fareeda 6 Arbab Khatoon
7 Abdul Hameed 7 Zainab
8 Muhammad Nawaz 8 Benezir Bibi
Balochistan – Interviews/KIIs S. #
Name Designation/Organization Interview District/Village
1 Dr. Jamal Din Jamal District Health Officer (DHO) Interview Jhatpat, Jaffarabad
2 Rafique Ahmed Khoso
JWP District Leader – Political Activitist
Interview Jhatpat, Jaffarabad
3 Rafique Kurd DPM BRSP Interview Jaffarabad 4 Alam Gurgag PM WDCO Interview Jaffarabad 5 Tariq Magsi Training Coordinator WDCO Interview Jaffarabad
Total # of Interviews Total Interviews = 5
Interview Jaffarabad
Total # of FGDs & Groups
Total FGDs = 10
Interview Jaffarabad
[131]
Punjab – FGDs FGD – Community Resource Person -
Male Kot Chuta – DG Khan FGD – School Children – (Boys)
Mukhtar Arain – Guristaniwala – Mutafarik Chohan DG Khan
1 Abdul Shakoor CRP Birmani Gharbi 1 M. Ahmed
2 Riaz Hussain CRP Birmani Sharki 2 M. Idrees
3 Habib Ullah Basti Hasanabad Colony 3 M. Owais
4 4 M. Shahzad
5 5 M. Rab Nawaz
6 6 Sheryar Ahmed
7 7 Ali Haider
8 8 Husnain Abbass
9 9 M. Shahzad
FGD – Teachers (Male) – Basti Birmani Kot Chuta – DG Khan
FGD – Village Sanitation Committee - Male – Kot Chuta - DG Khan
1 Ashiq Husain 1 Sadiq Hussain (CRP Phatuwala)
2 M. Afzal 2 Zulfiqar Phutuwala
3 Imtiaz Ahmed 3 Nadeem
4 Munir Ahmed 4 Mir Hazaar (VSC Member)
5 Irshad Husaain 5 Haqim Husaain
6 Rashid Ahmed 6 Munir Ahmed (VCS Member)
7 7 Zahid Hussaing (VCS/MO)
FGD – Teachers (Female) – Basti Birmani Kot Chuta – DG Khan
FGD – Village Sanitation Committee - Female – Kot Chuta - DG Khan
1 Rukhe Zara 1 Samrina Bibi
2 Mrs. Ghulam Hafsa 2 Khurshid Bibi
3 Nargis Zahra 3 Naheed Riaz
4 Zahida Perveen 4
5 Tasleem Zahra 5
FGD – Children (Girls) – Mukhtar Arain – Guristaniwala – Mutafarik ChohanDG Khan
1 Aneela Aslam 8 Tahira Batool
2 Asifa Batool 9 Shaista Atta
3 Arifa Batool 10 Saira Manzoor
4 Uzma Karim 11 Azra Bibi
5 Rukia Bibi 12 Kanwal Tehseen
6 Nasreen 13 Nazia Mukhtar
7 Mehwish 14
Punjab – interviews / KIIs – updated asmat
S # Name Designation/Organization Interview District/Village
1 Younas Leghari Secretary Union Council Mutafariq Chohan UC
Interview DG Khan
2 Muhammad Iqbal
Secretary Union Council Nawan UC
Interview DG Khan
3 Nazim Abbas CRP Interview Mazari wala DG Khan
Nasir Abbas Supervisor Health & Nutrition Interview Choti Zeren DG Khan
4 Asif Qureshi DCO – DG Khan Interview DG Khan
5 Amjad Tareen ADC Interview DG Khan
6 Saif ur Rehman AC Revenue Addition Director of ADLG&CD
Interview DG Khan
7 Rafia Bibi Primary Class Teacher Interview DG Khan
8 Tariq Leghari Community Resource Person Interview DG Khan
9 Sehar Abrar Community Resource Person Interview DG Khan
10 M Akhtaer Supervisor Head Teacher Interview DG Khan
11 Mushtaq Ahmed AEO Dept. of Education Interview Kot Chuta/Choti
[132]
Talpur DG Khan
12 Zahid Hussain Mart Owner Mutafariq Chohan Interview Mutafariq Chohan DG Khan
13 M Yousaf Entrepreneur Interview Kot Chuta/ Mutafariq Chohan DG Khan
14 Zubeda Bibi CRP Interview Mutafariq Chohan DG Khan
Total # of Interviews
Total Interviews = 14
Total # of FGDs & Groups
Total # of FGDs = 7
Khyber Pakhtunkhwa – FGDs FGD – Village Sanitation Committee -
Kohat FGD – Children (Boys)- village Ali
Zai,Kohat
1 Israr Din 1 Muqadas Ali
2 Mohammad Ishtiaq 2 Mohammad Ali
3 Sabz Ali Khan 3 Zain Hassan
4 Gul Mohammad 4 Ali Haider
5 Mohammad Naeem 5 Shaiq Hussain
6 Qambar Gul 6 Noor Hussain
7 Adil 7 Sadaqat Ali
8 Sajjad 8 Anwar Iqbal
9 Ghazanfar Abbas
10 Asher Hussain
FGD – Children (Girls) - Kohat FGD – Community Resource Person (Girls) - Kohat
Fareeda Musarrat
Fatima Basharat
Lal meena Uzma
Leeda Rabia
Subeena
Zarmeena
Rubeena
Sakeena
Nazia
Laila
Khyber Pakhtunkhwa KIIs / Interviews S # Name Designation/Organization Interview District/Village
1 Hazrat Gul ADEO/Education department IDI Kohat
2 Mohamad Hayat Secretary UCs/Local Government IDI Kohat
3 Abdul Wahab Social organizer IDI Kohat
4 Qamar Gul Community resource person IDI Kohat
5 Abdur Rehman Market owner/entrepreneur IDI Kohat
6 Mohammad Yousaf Teacher IDI Kohat
7 Noor Naz Social Organizer IDI
8 Shamim Akhter Teacher
Total # of Interviews Total Interviews = 8
Total # of FGDs & Groups Total # of FGDs = 3
[134]
17. Annex 08 17.1. List of Participants Provincial Bottleneck Analysis &
Government Partners Capacity Development Workshops
KPK-FATA-Bottleneck Analysis and Capacity Development Workshop - 8th April, 2014
Sr. No NAME DESIGNATION ORGANIZATION
1 Haroon Rashid S.O K.P.H
2 Majeed Gul AEO Kurrum Agency EDUCATION DEPARTMENT
3 Azhar Iqbal Team Leader SRSP
4 Qabil Shah MT (health) MT (health)
5 Mohammad Arif Team Leader SRSP
6 Alam Zeb Assistant Director LG FATA
7 Shah Behram TANK PHED
8 Khalid Naeem M.O.I MC Tank
9 Abdul Hafeez ADO EDUCATION DEPARTMENT
10 Sanaullah S.M.O Health
11 Nahid Khan S.D.O LG & RDD
12 Najeeb ullah Sub Engineer PHE BAJAUR
13 Muhammad Idrees Director Programme IRSP
14 Zaki ullah Consultant AAN ASSOCIATES
15 Hazrat Gul ADEO Kohat EDUCATION DEPARTMENT KOHAT
16 Muhammad Hayat LG & RDD UNION COUNCIL
17 Liaqat Zaman Sub Engineer PHE KOHAT
18 Anwar Zaib Programme Manager SABAWON
19 Sahibzada M.Shoaib CEO SEED
20 Shahzad Hadi Assistant Director (tech) LG & RDD
21 Sajjad Akbar UNICEF
22 Zain Raza Project Coordinator AAN ASSOCIATES
23 Nadeem Haider MD AAN ASSOCIATES
24 Zarqa Khan PO Operations SRSP
25 M.Shakaib Jan WASH Officer UNICEF
[135]
Balochistan Bottleneck Analysis and Capacity Development Workshop – March 13, 2014 Sr. No NAME DESIGNATION ORGANIZATION
1 Abdul Qayyum Director Finance PHED 2 Fatehullah Manager Field
Operation BRSP
3 Rafique Khan DPM BRSP 4 Aftab Aziz ADLG LG 5 Abdul Khaliq AIE LG LG 6 Ahmed Nawaz CEO -NCBP NCBP 7 Mohammad Yousaf Chief Health Educator Health DEPT
8 Zaqir Hussain Training Coordinator WDCO 9 Alam Khan Programme Manaer WDCO 10 Khalil Khetran Feid Monitor APEX 11 Mohammad Ahsan R.O APEX 12 Amanullah S/M BRSP 13 Syed Shaed Nawaz M&E Officer BRSP 14 Amoon Pervez Finance Officer NCBP 15 Abid Hussain H-R Officer NCBP 16 Naqeeb Agha PA,WASH UNICEF 17 Shazia Aman WASH Officer UNICEF 18 Nadeem Haider MD Consultant AAN ASSOCIATES 19 Mukhtar WASH UNICEF
Sindh - Bottleneck Analysis and Capacity Development Workshop – March 25, 2014
Sr. No NAME DESIGNATION ORGANIZATION
1 Kamran Naeem WASH Specialist UNICEF 2 Nadeem Haider Consultant AAN ASSOCIATES
3 Zahida Jamali Manager WASH SAFWCO 4 Qurban Ali ED SAFWCO 5 Rafique Ahmed G.M Operation SAFWCO 6 Khurram Aslam Chief Munciple Office LG DEPARTMENT 7 Mubashara-Iram WASH Officer UNICEF 8 Zahid Durrani WASH Specialist UNICEF 9 Amjad Khan Regional General
Manager NRSP
10 Qaiser Shah Wash Officer UNICEF 11 Muhammad Usman Executive Engineer PH.E.D 12 Engineer Ali Director (tech) PH.E.D 13 Faheem Junejo LG M&E LG Department
[136]
Punjab - Bottleneck Analysis and Capacity Development Workshop Sr. No NAME
1 Zulqarnain (SO(PH))
2 Abdullah Fahad
3 Tariq Mahmood
4 Sabahat Ambreen
5 Najeeb Aslam
6 Nadeem Haider
7 Salman Yousuf
8 Mustajab Ahmad
9 Syed Waseem Hassan
10 Abdullah Asad Khan
11 Umer Ali Aziz
12 Faisal
13 Aqeel
14 Atif
15 Usman
[137]
18. Annex 09 Sample Size and List of Villages for Post-KAP
Actual Sample Size
Province District Female Male Grand Total
Punjab
DG Khan 200 200 400
Rajan Pur 200 200 400
Sub Total - Punjab 400 400 800
KPK
D.I.Khan 200 200 400
Kohat 199 201 400
Tank 200 200 400
Sub Total - KPK 599 601 1200
Balochistan
JAFFARABAD 200 200 400
NASIRABAD 193 206 399
Sub Total - Balochistan 393 406 799
Sindh
JACOBABAD 206 197 403
KASHMORE 199 199 398
SHIKAR PUR 199 201 400
Sub Total - FATA 604 597 1201
FATA
Bajur Agency 0 400 400
Kurram Agency 0 385 385
Mohmand Agency 0 400 400
Sub Total - Punjab 0 1185 1185
GRAND TOTAL 1996 3189 5185
[138]
Sindh Jacobabad
IP Name Union Council Village Name
SUKKAR Misripurr Ahmed Abad
Habibullah Bugti
Muhammad Abid Khoso
Piyaro Khoso
Ramzan Jakhrani
Hussain Bux Ghunio
Wakil Ahmed
PREPARED Allan Pur New Goth Jakhro
Noor Muhammad Brohi
Hazar Khan Brohi
Taro Khan Dasti
Dawoo Syed bukhari
Meeran Khan Luhar
FRDP Qadirpur Nazeer Husain Mugheri
Qadir Bux Bhayo
Mundar khoso
Mazar khan Mirali
Qadir pur city-2
Hazoor Bux Mugheri
Haji Fakhruddin khoso
Kashmore
Union Council Name of Village Village Changed
Lalao
Rabban Khan Dahani
Mohd Mureed Bajkani Sahab Khan Bhalkani
Hadi BuxBahalkani
Abdul Haq Bahlkqni
Jeeand Khan Dahani
Saifal
Naik muhammad bajkani Piyaro Khan Nindwani
Allah wasayo nindwani
Tagio bijarani
Yar muhammad nindwani
Baland khan nindwani
Suhliani Gulan Khan Dahani Motio Lashari
Molvi Suraj Din Khoso 1
Ghulam Nabi Jakhrani
Nazar Mohammad Gujrani
Mir Sobho Sadique Khoso
Tangwani Imam Bux Bajkani
Mohd Hassan Malak
Yar Mohd Golo
Sounharo Malak
Kareem Bux Nindwani
[139]
Shikarpur
Tehsil Union Council Village
Khanpur
Khanpur faqir Wahid bux
Moor Unchar
Taj Muhammad brohi
Eid Muhammad brohi
Muhammad Hassan
Mian Sahib Abdullah khoharo
Faiz Muhammad Meer badar
Naik Muhammad brohi
Mageera Wahi
Din muhammad panwar
Shikarpur Jahan Wah Agha Jan Akhtar
Aghan Jan Pathan
Rasool Bux chandio
Ali Bahar (Gull Muhammad Marfani)
Sardar Naseer, Muhammad Wesar
Sultankot Qabil khan chandio
Aitbar Ali
Abdil
Fatah muhammad brohi
Suhbat Machi
[140]
Punjab Dera Ghazi Khan
UC Name Village/Basti Name
Mutfariq Chahan Norang Shah (Gharbi)
Basti Haji hussain Khan
Basti Mazari
Gadi wala
Basti Pushiya Sharqi
Basti Pushiya Gharbi
Mana Ahmedani Ramay Wala
Chah Charhoye wala
Basti Qureshi
Chakir Wala
Dubba Duri
Basti Changwani (Budhan)
Nawan Basti Hajan Muraad Khan(Masu wani+lashari+bhatti+Faqeeri)
Dodhay Wala
Khan Wala(Wahnes Wala+G Serwer+G Yasin Aslam+Shumla Khosa)
Gadray Wala(tariq Bajwa+Ahmad Bux+Abdul aziz+Rahim bux)
Basti Muhammad Afzal Khan
Basti Haji Faiz Muhammad+ Durrhatta
Rajanpur
UC Name Village/Basti Name
Noshehra Gharbi / Burrey wala
Basti Channar # 0
Basti Channar # 1
Basti Ada Khalil Abad
Mud Bhutta
Basti Rasool Bakhsh
Burrey wala Basti Haji Riaz Sirla
Basti Peer Bakhsh jogyani
Dur Muhammad Bazgir
Basti Nazoo Khan Bazgeer
Basti Ameer Shameer
Jahanpur Basti Nazeer Ahmad
Basti Muhammad Ramzan
Basti Ghulam Husain Sandhi/Mumtaz Lohar
Basti Abdul Sattar Nariwal
Basti Pati Juma Arain
Rakh Fazilpur Basti Iqbal jut
Basti Haji Jind wada/ hasil/Saich
Dunia Pur
Basti Shah pur
Basti Ghulam Mustafa Horra
[141]
Balochistan Naseerabad
UC Name Village/Basti Name Villages Changed
Bedar Goth Shakoor Khoso
Goth Haji Sobdar Khoso
Goth Ghoram Chinjni
Goth Nassur Din Khoso
Changaiz Khan Sasoli Lanja Khan
Quba Sher Khan Goth Bangul Khan Sasoli
Goth Badal Khan Nichari
Goth Bilawal Khan Pyanhar
Goth Haji M sadiq Talani
Goth Haji Shair Dil Nicharri
Manjhoti Sharqi Goth Chaker Khan Jhakrani
Goth Ali Madad Umrani/Peral
Goth Dildar Khan Umrani
Raja KHAN Umrani
Haji Hazaar Khan Umrani
Shori Drabhi Goth Abdul Rehman Shar
Goth Jamal Din Baghdar
Goth Musa Khan Bugti Goth: Peer Bux Shar
Sardar Allah Waraya Shar Goth:Ghulam Muhammad Shar
Abdul Kareem Domki
Jaffarabad
UC Name Village Name Replacement
Khuda e Dad Darya Khan Javeed Abad Muhammad Ismail Langhu Sohrab Khan Nizam ud Din / Abdul Aziz / Nisar Ahmad Name Change
Noz Band Nozband
Masood Abad Azam Khan Izhar Khan Khoso Mujeeb Khan Khoso Hamid Pur Jan Mohammad Mohammad Salah Abdul Aziz Meeral Khan Goth Kareem Baig Sohabat Pur City Goth Nawab Khan Bugti Mohalla Goth Ghulam Qadir Brohi
[143]
Khyber Pakhtunkhwa Kohat
UC Name Village/Basti Name
Urban VI Afridi Banda
Khwaja Abad
Gul Abad
Eidgah colony merozai
Afzal Abad
Tappi
Muhammad Zai Muhammad Zai
Naway Kally
Kooz Kaley
Rajgaan Banda
Haider Rehman Banda
Speen Jumaat Mohalla
Ali Zai Ali Zai
Narey Kale
Yourd Kaley
Zear Kamar
Gul Kada (Kana)
Chashma Metha Khan
Dera Ismail Khan
UC Name Village/Basti Name
Parova Parova 1
Babber Kacha
Chah Pattar
Jhok Jhandir
Jhok Sikhani
Naiwela Jatta
Naiwela City
Gara Aashiq
Gara Tahir
Kotha Kumar
Lunda Sharif Umar Boba
Lunda Sharif
Shameer
Babbi
Rang Pur Janubi
Mahra Sikandar Janubi 2
Jhok Daruli
Lunda Para
Pachani
Dhawa
[144]
Tank
UC Name Village/Basti Name
Uttar Uttar
Shieikh Sultaan
Garra Matta
MamraizPathan
Jammal Awan
Gul Imam Daraki
Ghazi khel
Gul Imam
andary
Naway gara
Jattatar Tatoor
Adam Abad
Garra Hayyat
Garra Budda
Garra Pathaar
Gara Baloch Proper Gara Baloch
Razzaq colony
Dhari aba khel
Kot piran
Kirri saidal
FATA
Kurram Agency
Tehsil / Area Village/Basti Name
Lower Kurram
Baghzai Kalay
Dagar Kalay
Jabba Kalay
Baleesh Khel
Ibrahim Zai
Lower Basatu
Central Kurram Khapa Kalay
Johdara
Said Ali Mela
Tanorak
Kotkai
Koda Kalay
[145]
Mohmand Agency
Village/Basti Name
Monda/sapare
Dew Dhere
Kirra
Khadi Kas(Khas)
Palosay kally
Speena Khawra/Prang ghar
Boocha kally
Toor Zerik Boocha
Nao Kally
Sroo Shaheed Kally
Pata Saly/Patta Salle
Karkarna kally
Bajaur Agency
Tehsil UC Name Village/Basti Name
Khar Loe Sum Rashakai
Miangano Kalay
Loe Sum
Zega Dherai
Zoor Bandar Gundo
Shawo
Zoor Bandar
Sasmasai
Nawagai Kotkai Sharif Khana
Kotkai Charmang
Ghari Kalay
Shingar
Nawagai Redawan
Lawargai
Anar Bagh
Charmango Kalay
List of Villages Replaced Province / District
UC / Area Sampled Village/Basti Name
Replaced Villages
Sindh
Kashmore Lalao Mohd Mureed Bajkani Sahab Khan Bhalkani
Saifal Naik muhammad bajkani Piyaro Khan Nindwani
Suhliani Gulan Khan Dahani Motio Lashari
[146]
20. Annex 10 List of Documents Reviewed
(2010). Design of School WES.
González, A. H. (2013). 5th South Asian Conference on Sanitation.
Group, U. M. (2012). Global Evaluation of the Application of a Human Rights Based Approach in
UNICEF Programming - Final Report Vol II.
Pakistan Bureau of Statistics, G. (2011-12). Pakistan Social Living Measurement Survey.
SAARC. (2013). The Kathmandu Declaration. Kathmandu.
Syed Shah Nasir Khisro, I. (n.d.). CLTS Intervention in Post Emergency Context in Pakistan.
(n.d.). The 'shame question' in CLTS.
(n.d.). The 'shame question' in CLTS.
UNDP. (n.d.). General Guidelines for Inception Reports.
UNEG. (2005). Norms for Evaluation in the UN System.
UNEG. (2005). Standards for Evaluation in the UN System.
UNEG. (2008). Ethical Guidelines for Evaluation.
UNICEF & Ministry of CA&D, G. (n.d.). Child Friendly School Standards.
UNICEF. (2007-08). Multiple Indicators Cluster Survey .
UNICEF. (2009). Community Approaches to Total Sanitation.
UNICEF. (2010). Policy on Gender Equality and the Empowerment of Girls and Women.
UNICEF. (2011). Guidance on Conducting a Situation Analysis of Children’s and Women’s Rights.
UNICEF. (2011). WASH in Schools, Monitoring Package.
UNICEF. (2011-12). End of Programme Evaluation, Rural Sanitation in Flood-affected Districts
(RuSFAD III).
UNICEF. (2012). Funding Proposal to DFID UK for Life-saving WASH and Child Protection Assistance to
Flood Affected Populations in Balochistan, Punjab and Sindh Provinces.
UNICEF. (2012). Funding Proposal to DFID UK for WASH Humanitarian Assistance to Insecurity
Affected People in the Federally Administered Tribal Areas (FATA) and Khyber Pakhtunkhwa
(KP).
UNICEF. (2012). UNICEF Funding Proposal to DFID UK.
[147]
UNICEF. (2012). UNICEF Funding Proposal to DFID UK for Life-saving WASH and Child Protection
Assistance to Flood Affected Populations in Balochistan, Punjab and Sindh Provinces.
UNICEF. (2013). Baseline Report for Sanitation Programme at Scale in Pakistan – SPSP Rural.
UNICEF. (2013). Bottleneck Analysis for Sanitation Programme at Scale in Pakistan – SPSP Rural.
UNICEF. (2013). Contextual Analysis for Sanitation Programme at Scale in Pakistan – SPSP Rural.
UNICEF. (2013). Strategic Guidance Note on Institutionalizing Ethical Practice for UNICEF Research.
UNICEF. (n.d.). Global CATS Evaluation: Draft Theory of Change.
UNICEF. (n.d.). Programme Results Framework SPSP-Rural.
UNICEF. (n.d.). Terms of Reference (ToR) Technical Assistance for SPSP-Rural End of Programme
Evaluation.
UNICEF. (n.d.). Water, Sanitation & Hygiene (WASH) in Schools.
[148]
22. Annex 11 Research Instruments
End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in
Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)
INDEPTH INTERVIEW UNICEF & PARTNERs
Introduction to the Interviewee: I am ,a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation of UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. I am here to get your reflections, experiences and recommendations about the programme, which will help us to assess, analyse and document programme progress, results and areas for further improvement. This will help at large in designing or developing future sanitation programmes in the country and/or extension or scale-up of this initiative. I thank you for taking time for this interview today. I shall be asking you questions about how the programme activities were implemented in your area and what changes this programme brought in the lives of the community in regard to improved access and availability of sanitation services. I anticipate the interview will last about 60 minutes, and appreciate any information you can provide. Please note that this interview is important for the success of this assessment. Your answers are completely confidential and shall remain with the evaluation team. At the start we seek your consent to make use of this information for analysis and reporting and may also cite it in the report while making reference to your name, if needed. Also, we are recording the proceeding to avoid any loss of information and seek your consent to record our discussion. Finally, we would request you to please allow us to make photographs of project interventions and yourself for evidence and communication purposes.
[149]
Note: This is to underline that the IDI instrument has been designed as such to offer flexibility to the interviewers to adapt questions as to the specific role of the organization, respondents profile, understanding of programme operations and overall knowledge of sector. Hence one consolidated tool has been developed for multiple stakeholders (as listed above).
Name of Respondent: _____________________ Designation _____________________ Organisation _____________________ Location: _____________________ Date of Interview: _____________________
Name of the Interviewer: _____________________
Interview Questions: Please listen carefully to the questions and provide your candid feedback.
RELEVANCE Q 1. Was WASH the priority/one of the priority needs in rural context? If
yes, how was this established by UNICEF? How does the
programme design addresses the needs of the rural poor (equity)?
Q 2. To what extent the SPSP results framework is consistent with
results based management principles & approach (impact,
outcome, outputs, inputs)? If so, how does this approach manifest
and to what extent the vertical logic was valid (at design stage)?
Q 3. What was the process and basis/criteria used by UNICEF and IPs
to select the geographic regions where the programme has been
implemented (for instance selection of provinces, districts,
agencies, tehsil, UCs and villages)? In your view if the most
deserving regions were selected for programme interventions?
[150]
Q 4. To what extent you find the two models i.e. Community and School
Led Total Sanitation (CLTS & SLTS) complementary? Did the
programme achieve desired results by using both models, please
elaborate?
Q 5. What experiences/lessons were learned by UNICEF and IPs around
programme design and implementation (list what worked, what did
not and why? Substantiate with measure of success, scale, and
lessons learnt for future)?
Note: The analysis to be desegregated for following (components/subcomponents)
1. DEMAND CREATION
What Worked What Didn’t Work Lessons Learnt Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
[151]
2. SUPPLY FACILITATION
What Worked What Didn’t Work
Lessons Learnt Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
3. HEALTH AND HYGIENE PROMOTION
What Worked What Didn’t Work
Lessons Learnt Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
4. LINKAGES WITH DUTY BEARERS FOR PATS
What Worked What Didn’t Work
Lessons Learnt Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
5. VILLAGES SUPPORTED IN TOTAL SANITATION STATUS
What Worked What Didn’t Work
Lessons Learnt Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
[152]
6. PATS KNOWLEDGE MANAGEMENT FRAMEWORK
What Worked What Didn’t Work
Lessons Learnt Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
7. PARTNERSHIP MANAGEMENT
What Worked What Didn’t Work
Lessons Learnt Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
8. INTEGRATION OF CROSS CUTTING PRIORITIES
(Gender & Equity, Unforeseen Events, Rights Based Approach) What Worked What Didn’t
Work Lessons Learnt Success
Scale Rank
Highly
Successful
Moderate
Poor
Fail
9. SCHOOL LED TOTAL SANITATION
What Worked What Didn’t Work
Lessons Learnt Success Scale
Rank
Highly
Successful
[153]
Moderate
Poor
Fail
Q 6. In the provinces where project was implemented, what is the
status of water and sanitation policies? Are these
compliant/consistent with Pakistan Approach to Total Sanitation?
Q 7. In which way and to what extent the current programme design
(SPSP Rural) relates to PATS, please elaborate? In what respects
the programme differs/introduces elements different to PATS?
Q 8. Are you aware of any mechanisms (coordination meetings/forums
etc) that relevant government departments (at national and
provincial levels) are using to align the efforts of government and
development partners in WASH (as per PATS)? To what extent are
[154]
they successful in their efforts? In which ways UNICEF contributed
to harmonization and what could be improved further (including
the role of development partners)?
EFFECTIVENESS
Q 9. To what extent the programme design (UNICEF project document
to DFID and IP PCAs) offered clarity into project implementation –
defining expectations, standards of achievement, contributions,
coordination, etc required from Implementation Partners?
Q 10. To what extent the actual programme implementation followed the
design and how much has been achieved as to the stated outputs?
Were there any departures from the planned
methods/implementation plan? What were the points of departure,
and what were the reasons for this? To what extent did this
departure help in achieving the programme results?
[155]
Q 11. Did the programme design have a clear and articulated Theory of
Change (with clarity around various stages)? Can you elaborate on
the theory of change? What inputs contributed to which outputs
and on to Outcomes/Goals? Did the change happen as per the
assumed/perceived theory? If not what changes need to be
factored into future programme design/theory of change?
Q 12. To what extent has the project delivered on its outputs and
achieved the outcomes/results through the various interventions
undertaken? How do the results compare geographically (in
provincial context) and across partners?
[156]
Q 13. What factors helped/impeded the delivery of the outputs and
achievement of the outcomes across regions and partners?
Q 14. How has it changed the perception of improved services and who
has benefitted the most (men, women, children, or others
including the disabled) and in what ways?
Q 15. How far did the UNICEF partners’ selection guidelines offer clarity
(processes, controls and accountabilities) about the requisite
capacities (experience, expertise, geographic footprint, attitudes)
amongst partners for PATS programme implementation?
Q 16. To what extent were partner selection guidelines/standards (of
UNICEF) adhered to? And to what extent did this help in achieving
results effectively and in time? In which areas UNICEF may need to
improve the process/guidelines and controls for partners selection
particularly for WASH/PATS programmes?
[157]
EFFICIENCY Q 17. Has the project achieved the desired outputs in time, technical
resources & allocated financial resources? If not what elements
constrained the project delivery & why?
Q 18. Were any quality standards prescribed and set for outputs/pillars
before implementation? To what extent the outputs were delivered
in line with the prescribed quality standards e.g. community
mobilization (in terms of strategy, PRAs, time available, skills of
SOs, status of VSCs, role of NRSP trainers), BCC (IEC materials,
communication guidelines, etc.), supplies facilitation (training of
masons, latrine designs, demo latrines, marts/entrepreneurs),
health and hygiene awareness and rewards?
Q 19. To what extent the 3rd Party Process Monitoring (guidelines/tools,
reports and actual application) contributed to management
decision making in terms of efficient delivery (timeliness and
resources utilization), consistency with standards/quality? In
[158]
which ways the third party monitoring process/implementation
could be improved to help decision makers?
Q 20. To what extent the knowledge management interventions were
carried out as planned (please elaborate what were the
interventions - technical reviews, baseline/evaluation results, third
party monitoring, best practices, periodic reviews, research and
policy advocacy)? Which ways it helped accumulating
knowledge/resources, dissemination (internally and externally)
and informed the PATS standardization and delivery across
Pakistan (for all stakeholders involved in PATS)?
Q 21. To what extent the centralized capacity building arrangements
(through a partners NRSP) facilitated building capacities of
implementing partners to what extend it helped achieving time and
resource efficiencies (including contract management)? What were
the advantages/disadvantages of the centralization capacity
building approach and what needs to change to effectively build
capacities (of IPs and government) on sustainable basis?
[159]
SUSTAINABILITY Q 22. Did the project design have a sustainability approach/strategy for
the various initiatives undertaken? To what extent those been
implemented successfully, if not what needs to be
incorporated/changed in sustainability design and
implementation?
Q 23. [NOTE: From Implementing Partners Only]
What is the level of commitment amongst stakeholders (i.e. GoP, IPs, communities) to sustain the initiative without external support? In your view which of the project interventions and results may sustain whilst others may not? And why and what could be done to improve the sustainability of those elements in future?
UNFORESEEN EVENTS Q 24. Did the programme design (UNICEF and IP PCAs) have a Risk
Assessment and Mitigation Strategy (both for natural disaster and
conflict)? Was it clearly articulated and to what extent was it
implemented effectively?
[160]
Q 25. Was the project implementation affected at any time by security,
conflict or any disaster/emergency situation? If so, what aspects
were affected and what measures were taken to mitigate/respond
to the situation? Were the measures taken successful, if not then
why not? What are the lessons learned for future?
Note: Please ask if previous natural disaster response/recovery context was taken into account for design of interventions/approach in SPSP Rural.
GENDER & EQUITY Q 26. To what extent outputs and outcomes were designed from the
onset to contribute to attainment of gender (men, women, children,
older person, disabled, adolescent and others) equality/equity (in
relation to UNICEF policy)? Did the planned programme results
(including M&E systems) clearly state in measureable ways
contribution to gender equality, and reduction of possible gender
discrimination in any particular aspect? Can results of the
programme be disaggregated by sex, age, and disability?
[161]
Q 27. Did UNICEF or IPs undertake structured/guided (UNICEF and/or IP
guidelines) gender analysis for programme design and delivery (at
critical programme stages)? To what extent it helped mapping the
possible effect on burden on women/vulnerable groups, and/or
possible benefits (e.g. possible increase in girls’ retention in
schools)? In which ways the programme strategies and
interventions were realigned to the findings of gender analysis?
Q 28. In which ways the SPSP Rural facilitated larger policy and
implementation related changes as to promote gender equity (with
Government, Implementing Partners, Beneficiaries)? For instance,
in terms of gender policy formulation, implementation, positive
discrimination in staffing recruitments (equal number of field staff
SOs, beneficiary groups VSCs), project/intervention specific
gender analysis?
Q 29. To what extent the equity (vulnerability) principles were integrated
in the social mobilization, BCC, supplies facilitation, and others
and substantiated by programme results e.g. targeting of poorest
of poor for demo latrines, change in access to WASH services and
[162]
facilities for the lowest quintile? What processes evolved for
equity integration, how successful were those and what may need
to change in future?
Q 30. To what extent the programme supported the Government’s use of
PATS approach in making progress on the Environmental
Sustainability MDG i.e. Target 10: Halving, by 2015, the proportion
of people without sustainable access to safe drinking water and
basic sanitation?
Note: Did the programming try to address, directly or indirectly, the issues identified by the CRC, CEDAW and/or UPR in SPSP-Rural?
Q 31. To what extent and in which ways the HRBAP (Human Rights
Based Approaches to Programming) is manifested in SPSP Rural
programme design? Give examples of proposed approaches,
interventions, and results? What challenges if any were faced in
application of HRBAP?
Q 32. To what extent there was consistency in understanding of HRBAP
by UNICEF, IPs and government partners? If not, what was done to
bring all at same level and how successful has that been?
[163]
Q 33. What formal and informal mechanisms were put in place/existed
for duty bearers and rights holders to participate/have a voice in
the various activities/interventions of SPSP-Rural in terms of
programming planning, implementation, monitoring and
evaluation? In what ways participation was ensured?
Q 34. Has the programme ensured that all schools in the target area
have adequate child friendly water and sanitation facilities and
hygiene education programme? How was children participation
ensured in the process?
Q 35. How do you compare the application of the ‘respect, integrity’
approach vs ‘shame shock disgust’ approach in WASH in
Pakistan? Which one has been more effective and why?
[165]
End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in
Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)
INDEPTH INTERVIEW
GOVERNMENT & POLITICIANS
Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation of UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. I am here to get your reflections, experiences and recommendations about the programme, which will help us to assess, analyse and document programme progress, results and areas for further improvement. This will help at large in designing or developing future sanitation programmes in the country and/or extension or scale-up of this initiative. I thank you for taking time for this interview today. I shall be asking you questions about how the programme activities were implemented in your area and what changes this programme brought in the lives of the community in regard to improved access and availability of sanitation services. I anticipate the interview will last about 60 minutes, and appreciate any information you can provide. Please note that this interview is important for the success of this assessment. Your answers are completely confidential and shall remain with the evaluation team. At the start we seek your consent to make use of this information for analysis and reporting and may also cite it in the report while making reference to your name, if needed. Also, we are recording the proceeding to avoid any loss of information and seek your consent to record our discussion. Finally, we would request you to please allow us to make photographs of project interventions and yourself for evidence and communication purposes. Note: This is to underline that the IDI instrument has been designed as such to offer flexibility to the interviewers to adapt questions as to the specific role of the organization, respondents’ profile, and understanding of programme operations and overall knowledge of sector. Hence one consolidated tool has been developed for multiple stakeholders (as listed above).
Name of Respondent: _____________________ Designation _____________________ Organisation _____________________ Location: _____________________ Date of Interview: _____________________
Name of the Interviewer: _____________________
[166]
Interview Questions: Please listen carefully to the questions and provide your candid feedback.
Section 1: RELEVANCE Q 36. Is WASH one of the priority needs in rural (your district/province)
context? If yes, which regions/groups need the services most and
why (probe men, women, equity)? Which regions and groups are
at greatest risk for poor WASH conditions and what are the
implications (including potential)?
Q 37. Are you aware of UNICEF’s SPSP-Rural programme? Was there
any coordination between your department, UNICEF and the
Implementing Partners on geographic area selection (for instance
selection of districts, agencies, Tehsil, UCs and villages),? If yes,
was this based on any criteria? In your view do you feel that the
most deserving regions were selected for programme
interventions?
Q 38. What is the role of your dept. /organization in delivery of WASH
services especially PATS operationalization? Do you feel the
UNICEF SPSP is aligned to PATS principles and pillars (explore if
he/she looks at logic elements as - partnerships, capacity building,
demand creation, facilitation of supplies) and how?
Q 39. Have you/department been involved in SPSP design and
implementation, if yes how? Also, share your thoughts on
your/depts. role at any/various stages of the programme
(consultation, design, assessment, planning, co-ordination &
information sharing, sharing of technical resources, services
delivery, monitoring, evaluation and learning, policy advice)?
Q 40. Can you share your thoughts if this project i.e. SPSP (design and
implementation) could have benefitted more from engaging with
your organization and other public agencies for PATS
implementation & potential replication/scale-up, and how??
Q 41. In which ways and extent your department benefitted from the
engagement with the SPSP project e.g. District level capacity
enhancement, improved planning and identification and others?
What areas should the future programme focus (in terms of
capacity development) to help government departments adopt,
[167]
replicate and scale-up PATS? Probe more for ideas for future
programme design?
Q 42. To what extent is the project design consistent with National
Sanitation Policy (2006)/PATS and/or to Provincial policies on
water and sanitation? Are PATS or Provincial Strategy/Action
Plans available (for national and provincial stakeholders)?
Q 43. Is your department/organization involved in harmonizing efforts
for PATS implementation and if yes what practical steps been
taken (for instance working groups, coordination meetings etc) to
align the efforts of government and development partners in
WASH/PATS? If these been successful, how and if not what needs
to be done and how UNICEF/other development stakeholders
could facilitate that?
Q 44. PATS outlines engaging multiple stakeholders and also
introducing CATS and SLTS? Do you find two approaches
complementary? Did SPSP achieve desired results by using both
models, please elaborate and how your dept contributed or
potentially contribute to application of either one or both?
[168]
Section 2: Effectiveness
Q 45. To what extent you see the SPSP programme (in particular your
organization given engagement) has been able to achieve the
desired outputs/outcomes? In which particular areas the SPSP
performed better and less in others and why?
Q 46. To what extent the SPSP has been in application of various PATS
pillars i.e. mobilization, demand creation, supplies facilitation,
achieving and sustaining ODF, rewards? Probe into the reasons
for their assessment/perception and share your dept./organization
contributions? Which areas (design and implementation) the next
project needs to improve to achieve better results?
Q 47. To what extent this project has contributed to building capacities
of public sector partners in particular your organization in order to
achieved desired results? How capacity needs were appraised,
what strategies were evolved to address needs (training, systems
development, equipment, finances and others)? Which ones
worked well & why – why others were not as successful?
Q 48. To what extent the engagement in SPSP and capacity development
have lead to institutionalization of PATS in terms of new policies,
structures, systems, practices within your department? What are
the factors that are still hindering PATS institutionalization?
Section 3: Efficiency
Q 49. To what extent do you find that there was a balance between the
project targets and resources allocated/utilized (technical,
financial and time)? Probe more, if respondent is aware only of
public organizations targets, times and resources?
Q 50. In your view were any measures put in place to avoid/control
wastage and misuse of resources at all levels (performance
management, audits, others) while working with partners as well
[169]
as government? What were those measures and to what extent
those measures contributed to achieving transparency,
accountability and cost efficiencies, please explain?
Q 51. Are you familiar with any cost benefit analysis undertaken of
CATS/CLTS implementation (within public sector or other non-
profit sector inside Pakistan or abroad) approaches in terms
reaching out with messages, convincing to construct latrine and
others? if yes please share references/documents?
Q 52. What changes could be introduced in the project design and
implementation to achieve maximum resource efficiencies? If yes,
please share what are those and how would these contribute to
achieving resource efficiency?
Section 4: SUSTAINABILITY
Q 53. In your view, if SPSP had an articulated sustainability
approach/strategy for project interventions particularly those in
which your organization partnered? How successful has that been
implemented and if that links it with relevant government depts.?
Q 54. What is the level of commitment amongst stakeholders (i.e. GoP,
IPs, communities) to sustain the initiative without external
support? In your view which of the project interventions and
results may sustain whilst others may not? And why?
Q 55. What is the level of GOP commitment to PATS/Rural sanitation
replication & scale-up in terms of increased budgetary allocation,
organizational restructuring, improved multi-stakeholder
coordination for planning, implementation and monitoring, intra-
government understanding of PATS to deliver better results)?
Section 5: GENDER, EQUITY & RIGHTS BASED APPROACH Q 56. To what extent the SPSP design and implementation (mobilization,
BCC, supplies facilitation, rewards) is informed of gender analysis
in terms of engaging and benefitting varied groups particularly
vulnerable (children, women, older person, men, disabled)? In your
[170]
view if any of the project interventions caused any negative impact
on gender relations & exclusion, please elaborate?
Q 57. To what extent the SPSP design and implementation (mobilization,
BCC, supplies facilitation, rewards) is informed of equity
considerations in terms of engaging and benefitting poor and
marginalized (poor, disabled)? In your view if any of the project
interventions caused any negative impact on equity divide &
exclusion, please elaborate?
Q 58. Did the programme support Government’s use of PATS approach
in making progress on the Environmental Sustainability MDG i.e.
Target 10: Halving, by 2015, the proportion of people without
sustainable access to safe drinking water and basic sanitation?
Did the programming try to address, directly or indirectly, the
issues identified by the CRC, CEDAW and/or UPR in SPSP-Rural?
Thank You for your time and candid response.
[171]
End of Programme Evaluation
UNICEF’s Sanitation Programme at Scale in Pakistan (SPSP) – Rural Through
Pakistan Approach to Total Sanitation (PATS)
FOCUS GROUP DISCUSSION (FGDS) CHILDREN
(Boys & Girls)
Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation for UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. We thank you for taking time for this focus group today. We shall be asking you questions about how the programme activities were implemented and what changes this programme brought in your life. During the FDG, please help yourself to tea and water. If you need to get up to use the rest room, feel free. We expect the focus group to take 1 to 1-½. We shall record/document the proceedings and seek your consent to allow us to use this discussion and/or interpretation for our report. Purpose: Before we get started, we would like to give you a little background about focus groups and why we are doing this focus group discussion with you. Focus groups are intended to give us a sense of how a group of people feels about a certain subject. We often want confirmation that others feel this way, so we might follow up a question with something like “Do others feel this way” or “Is this important for other people to know about?” We want to hear from everyone, and so at times we may interact to help the conversation keep moving. We will be taking your comments and summarizing them to share with the evaluation team and the programme implementers. You and your individual opinions won’t be identified in any way.
[172]
Location: Village ____________________
Tehsil ____________________ District ____________________
Date: / / Participants List:
Name Age Name Age
1. 2.
3. 4.
5. 6.
7. 8.
9. 10.
Guidelines For Participants: Before start of discussion, we would like to let you about some recommended guidelines or “ground rules” that help establish the group norms so that the group proceeds smoothly and respectfully for all participants:
Only one person talks at a time.
Be sure about the Confidentiality. “What is shared in the room stays in the room.”
It is important for us to hear everyone’s ideas and opinions. There are no right or wrong answers to questions – just ideas, experiences and opinions, which are all valuable.
It is important for us to hear all sides of an issue – both the positive and the negative.
Focus Group Questions: Please listen carefully to the questions and provide your feedback as per the guidelines provided.
Q-1: Over the last year has there been any changes/improvements in
the school water and sanitation facilities? What has
changed/improved and share if you know which organization/s
have helped improve these facilities?
[173]
Q-2: Can you please share your impressions (to assess child friendly
facilities)?
a) Are the facilities sufficient in number (or are there long queues)?
b) Do the facilities have soap, running water, and water/ablution
jug/mug?
c) Is the height of the water stand post accessible for children?
d) Are the latrines properly ventilated and clean at all time?
[174]
e) Can all children (girls, boys as well as disabled children) access
the Water and Sanitation facilities equitably?
f) Is the area around the latrine and water stand post kept clean?
g) Do the latrine facilities provide privacy?
h) Are all facilities still functional and in use of school children? Are
there any which are either non-functional or inaccessible to you
(used by teachers or others)?
Q-3: Can you share if you/children were involved in the process of
identification, design, implementation and sustainable use of
facilities (design, location, number etc.)?
Q-4: In which you/children benefitted from improved facilities – probe in
terms of benefits for boys and girls, children of different age
groups, reduced student sickness and absenteeism, lower
dropouts, reduction in open defecation and others?
[175]
Q-5: Is there a Wash Club in your school? If yes, how & when was it
formed and who are its members? What has it done for highlight
water, sanitation and health/hygiene issues during the
implementation of SPSP, Please elaborate?
Q-6: Have you received health and hygiene messages in last one year,
what were those messages about and which organizations
provided that information? Probe what messages and delivery
methods were found useful and why? Ask children to recall and
share at least 1 message/ learning with regards to water, sanitation
and health/hygiene?
Q-7: How many of you have adopted health and hygiene learning
(messages/practices) in school? If yes how? If not, why? Have you
passed on this information to your families/village, and did you
see any changes in family/neighbours health and hygiene
practices? Probe: how many of the students had washed hands
that day? When and how? Why is essential to wash hands? List
the most essential times when we must wash our hands? Are their
children at school and adults in your community who still continue
to defecate in open? Why?
Thank You for your time and candid response.
[176]
End of Programme Evaluation
UNICEF’s Sanitation Programme at Scale in Pakistan (SPSP) – Rural Through
Pakistan Approach to Total Sanitation (PATS)
FOCUS GROUP DISCUSSION (FGDS) SOCIAL ORGNIZERS (SOs)
(MEN AND WOMEN)
Introduction to the Interviewee: I am ,a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation for UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. We thank you for taking time for this focus group today. We shall be asking you questions about how the programme activities were implemented and what changes this programme brought in your life. During the FDG, please help yourself to tea and water. If you need to get up to use the rest room, feel free. We expect the focus group to take 1 to 1-½. We shall record/document the proceedings and seek your consent to allow us to use this discussion and/or interpretation for our report. Purpose: Before we get started, we would like to give you a little background about focus groups and why we are doing this focus group discussion with you. Focus groups are intended to give us a sense of how a group of people feels about a certain subject. We often want confirmation that others feel this way, so we might follow up a question with something like “Do others feel this way” or “Is this important for other people to know about?” We want to hear from everyone, and so at times we may interact to help the conversation keep moving. We will be taking your comments and summarizing them to share with the evaluation team and the programme implementers. You and your individual opinions won’t be identified in any way.
[177]
Location: Village ____________________
Tehsil ____________________ District ____________________
Date: / / Participants List:
Name Age Name Age
11. 12.
13. 14.
15. 16.
17. 18.
19. 20.
Guidelines For Participants: Before start of discussion, we would like to let you about some recommended guidelines or “ground rules” that help establish the group norms so that the group proceeds smoothly and respectfully for all participants:
Only one person talks at a time.
Be sure about the Confidentiality. “What is shared in the room stays in the room.”
It is important for us to hear everyone’s ideas and opinions. There are no right or wrong answers to questions – just ideas, experiences and opinions, which are all valuable.
It is important for us to hear all sides of an issue – both the positive and the negative.
[178]
Focus Group Questions: Please listen carefully to the questions and provide your feedback as per the guidelines provided.
Q-8: What were priority development needs in the community? Was
Water, Sanitation and Health & Hygiene (sanitation in particular)
amongst the priority needs of the people? If yes, what are the
causes of poor WASH conditions and how/who gets affected the
most by that?
Q-9: What is your understanding of the intended role of SOs (trigger
agents, motivation, social mobilization, demand creation, bridge
between the sanitation mart, VSC and beneficiaries, and others)?
To what extent you feel that SOs have been able to deliver on their
envisaged/expected role?
Q-10: Did the processes undertaken during the PRA exercises (Poverty
and Social Mapping) and Social Mobilization help identify and
target the poorest (lowest quintiles), the vulnerable including the
disabled as a priority? If so, what was done to address their WASH
related needs? How was the equity focus maintained during the
identification, selection and implementation?
[179]
Q-11: To what extent were the mobilization and facilitation strategies
(including topics/issues chosen for community awareness)
relevant to community needs (of different ages/sex)? Did your
training contents, training delivery, duration, IEC materials
(messages, medium, materials), community session plan
(duration, schedule and coverage area), help in raising your
understanding & ability (facilitated) to counsel/educate community
beneficiaries (men, women and others) & How?
Q-12: To what extent was the social mobilization, VSC, CRP formation,
construction of demo latrines, and linkages with supply side
interventions effective? What can be done differently in the future
to make it more effect?
Q-13: To what extent was your message responded to by different
groups? Which groups (of men, women, children and others
including the disabled) were more receptive and why? What
messages & issues people (men, women, children and others
including disabled) were more attentive to and adapted early and
why? Which mediums of communication (talks, posters, electronic
messages – radio, tv, theatre and others) were people (of age and
sex) more receptive of and why?
[180]
Q-14: To what extent you feel you were provided necessary/adequate
support (e.g. training, time, IEC materials, coverage area and
supervision) by the project to deliver on the expected roles? If not
what were the constraints?
Q-15: How has the project changed the perception of improved WASH
services and who has benefitted the most (men, women, children,
or others including the disabled) and in what ways? What changes
have been/can be observed as a result of the various programme
outputs? How were these changes brought about in your villages?
Q-16: To what extent you feel that the project approaches/interventions
were most suited/appropriate and of the quality to address the
varied needs (of different groups),
a) Community mobilization (including the strategy, use of PRAs,
usage and availability of time, skills of SOs/CRPs, status of
VSCs, role of NRSP Trainers)
b) Behavior Change Communication (demand generation)
activities (IEC, Print, Electronic, Community Shows),
[181]
Q-17: What were the key features of Community Action Plans (CAPs)
developed by the VSC? To what extent interventions of different
community groups were included in the plan e.g. Joint activities of
male & female VSC, SLTS and others? To what extent did project
provide adequate and timely resources (community
mobilization/education, technical training, and availability of low
cost toilet supplies) for implementation of this plan?
Q-18: To what extent project interventions were designed and
implemented from the onset to contribute to attainment of gender
(men, women, children, older person, disabled, adolescent and
others) equality/equity (in relation to UNICEF policy)? Was a
gender analysis done? What positive changes have been
supported through this programme (with Government,
Implementing Partners, and Beneficiaries) that have contributed to
gender equality? Were there unintended/negative effects of the
programme implementation (such as increase in work burden)?
Q-19: Did the programme have an equity focus (as per UNICEF
guidelines for instance vulnerability criteria)? To what extent the
equity principles were integrated in the social mobilization, BCC,
supplies facilitation, etc? How did the interventions cater to the
needs of the disabled or women/children?
[182]
Q-20: Did your village achieve the ODF status, is it certified /notified by
Government? What rewards and incentives has your village been
able to draw down from the Government? Are there any issues in
accessing the rewards/incentives? In your opinion has the
rewards/incentives approach been an effective one?
Q-21: In your view which of the project interventions and results may
sustain (ODF status, CRPs, VSC, hand wash practice, construction
of low cost latrines, sanitary marts, entrepreneurs)? Which
interventions may not sustain and why? What should be done
differently in future to achieve better results?
Q-22: What experiences/lessons were learned around programme design
and implementation (list what worked, what did not and why?
Substantiate with measure of success, scale, and lessons learnt
for future)?
Note: The analysis to be desegregated for following (components/subcomponents)
10. DEMAND CREATION
What Worked What Didn’t Work
Lessons Learnt Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
[183]
11. SUPPLY FACILITATION
What Worked What Didn’t Work
Lessons Learnt Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
12. HEALTH AND HYGIENE PROMOTION
What Worked What Didn’t Work
Lessons Learnt Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
13. LINKAGES WITH DUTY BEARERS FOR PATS
What Worked What Didn’t Work
Lessons Learnt Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
14. VILLAGES SUPPORTED IN TOTAL SANITATION STATUS
What Worked What Didn’t Work
Lessons Learnt Success Scale
Rank
Highly
[184]
Successful
Moderate
Poor
Fail
15. PATS KNOWLEDGE MANAGEMENT FRAMEWORK
What Worked What Didn’t Work
Lessons Learnt Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
16. PARTNERSHIP MANAGEMENT
What Worked What Didn’t Work
Lessons Learnt Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
17. INTEGRATION OF CROSS CUTTING PRIORITIES
(Gender & Equity, Unforeseen Events, Rights Based Approach) What Worked What Didn’t
Work Lessons Learnt Success
Scale Rank
Highly
Successful
Moderate
Poor
Fail
[185]
18. SCHOOL LED TOTAL SANITATION
What Worked What Didn’t Work
Lessons Learnt Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
Thank You for your time and candid response.
[186]
End of Programme Evaluation
UNICEF’s Sanitation Programme at Scale in Pakistan (SPSP) – Rural Through
Pakistan Approach to Total Sanitation (PATS)
FOCUS GROUP DISCUSSION (FGDS) COMMUNITY RESOURCE PERSONS (CRPs)
(MEN AND WOMEN)
Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation for UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. We thank you for taking time for this focus group today. We shall be asking you questions about how the programme activities were implemented and what changes this programme brought in your life. During the FDG, please help yourself to tea and water. If you need to get up to use the rest room, feel free. We expect the focus group to take 1 to 1-½. We shall record/document the proceedings and seek your consent to allow us to use this discussion and/or interpretation for our report. Purpose: Before we get started, we would like to give you a little background about focus groups and why we are doing this focus group discussion with you. Focus groups are intended to give us a sense of how a group of people feels about a certain subject. We often want confirmation that others feel this way, so we might follow up a question with something like “Do others feel this way” or “Is this important for other people to know about?” We want to hear from everyone, and so at times we may interact to help the conversation keep moving. We will be taking your comments and summarizing them to share with the evaluation team and the programme implementers. You and your individual opinions won’t be identified in any way.
[187]
Location: Village ____________________
Tehsil ____________________ District ____________________
Date: / / Participants List:
Name Age Name Age
21. 22.
23. 24.
25. 26.
27. 28.
29. 30.
Guidelines for Participants: Before start of discussion, we would like to let you about some recommended guidelines or “ground rules” that help establish the group norms so that the group proceeds smoothly and respectfully for all participants:
Only one person talks at a time.
Be sure about the Confidentiality. “What is shared in the room stays in the room.”
It is important for us to hear everyone’s ideas and opinions. There are no right or wrong answers to questions – just ideas, experiences and opinions, which are all valuable.
It is important for us to hear all sides of an issue – both the positive and the negative.
Focus Group Questions: Please listen carefully to the questions and provide your feedback as per the guidelines provided.
Q-23: Are you familiar with UNICEF SPSP? What has been its objectives
and approach, please share your understanding? To what extent
the programme has been able to achieve desired results?
[188]
Q-24: Is WASH (particularly Sanitation) one of the priority needs in the
rural context (your villages/regions)? If yes, which groups need the
services most and why (probe men, women, children, poor)?
Which groups are at greatest risk for poor WASH (sanitation in
particular) conditions and what are its implications (including
potential) for rural communities?
Q-25: What is your understanding of the intended role of CRPs (trigger
agents, motivation, demand creation, bridge between the
sanitation mart, VSC and beneficiaries, and others)? To what
extent you feel that CRPs have been able to deliver on their
envisaged/expected role? If yes how and if not why?
Q-26: To what extent the topics/issues chosen for community awareness
were relevant to community needs (of different ages/sexes)? Did
your training contents, training delivery, duration, IEC materials
(messages, medium, materials), community session plan
(duration, schedule and coverage area), help in raising your
understanding & ability (facilitated) to counsel/educate community
(men, women and others) & How?
[189]
Q-27: Which mediums of communication (talks, posters, electronic
messages – radio, tv, theatre and others) were people (of age and
sex) more receptive of and why? How could this be made more
effective?
Q-28: Please share your impressions of the most visible changes that
this programme overall and CRP lead interventions/BCC have
been able to achieve? Which groups (e.g. men, women, children,
older people) have been relatively more receptive to BCC/H&H
awareness (attentive, demonstrated attitudinal and behavioural
change) and why others been not so receptive?
Q-29: To what extent you feel you were provided necessary/adequate
and timely support e.g. training, financial compensation, time, IEC
materials, coverage area, supervision) to enable you to deliver on
the expected roles? If yes, please elaborate and if not why/how?
Q-30: Which of the project achievements e.g. community groups, ODF,
changed H&H behaviours, CRPs and others, may sustain after
completion of the programme and why? Why others may not
sustain? What needs to be done to create self-reinforcement and
sustaining the programme achievements within communities
please elaborate?
[190]
Thank You for your time and candid response.
End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in
Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)
FOCUS GROUP DISCUSSION (FGDS)
[191]
Marts Owners/Entrepreneurs
Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation for UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. We thank you for taking time for this focus group today. We shall be asking you questions about how the programme activities were implemented and what changes this programme brought in your life. During the FDG, please help yourself to tea and water. If you need to get up to use the rest room, feel free. We expect the focus group to take 1 to 1-½. We shall record/document the proceedings and seek your consent to allow us to use this discussion and/or interpretation for our report. Purpose: Before we get started, we would like to give you a little background about focus groups and why we are doing this focus group discussion with you. Focus groups are intended to give us a sense of how a group of people feels about a certain subject. We often want confirmation that others feel this way, so we might follow up a question with something like “Do others feel this way” or “Is this important for other people to know about?” We want to hear from everyone, and so at times we may interact to help the conversation keep moving. We will be taking your comments and summarizing them to share with the evaluation team and the programme implementers. You and your individual opinions won’t be identified in any way. Location: Village ____________________
Tehsil ____________________ District ____________________
Date: / / Participants List:
[192]
Name Age Name Age
31. 32.
33. 34.
35. 36.
37. 38.
39. 40.
Guidelines for Participants: Before start of discussion, we would like to let you about some recommended guidelines or “ground rules” that help establish the group norms so that the group proceeds smoothly and respectfully for all participants:
Only one person talks at a time.
Be sure about the Confidentiality. “What is shared in the room stays in the room.”
It is important for us to hear everyone’s ideas and opinions. There are no right or wrong answers to questions – just ideas, experiences and opinions, which are all valuable.
It is important for us to hear all sides of an issue – both the positive and the negative.
Focus Group Questions: Please listen carefully to the questions and provide your feedback as per the guidelines provided.
Q-31: Are you familiar with UNICEF SPSP, if yes, what do you know
about the programme? To what extent the programme has been
able to achieve desired results?
[193]
Q-32: How have you been approached/engaged (in supplies facilitation)
in the SPSP? To what extent has this project increased
community demand for sanitation supplies? Probe if number of
latrine materials sold has increased, ask for monthly sale of latrine
materials?
Q-33: In your view how has the introduction of entrepreneurs/marts
benefitted the communities (for mart owners probe further on
people, price, promotion, product and placement)? To what extent
has it contributed to facilitating community access to latrine
material supplies (lower costs, less transportation, year round
availability, quality controls and others) Probe further and ask for
examples?
Q-34: To what extent the project allocated resources (technical, financial
and time) for supplies facilitation were adequate e.g. number of
marts, entrepreneurs, products diversity, timing of establishing
facilitation chain, promotion, and others? In case of any disruption
in supply chain, was project able to identify and resolve the supply
chain disruption in time?
[194]
Q-35: What changes have happened in the latrine/ sanitation supplies
market because of this project (prices, products diversity,
competition, quality)? What could have been done to achieve
better results?
Q-36: Do you plan to continue after project completion? Do you foresee
the demand for latrines would continue to increase or otherwise
after completion of the project? What could this project do (or
future phase) for continued demand creation and provision of
materials?
Q-37: What do you feel were some of the factors that either facilitated or
hindered your work? What can be done in to enable easy access to
products/services for construction of household latrines?
[196]
End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in
Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)
FOCUS GROUP DISCUSSION (FGDS)
MASONs
Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation for UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. We thank you for taking time for this focus group today. We shall be asking you questions about how the programme activities were implemented and what changes this programme brought in your life. During the FDG, please help yourself to tea and water. If you need to get up to use the rest room, feel free. We expect the focus group to take 1 to 1-½. We shall record/document the proceedings and seek your consent to allow us to use this discussion and/or interpretation for our report. Purpose: Before we get started, we would like to give you a little background about focus groups and why we are doing this focus group discussion with you. Focus groups are intended to give us a sense of how a group of people feels about a certain subject. We often want confirmation that others feel this way, so we might follow up a question with something like “Do others feel this way” or “Is this important for other people to know about?” We want to hear from everyone, and so at times we may interact to help the conversation keep moving. We will be taking your comments and summarizing them to share with the evaluation team and the programme implementers. You and your individual opinions won’t be identified in any way.
[197]
Location: Village ____________________
Tehsil ____________________ District ____________________
Date: / / Participants List:
Name Age Name Age
41. 42.
43. 44.
45. 46.
47. 48.
49. 50.
Guidelines for Participants: Before start of discussion, we would like to let you about some recommended guidelines or “ground rules” that help establish the group norms so that the group proceeds smoothly and respectfully for all participants:
Only one person talks at a time.
Be sure about the Confidentiality. “What is shared in the room stays in the room.”
It is important for us to hear everyone’s ideas and opinions. There are no right or wrong answers to questions – just ideas, experiences and opinions, which are all valuable.
It is important for us to hear all sides of an issue – both the positive and the negative.
[198]
Focus Group Questions: Please listen carefully to the questions and provide your feedback as per the guidelines provided.
Q-38: Are you familiar with UNICEF SPSP, if yes, what do you know
about the programme? To what extent the programme has been
able to achieve desired results?
Q-39: How have you been approached/engaged (in supplies facilitation)
in the SPSP? To what extent has this project increased
community demand, supplies facilitation including creation of
skilled labour force? Probe if number of latrine constructed every
month increased in the last one year?
Q-40: To what extent you find masons’ training useful/effective/required (
in terms of introducing new knowledge & skills, timing and
duration, contents and methodology, skills of trainer) in
addressing design/implementation issues to construct safer
latrines? In your view what could be improved in future e.g.
additional contents, different methods, and time/duration?
[199]
Q-41: To what extent the programme interventions (especially design of
latrines, masons training, and marts/entrepreneurs) have helped
making the availability of hardware and skills for safer, quicker,
contextually/environmentally relevant, low cost latrines? Probe
how?
Q-42: To what extent has this project been able to introduce new/low
cost/context specific and long lasting (including environment
friendly) technologies and designs? Which designs and products
are more in demand and why? Are these latrines durable (life
span), what are the tentative maintenance/utility costs and can
community afford that?
Q-43: To what extent the project allocated resources (technical, financial
and time) for supplies facilitation were adequate e.g. number of
marts, entrepreneurs, products diversity, timing of establishing
facilitation chain, promotion, and others? In case of any disruption
in supply chain, was project able to identify and resolve the supply
chain disruption in time?
[200]
Q-44: Who (groups and individuals) has benefitted most because of
latrines construction and how? Are you familiar with project
financed demo latrines and subsidies, if yes, to what extent those
been given to the most deserving? Did it contribute to replication
(at least design and technologies), if not why?
Q-45: To what extent do you feel that this increased demand, improved
supplies facilitation (hard ware and skills) would continue after
closure of the programme? If yes how/why and if not why?
Thank You for your time and candid response.
[201]
End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in
Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)
FOCUS GROUP DISCUSSION (FGDS)
Teachers (Male and Female) School Management Committees
Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation for UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. We thank you for taking time for this focus group today. We shall be asking you questions about how the programme activities were implemented and what changes this programme brought in your life. During the FDG, please help yourself to tea and water. If you need to get up to use the rest room, feel free. We expect the focus group to take 1 to 1-½. We shall record/document the proceedings and seek your consent to allow us to use this discussion and/or interpretation for our report. Purpose: Before we get started, we would like to give you a little background about focus groups and why we are doing this focus group discussion with you. Focus groups are intended to give us a sense of how a group of people feels about a certain subject. We often want confirmation that others feel this way, so we might follow up a question with something like “Do others feel this way” or “Is this important for other people to know about?” We want to hear from everyone, and so at times we may interact to help the conversation keep moving. We will be taking your comments and summarizing them to share with the evaluation team and the programme implementers. You and your individual opinions won’t be identified in any way.
[202]
Location: Village ____________________
Tehsil ____________________ District ____________________
Date: / / Participants List:
Name Age Name Age
51. 52.
53. 54.
55. 56.
57. 58.
59. 60.
Guidelines for Participants: Before start of discussion, we would like to let you about some recommended guidelines or “ground rules” that help establish the group norms so that the group proceeds smoothly and respectfully for all participants:
Only one person talks at a time.
Be sure about the Confidentiality. “What is shared in the room stays in the room.”
It is important for us to hear everyone’s ideas and opinions. There are no right or wrong answers to questions – just ideas, experiences and opinions, which are all valuable.
It is important for us to hear all sides of an issue – both the positive and the negative.
[203]
Focus Group Questions: Please listen carefully to the questions and provide your feedback as per the guidelines provided.
Q-46: Over the last year has there been any changes/improvements in
the school water and sanitation facilities? What has
changed/improved and share if you know which organization/s
have helped improve these facilities?
Q-47: Can you please share your impressions (to assess child friendly
facilities)?
i) Are the facilities sufficient in number (or are there long queues)?
j) Do the facilities have soap, running water, and water/ablution
jug/mug?
k) Is the height of the water stand post accessible for children?
l) Are the latrines properly ventilated and clean at all time?
[204]
m) Can all children (girls, boys as well as disabled children) access
the Water and Sanitation facilities equitably?
n) Is the area around the latrine and water stand post kept clean?
o) Do the latrine facilities provide privacy?
p) Are all facilities still functional and in use of school children? Are
there any which are either non-functional or inaccessible to
you/children (used by teachers or others)?
Q-48: In which ways and extent you/children benefitted from improved
facilities – probe in terms of benefits for boys and girls, children of
different age groups, reduced student sickness and absentism,
lower dropouts, reduction in open defecation and others?
Q-49: Can you share if you/children were involved in the process of
identification, design, implementation and sustainable use of
facilities (design, location, number etc.)?
[205]
Q-50: Is there a SMC/Wash Club in your school? If yes, how & when was
it formed and who are its members (both SMC/WC)? What has it
done for highlight/address the water, sanitation and health/hygiene
issues in last one year, Please elaborate?
Q-51: Have you/children received health and hygiene messages in last
one year, what were those messages about, which organization/s
provided that information and by whom (teachers, CRP, SOs
others)? Probe what messages and delivery methods were found
useful and why?
Q-52: In your view has the approach of engaging schools (teachers,
children) played a role in improving knowledge and helping
change practices around of water, sanitation and health/hygiene
both at school and community levels?
Q-53: How schools (children/teachers/parents) could be involved more
effectively to improve school WASH and catalysing change in the
larger community (around WASH)?
[206]
Thank You for your time and candid response.
End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in
Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)
FOCUS GROUP DISCUSSION (FGDS)
VILLAGE SANITATION COMMITTEE (VSC) (MEN AND WOMEN GROUPS)
Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation for UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. We thank you for taking time for this focus group today. We shall be asking you questions about how the programme activities were implemented and what changes this programme brought in your life. During the FDG, please help yourself to tea and water. If you need to get up to use the rest room, feel free. We expect the focus group to take 1 to 1-½. We shall record/document the proceedings and seek your consent to allow us to use this discussion and/or interpretation for our report. Purpose: Before we get started, we would like to give you a little background about focus groups and why we are doing this focus group discussion with you. Focus groups are intended to give us a sense of how a group of people feels about a certain subject. We often want confirmation that others feel this way, so we might follow up a question with something like “Do others feel this way” or “Is this important for other people to know about?” We want to hear from everyone, and so at times we may interact to help the conversation keep moving. We will be taking your comments and summarizing them to share with the evaluation team and the programme implementers. You and your individual opinions won’t be identified in any way.
[207]
Location: Village ____________________
Tehsil ____________________ District ____________________
Date: / / Participants List:
Name Age Name Age
61. 62.
63. 64.
65. 66.
67. 68.
69. 70.
Guidelines For Participants: Before start of discussion, we would like to let you about some recommended guidelines or “ground rules” that help establish the group norms so that the group proceeds smoothly and respectfully for all participants:
Only one person talks at a time.
Be sure about the Confidentiality. “What is shared in the room stays in the room.”
It is important for us to hear everyone’s ideas and opinions. There are no right or wrong answers to questions – just ideas, experiences and opinions, which are all valuable.
It is important for us to hear all sides of an issue – both the positive and the negative.
[208]
Focus Group Questions: Please listen carefully to the questions and provide your feedback as per the guidelines provided.
Q-54: What are priority development needs in your community? Probe if
Water, Sanitation and Health & Hygiene (sanitation in particular) is
one amongst the priority needs? If yes, what are the causes of
poor WASH conditions and how/who gets affected the most by
that?
Q-55: When was the VSC formed and how was it formed? Probe if VSC
process of participatory and all inclusive (all key groups are
represented)? What were the key functions of VSC and what have
you achieved with respect to the anticipated roles (assessment,
identification of vulnerable households, village development
planning, management of interventions, rewards management?
Please elaborate? To what extent VSC was involved in design of
interventions and implementation?
Q-56: Which organizations/individuals have worked on WASH with you
over the last year? To what extend did you find the interventions
and implementation approach relevant to your needs/context
(mobilization, demo latrines construction, supply side facilitation,
BCC etc.) and been able to successfully address the issues?
Probe if they feel there were any gaps, constraints and
weaknesses in the design and implementation?
[209]
Q-57: To what extent did organization/project provide adequate and
timely resources (community mobilization/education, technical
training, availability of low cost toilet supplies) for implementation
of this plan? Were the most deserving received the demo latrines
and subsidies (assess if VSC are satisfied with targeting)?
Q-58: What has changed in the village because of the project/VSC
efforts? Probe more with respect to village achieving ODF,
changes in the village environment, changes in people’s
perception and practices, impact on health and reduced diseases,
work load for men/women)? Which groups i.e. men, women,
children, or others including the disabled benefitted and in what
ways? Probe if there were no project what may have been the loss
or may have not changed?
Q-59: Did your village achieve the ODF status, is it certified /notified by
Government? If yes, did you receive rewards, what are those &
how were those identified? Probe if women and children were part
of rewards consultations and if any group is excluded from the
benefits of rewards?
Q-60: In your view which of the project interventions and results may
sustain (ODF status, CRPs, VSC, hand wash practice, construction
[210]
of low cost latrines, sanitary marts, entrepreneurs)? Which
interventions may not sustain and why? What should be done
differently in future to achieve better results?
Thank You for your time and candid response.
End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in
Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)
INDEPTH INTERVIEW
SANITATION MARKET Consultant Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation of UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. I am here to get your reflections, experiences and recommendations about the programme, which will help us to assess, analyse and document programme progress, results and areas for further improvement. This will help at large in designing or developing future sanitation programmes in the country and/or extension or scale-up of this initiative. I thank you for taking time for this interview today. I shall be asking you questions about how the programme activities were implemented in your area and what changes this programme brought in the lives of the community in regard to improved access and availability of sanitation services. I anticipate the interview will last about 60 minutes, and appreciate any information you can provide. Please note that this interview is important for the success of this assessment. Your answers are completely confidential and shall remain with the evaluation team. At the start we seek your consent to make use of this information for analysis and reporting and may also cite it in the report while making reference to your name, if needed. Also, we are recording the proceeding to avoid any loss of information and seek your consent to record our discussion. Finally, we would request you to please allow us to make photographs of project interventions and yourself for evidence and communication purposes. Note: This is to underline that the IDI instrument has been designed as such to offer flexibility to the interviewers to adapt questions as to the specific role of the organization, respondents profile, and understanding of programme operations and overall knowledge of sector. Hence one consolidated tool has been developed for multiple stakeholders (as listed above).
[211]
Name of Respondent: _____________________ Designation _____________________ Organisation _____________________ Location: _____________________ Date of Interview: _____________________
Name of the Interviewer: _____________________
Interview Questions: Please listen carefully to the questions and provide your candid feedback.
Q 59. Were there any quality standards for outputs & to what extent the
outputs have been delivered in line with planned quality
standards?
Q 60. What were the key quality aspects/elements of supply facilitation
interventions? Were there any issues encountered related to
specific quality standards?
1. Latrine designs
2. Quality of demo latrines
3. Mason training
[212]
4. Entrepreneurs/Marts
5. Rewards design/Implementation (for model villages)
6. Any Other Aspect
Q 61. Did the programme document clearly define the equity focus (as
per UNICEF guidelines for instance vulnerability criteria)? How
was it implemented and how were the systems/processes evolved
for equity integration into the project deliverables (how did the
interventions cater to the needs of the disabled or
women/children?
[213]
Q 62. To what extent the equity principles were integrated in the
approaches and deliverables i.e. BCC, supplies facilitation, e.g.
targeting of poorest of poor for demo latrines? Or bringing about a
change in access to WASH services and facilities for the lowest
quintile?
Q 63. What was done to ensure that the school based interventions in
the target area provide child friendly water and sanitation
facilities?
Thank You for your time and candid response.
[214]
End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in Pakistan (SPSP) – Rural through Pakistan
Approach to Total Sanitation (PATS) INDEPTH INTERVIEW
APEX THIRD PARTY MONITORS
Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation of UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. I am here to get your reflections, experiences and recommendations about the programme, which will help us to assess, analyse and document programme progress, results and areas for further improvement. This will help at large in designing or developing future sanitation programmes in the country and/or extension or scale-up of this initiative. I thank you for taking time for this interview today. I shall be asking you questions about how the programme activities were implemented in your area and what changes this programme brought in the lives of the community in regard to improved access and availability of sanitation services. I anticipate the interview will last about 60 minutes, and appreciate any information you can provide. Please note that this interview is important for the success of this assessment. Your answers are completely confidential and shall remain with the evaluation team. At the start we seek your consent to make use of this information for analysis and reporting and may also cite it in the report while making reference to your name, if needed. Also, we are recording the proceeding to avoid any loss of information and seek your consent to record our discussion. Finally, we would request you to please allow us to make photographs of project interventions and yourself for evidence and communication purposes. Note: This is to underline that the IDI instrument has been designed as such to offer flexibility to the interviewers to adapt questions as to the specific role of the organization, respondents profile, and understanding of programme operations and overall knowledge of sector. Hence one consolidated tool has been developed for multiple stakeholders (as listed above).
Name of Respondent: _____________________ Designation _____________________ Organisation _____________________ Location: _____________________ Date of Interview: _____________________
Name of the Interviewer: _____________________
[215]
Interview Questions:
Please listen carefully to the questions and provide your candid feedback.
1. Were you involved in initial orientation/training session introducing the
design and results framework of SPSP Rural? How and who was
involved from APEX? Was the introduction to the programme sufficient?
2. How were the TPM criteria developed and agreed? Was APEX involved
in developing the monitoring tools, procedures and checklists? Were
there any standard monitoring targets set for various programme
components including soft and hard components?
3. What were the most common and frequent observations and how/to
whom were these communicated (UNICEF HQ, Provincial Offices)?
How frequent was the reporting? How were these issues taken forward
and resolved? How long did it take from identifying the issue to it being
resolved?
4. What challenges were encountered during the TPM activities? Were
they resolved and how? What lessons/recommendations need to be
factored into future interventions related to TPM?
5. How was the overall ongoing coordination with UNICEF Isb, Provincial
Offices as well as the range of Implementing Partners?
6. What internal checks and processes were put in place in APEX to
ensure the quality of TPM activities related to SPSP? Were Monitors
given standardized trainings and understanding of how to apply TPM
criteria?
7. How was RusFAD learning incorporated in SPSP as part of the
Knowledge management process? What learnings have been
generated during the SPSP implementation?
8. How do you view the actual delivery by IPs (in terms of no’s and targets
i.e. Villages ODF etc)? Was quality a consideration and to what extent
these elements are likely to sustain?
[216]
9. Did APEX have a monitoring role in the Action Research undertaken by
DSI in SPSP?
10. What is your perception of training provided by RSPN and their
mentoring role? Were there any issues or challenges observed?
11. Was it in the ambit of APEX to look at partnership management
and risk management (particularly financial)? Was cost efficiency and
value for money aspects part of TPM?
12. How was Gender, Disability and Equity factored into the project
and picked up in monitoring activities? What are the observations in
terms of
13. How and to what extent has TPM affected/influenced decision
making in SPSP? What key factors improved as a result?
[217]
End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in
Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)
INDEPTH INTERVIEW
THIRD PARTY MONITOR
Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation of UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. I am here to get your reflections, experiences and recommendations about the programme, which will help us to assess, analyse and document programme progress, results and areas for further improvement. This will help at large in designing or developing future sanitation programmes in the country and/or extension or scale-up of this initiative. I thank you for taking time for this interview today. I shall be asking you questions about how the programme activities were implemented in your area and what changes this programme brought in the lives of the community in regard to improved access and availability of sanitation services. I anticipate the interview will last about 60 minutes, and appreciate any information you can provide. Please note that this interview is important for the success of this assessment. Your answers are completely confidential and shall remain with the evaluation team. At the start we seek your consent to make use of this information for analysis and reporting and may also cite it in the report while making reference to your name, if needed. Also, we are recording the proceeding to avoid any loss of information and seek your consent to record our discussion. Finally, we would request you to please allow us to make photographs of project interventions and yourself for evidence and communication purposes. Note: This is to underline that the IDI instrument has been designed as such to offer flexibility to the interviewers to adapt questions as to the specific role of the organization, respondents profile, and understanding of programme operations and overall knowledge of sector. Hence one consolidated tool has been developed for multiple stakeholders (as listed above).
Name of Respondent: _____________________ Designation _____________________ Organisation _____________________ Location: _____________________ Date of Interview: _____________________
Name of the Interviewer: _____________________
[218]
Interview Questions: Please listen carefully to the questions and provide your candid feedback.
Q 64. What experiences/lessons were identified as part of the monitoring
process around programme implementation (list what worked,
what did not and why? Substantiate with measure of success,
scale, and lessons identified for future)? The analysis to be
desegregated for following (components/subcomponents)
19. DEMAND CREATION
What Worked What Didn’t Work
Lessons Identified Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
20. SUPPLY FACILITATION
What Worked What Didn’t Work
Lessons Identified Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
21. HEALTH AND HYGIENE PROMOTION
What Worked What Didn’t Work
Lessons Identified Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
22. LINKAGES WITH DUTY BEARERS FOR PATS
[219]
What Worked What Didn’t Work
Lessons Identified Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
23. VILLAGES SUPPORTED IN TOTAL SANITATION STATUS
What Worked What Didn’t Work
Lessons Identified Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
24. PATS KNOWLEDGE MANAGEMENT FRAMEWORK
What Worked What Didn’t Work
Lessons Identified Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
25. PARTNERSHIP MANAGEMENT
What Worked What Didn’t Work
Lessons Identified Success Scale
Rank
Highly
Successful
Moderate
[220]
Poor
Fail
26. INTEGRATION OF CROSS CUTTING PRIORITIES
(Gender & Equity, Unforeseen Events, Rights Based Approach) What Worked What Didn’t
Work Lessons Identified Success
Scale Rank
Highly
Successful
Moderate
Poor
Fail
27. SCHOOL LED TOTAL SANITATION
What Worked What Didn’t Work
Lessons Identified Success Scale
Rank
Highly
Successful
Moderate
Poor
Fail
Q 65. What tools/guidelines were designed/implemented as part of the
3rd Party Monitoring process? How successful was the actual
application of these tools and methods in contributing to ensuring
quality of outputs and processes? What needs to be done
differently in terms of monitoring to improve the implementation of
the programme?
[221]
Q 66. In terms of knowledge management, did the 3rd party monitoring
process inform and contribute to improved project implementation
and decisions? Please elaborate with examples.
Q 67. How did UNICEF and the Implementing Partners respond to the 3rd
party monitoring process? Were any issues/challenges faced in
the monitoring process?
[222]
Thank You for your time and candid response.
End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in
Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)
[223]
INDEPTH INTERVIEW BCC CONSULTANT
Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation of UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. I am here to get your reflections, experiences and recommendations about the programme, which will help us to assess, analyse and document programme progress, results and areas for further improvement. This will help at large in designing or developing future sanitation programmes in the country and/or extension or scale-up of this initiative. I thank you for taking time for this interview today. I shall be asking you questions about how the programme activities were implemented in your area and what changes this programme brought in the lives of the community in regard to improved access and availability of sanitation services. I anticipate the interview will last about 60 minutes, and appreciate any information you can provide. Please note that this interview is important for the success of this assessment. Your answers are completely confidential and shall remain with the evaluation team. At the start we seek your consent to make use of this information for analysis and reporting and may also cite it in the report while making reference to your name, if needed. Also, we are recording the proceeding to avoid any loss of information and seek your consent to record our discussion. Finally, we would request you to please allow us to make photographs of project interventions and yourself for evidence and communication purposes. Note: This is to underline that the IDI instrument has been designed as such to offer flexibility to the interviewers to adapt questions as to the specific role of the organization, respondents profile, and understanding of programme operations and overall knowledge of sector. Hence one consolidated tool has been developed for multiple stakeholders (as listed above).
Name of Respondent: _____________________ Designation _____________________ Organisation _____________________ Location: _____________________ Date of Interview: _____________________
Name of the Interviewer: _____________________
Interview Questions: Please listen carefully to the questions and provide your candid feedback.
Q 68. Were any quality standards prescribed and set for Behavior
Change Communication (in line with UNICEF core strategy –
advocacy and communication for change) of communities for
improved health & safer hygiene practices/demand generation
[224]
interventions before implementation? To what extent the BCC
outputs were delivered in line with the prescribed quality
standards?
Q 69. What BCC strategies/instruments/messages/mediums were
evolved & to what extent were those effective? Which methods
received maximum attention, which groups of people
changed/responded, and which practices changed more? Which
BCC interventions worked well & why and why others failed?
[225]
Q 70. Did the programme design clearly define the equity focus (as per
UNICEF guidelines for instance vulnerability criteria)? How was it
incorporated into the BCC interventions and implemented (how did
the interventions cater to the needs of the disabled or
women/children?
Thank You for your time and candid response.
[226]
End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in
Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)
INDEPTH INTERVIEW
DONORS/INGOs
Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation of UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. I am here to get your reflections, experiences and recommendations about the programme, which will help us to assess, analyse and document programme progress, results and areas for further improvement. This will help at large in designing or developing future sanitation programmes in the country and/or extension or scale-up of this initiative. I thank you for taking time for this interview today. I shall be asking you questions about how the programme activities were implemented in your area and what changes this programme brought in the lives of the community in regard to improved access and availability of sanitation services. I anticipate the interview will last about 60 minutes, and appreciate any information you can provide. Please note that this interview is important for the success of this assessment. Your answers are completely confidential and shall remain with the evaluation team. At the start we seek your consent to make use of this information for analysis and reporting and may also cite it in the report while making reference to your name, if needed. Also, we are recording the proceeding to avoid any loss of information and seek your consent to record our discussion. Finally, we would request you to please allow us to make photographs of project interventions and yourself for evidence and communication purposes. Note: This is to underline that the IDI instrument has been designed as such to offer flexibility to the interviewers to adapt questions as to the specific role of the organization, respondents profile, and understanding of programme operations and overall knowledge of sector. Hence one consolidated tool has been developed for multiple stakeholders (as listed above).
Name of Respondent: _____________________ Designation _____________________ Organisation _____________________ Location: _____________________ Date of Interview: _____________________
Name of the Interviewer: _____________________
[227]
Interview Questions: Please listen carefully to the questions and provide your candid feedback.
Q 71. What is your organization doing in the context of WASH Sector? Is
the focus rural or urban? Is PATS approach being utilized? Are
there any differing factors/context in your approach as compared
to the PATS approach?
Q 72. Are there any consultative processes/mechanisms (at varied levels
across different stages) that are/were used to take stakeholders on
board for programme design and implementation? Were you
consulted/involved in the UNICEF SPSP-Rural Programme in any
way? If so, how?
Q 73. Are you aware of any other mechanisms (coordination
meetings/forums etc) that relevant government departments (at
national and provincial levels) are using to align the efforts of
government and development partners in WASH and PATS?
[228]
Q 74. To what extent are they successful in their efforts? In which ways
it could be improved further and what could the development
partners contribute towards this? What in your view are key
learning around consultative planning and what needs to be done
in future to make it more consultative/inclusive, both at the design
as well as during the implementation stages (at varied levels)?
Q 75. Does your approach involve implementation through Government,
Implementing partners/Private Sector? What has been more
successful and which approach do you see being mose
sustainable in addressing WASH in particular Sanitation needs?
[229]
Q 76. What has been the impact of the 18th Amendment in terms of
WASH priorities at the provincial and district levels?
Q 77. How do your programme costs incurred for different WASH
components to achieve results (in terms of mobilization,
beneficiaries awareness, construction of latrines etc on per
head/per facility cost or per beneficiary cost basis etc.) compare
with a similar nature/scale of programme as UNICEF’s SPSP-
Rural? And/or govt and/or in regional countries (CATS, CLTS,
PATS)?
[230]
Q 78. What approaches can be developed going into the future to
address WASH and Sanitation in particular that are aligned to
PATS? What needs to be addressed at Policy level to improve the
approach?
Thank You for your time and candid response.
[231]
End of Programme Evaluation UNICEF’s Sanitation Programme at Scale in
Pakistan (SPSP) – Rural Through Pakistan Approach to Total Sanitation (PATS)
INDEPTH INTERVIEW
SANITATION MARKET CONSULTANT SANITATION ENGINEER
Introduction to the Interviewee: I am , a member of the evaluation team deployed by an independent consulting entity i.e. the AAN Associates, who has been contracted for the End of Programme Evaluation of UNICEF’s Sanitation Programme at Scale in Pakistan. This programme aims to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy to ensure that the most vulnerable, marginalized children and women are using safe drinking water, improved sanitation and hygiene practices. The target areas include flood affected, high Polio risk and insecurity affected areas. I am here to get your reflections, experiences and recommendations about the programme, which will help us to assess, analyse and document programme progress, results and areas for further improvement. This will help at large in designing or developing future sanitation programmes in the country and/or extension or scale-up of this initiative. I thank you for taking time for this interview today. I shall be asking you questions about how the programme activities were implemented in your area and what changes this programme brought in the lives of the community in regard to improved access and availability of sanitation services. I anticipate the interview will last about 60 minutes, and appreciate any information you can provide. Please note that this interview is important for the success of this assessment. Your answers are completely confidential and shall remain with the evaluation team. At the start we seek your consent to make use of this information for analysis and reporting and may also cite it in the report while making reference to your name, if needed. Also, we are recording the proceeding to avoid any loss of information and seek your consent to record our discussion. Finally, we would request you to please allow us to make photographs of project interventions and yourself for evidence and communication purposes. Note: This is to underline that the IDI instrument has been designed as such to offer flexibility to the interviewers to adapt questions as to the specific role of the organization, respondents profile, and understanding of programme operations and overall knowledge of sector. Hence one consolidated tool has been developed for multiple stakeholders (as listed above).
Name of Respondent: _____________________ Designation _____________________ Organisation _____________________ Location: _____________________ Date of Interview: _____________________
[232]
Name of the Interviewer: _____________________
Interview Questions: Please listen carefully to the questions and provide your candid feedback.
Q 79. Were there any quality standards for outputs & to what extent the
outputs have been delivered in line with planned quality
standards?
Q 80. What were the key quality aspects/elements of supply facilitation
interventions? Were there any issues encountered related to
specific quality standards?
7. Latrine designs
8. Quality of demo latrines
9. Mason training
[233]
10. Entrepreneurs/Marts
11. Rewards design/Implementation (for model villages)
12. Any Other Aspect
Q 81. Did the programme document clearly define the equity focus (as
per UNICEF guidelines for instance vulnerability criteria)? How
was it implemented and how were the systems/processes evolved
for equity integration into the project deliverables (how did the
interventions cater to the needs of the disabled or
women/children?
[234]
Q 82. To what extent the equity principles were integrated in the
approaches and deliverables i.e. BCC, supplies facilitation, e.g.
targeting of poorest of poor for demo latrines? Or bringing about a
change in access to WASH services and facilities for the lowest
quintile?
Q 83. What was done to ensure that the school based interventions in
the target area provide child friendly water and sanitation
facilities?
Thank You for your time and candid response.
[235]
Questions for RSPN 1. To what extent was the PCA clear about the role and expectations
of RSPN as an umbrella training and mentoring organization?
Were there any changes in the expected role during the
implementation of the programme?
2. How was the range of training designed? Who was involved in the
process (UNICEF, RSPN etc?) Was the training derived from the
Theory of Change? If so, which model (PATS, CATS etc) was
used as the basis?
3. What was the training model (TOT, or Number of Trainers)? Were
training activities carried out in standard format across Provinces
and IPs?
4. What was the requirement for trainings in terms of trainees
capacities, as well as numbers of participants per training?
5. Was DSI’s report on Low-Cost Sanitation option used in field?
6. Was the training conducted by diverse teams or same set of
trainers provided the whole range of training (mason, CLTS, SLTS,
SO, CRPS, mart/entrepreneurs)? If no, please elaborate?
7. Were the agreed number of training and targets delivered
according to the PCA and Results Framework? What constrains
were faced during the training activities? Did these constrains
affect the quality of the training? Were they discussed with
UNICEF and how were they addressed?
8. How was it ensured that the trainings were effective in their
purpose? Were there any follow up/mentoring activities planned?
What was the result of these in terms of improvement?
9. What is the sustainability of the capacity building that has resulted
from the range of training, in terms of IP Staff and community?
10. What orientation/trainings/capacity building activities were
undertaken with government? At what levels? What was the key
component of the training? What was their effectiveness and how
was it measured?
[236]
23. Annex 12 KNOWLEDGE, ATTITUDES AND PRACTICE (KAP) HOUSEHOLD SURVEY
QUESTIONNAIRE UNICEF’S SANITATION PROGRAMME AT SCALE IN PAKISTAN
EVALUATION FEB 2014 INFORMED CONSENT
I am ____________________, a representative of an independent consulting entity i.e. the AAN Associates, who has been contracted for the Evaluation of UNICEF’s Sanitation Programme at Scale (SPSP) in Pakistan using the Pakistan Approach to Total Sanitation (PATS). This programme aimed to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy.
We are conducting a household survey and would appreciate your participation. I would like
to ask about health, hygiene and sanitation related aspects of the community and your
household. This information will help Unicef and partners to assess achievements under this
project in health, hygiene and sanitation. The survey takes around 30-45 minutes to
complete.
Whatever information you provide, will be kept strictly confidential and will be dealt with
anonymity.
Participation in this survey is voluntary and you can choose not to answer any individual
questions. However, we hope that you will participate in this survey since your views are
important. At this time, do you want to ask me anything about the survey?
RESPONDENT AGREES TO BE INTERVIEWED
RESPONDENT DOES NOT AGREE TO BE INTERVIEWED END
Edited/Verified by: Approved By:
District Supervisor Provincial Coordinator
Signature: Signature:
Date: Date:
[237]
INSTRUCTIONS
RESPONDENT: IS TO BE SLECTED WITH THE FOLLOWING CRITERIA: o SHOULD BE WITH A FAMILY; AND o SHOULD BE OF 18-45 YEARS IN AGE FOR BOTH
MALE AND FEMALE. o IF YOU DON’T FIND A RESPONDENT MEETING
THIS CRITERIA MOVE TO THE NEXT HOUSE
AND SO ON UNTIL YOU COME ACROSS A
HOUSE WITH THE REQUIRED RESPONDENT
CRITERIA
HOUSEHOLD: IS DEFINED AS A GROUP OF FAMILY MEMBERS
CONTRIBUTING TO A SINGLE KITCHEN.
MULTIPLE CHOICE QUESTIONS: ARE NEEDED TO BE PROBED FURTHER WITH ALL
CHOICES MENTIONED. THE QUESTIONS ARE ALSO
MARKED WITH A CAPTION “RECORD ALL
MENTIONED”.
RESPONSE: FOR EACH QUESTION CIRCLE THE
APPROPRIATE RESPONSE/S.
THIS SECTION NEEDS TO BE COMPLETED PRIOR TO STARTUP OF THE FIELD WORK
RECORD NUMBER: (001-TEAM XX) INTERVIEW DATE: ________/________/________ (DATE / MONTH/ YEAR) INTERVIEWER’S NAME: _________________________ SUPERVISOR’S NAME: _________________________ PROVINCE: _________________________ DISTRICT NAME: _________________________ TEHSIL NAME: _________________________ UNION COUNCIL NAME: _________________________ VILLAGE NAME: _________________________
[238]
ALL QUESTIONS ARE TO BE ADDRESSED TO THE RESPONDENT
A) RESPONDENT DETAILS
RESPONDENT NAME:
SEX OF RESPONDENT: MALE 1 FEMALE
2 AGE OF RESPONDENT IN YEARS: MARITAL STATUS MARRIED 1 RESPONDENT’S LEVEL OF EDUCATION: DIVORCED 2 NONE 1 WIDOW 3 PRIMARY 2 SECONDARY 3 HIGHER OTHERS:
B) HOUSEHOLD HEAD’S DETAILS
HOUSEHOLD HEAD’S NAME:
SEX OF HOUSEHOLD HEAD: MALE 1 FEMALE 2 RELATIONSHIP OF RESPONDENT TO THE HEAD OF HOUSEHOLD:
C) FAMILY CHARACTERISTICS
FAMILY TYPE: NUCLEAR FAMILY 1 JOINT FAMILY 2 FAMILY SIZE: ______MEMBERS (I.E. FAMILY MEMBERS CONTRIBUTING/LINKED TO A SINGLE
KITCHEN NUMBER OF CHILDREN: _____ (BOYS _____ & GIRLS _____ )
ANY FAMILY MEMBERS WITH PHYSICAL DISABILITIES: YES 1 NO 2 Disable: Men _____ Women ____ Girl/s (under 18) ____ Boy/s (under 18) ____ Note: Nuclear Family = Family unit consisting of a mother and father and their children Joint Family = Family that consists of parents, children, and close relatives/multiple generations
D) House Structure
HOUSE STRUCTURE: (TICK ONE FOR EACH CATEGORY A-C) A) FLOOR TYPE KATCHA (RUDIMENTARY) 1
PAKKA (FINISHED) 2 B) WALLS KATCHA (RUDIMENTARY) 3 PAKKA (FINISHED) 4 C)ROOF KATCHA
(RUDIMENTARY) 5 PAKKA (FINISHED) 6
[239]
E) ASSETS & DURABLE GOODS
DO YOU OWN ANY OF THE FOLLOWING ASSETS? (TICK ALL MENTIONED) HOUSE 1
LAND 2 LIVESTOCK 3
DO YOU OWN ANY OF THE FOLLOWING DURABLE GOODS? (TICK ALL MENTIONED)
BICYCLE 1 MOTORCYCLE/SCOOTER 2
RADIO 3 TELEVISION 4 FRIDGE 5 OTHER ____________________
NOTE: FOLLOW SKIP PATTERN WHERE APPLICABLE
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
1. 2 What work do you do? Record Only One Option
Does Not Work…………..………..………………..1 Household Chores………....………….………......2 Self Employed ….……….................................3 Salaried Worker……………………..…...………...4 Retired ………………………………………………5 Other: _______________________________ 96 (Specify)
….Go to 4 ….Go to 4 ….Go to 4
2. 3
What is the nature of work you do? Record up to a Maximum of 3 Responses (Rank in priority 1,2,3) With 1 being the most important Migrant worker is defined as those who work overseas Skilled worker
Daily Labourer………………………….………..…..1 Harvesting/Farming………...…..……..….............2 Shopkeeper…………………………………..…….3 Street Vendor……………………. …….................4 Domestic Help/Household Worker………....…….5 Raising Live Stock…………………………….……6 Carpet Weavers……………………………….……7 Skilled Worker…………………………..……….….8 Migrant Worker……………………..………….……9
[240]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
is defined as those including drivers, carpenters, cobblers, etc
Government Job…………….…………………..…10 Private Job……………………….……………....…11 Other: ________________________________96 (Specify)
3. 4 What is your average monthly income?
(Average will be used for those who work either on daily/weekly paid basis – an estimates average should be stated. In case of those earning on monthly basis then that should be stated).
Amount In PKR:
None: ...............................................................1 Did Not Respond.............................................98 DK...................................................................99
RESUME IN CASE OF SKIP FROM Q 1
4. Does your spouse work?
Yes……………………………………………………1 No……………………………………………………..2
….Go to 7
5. 3
What is the nature of work that your spouse does? Record up to a Maximum of 3 Responses (Rank in priority 1,2,3) With 1 being the most important Migrant worker is defined as those who work
Daily Labourer………………….…………………..1 Harvesting/Farming………...……....………........2 Shopkeeper……………………………..….……..3 Street Vendor……………………. …………...,....4 Domestic Help/Household Worker………..…….5 Raising Livestock…………………………………6 Carpet Weavers……………………..……………7 Skilled Worker………………………………….….8 Migrant
[241]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
overseas Skilled worker is defined as those including drivers, carpenters, cobblers, etc
Worker………………..……….…….……9 Government Job………………….…………..…10 Private Job……………………….…………....…11 Other: _______________________________96 (Specify)
6. What is your Spouse’s average monthly income?
None..................................................................1 Amount In PKR: Did Not Respond.............................................98 DK...................................................................99
RESUME IN CASE OF SKIP FROM Q 4
7. What is your household’s average monthly income?
Note: Confirm using the ‘Contributors to one Kitchen expense’ definition whether anyone other than the respondent and his/her spouse are contributing to the household income
Amount In PKR: Did Not Respond.............................................98 DK........................................................99
Access To Water, Use Of Household Water Treatment And Safe Storage
8. Do you have access to drinking water? Note: PROBE IF NO. People would normally have some access even if poor or from outside.
Yes ..................................................................1 No ..................................................................2
….Go to 16
[242]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
9. Where is the drinking water point located?
Inside the House...................................................1 Outside the house.................................................2 DK .................................................................99
10. 5 What is the main source of drinking water for your household? Record up to a Maximum of 3 Responses (Rank in priority 1,2,3) With 1 being the most important Surface water include: river, dam, lake, pond, stream, canal, irrigation channels HandPump: Note in Others 96
Piped Water Into Dwelling (House)..................….1 Piped Water To Yard/Plot………............………...2 Public Tap/Standpipe……………………..............3 Tube Well/Borehole……….............………….…...4 Protected Dug Well….……….........................…..5 Unprotected Dug Well………..........................….6 Protected Spring…………….............………….…7 Unprotected Spring ………............……………....8 Rainwater Collection………..............………..…..9 Bottled Water…………………..............………...10 Cart With Small Tank/Drum….............………....11 Tanker-Truck ……………….............…………....12 Surface Water………………………............……13 DK………………….....………...................99 Other______________________96 (Specify)
11. Do you treat your drinking water?
Yes……………………………………..1 No……………………………………2 DK…………………………………..99
….Go to 13 ….Go to 13
12. How do you treat your drinking water? Record up to a Maximum of 3 Responses
Boil……………………………………1 Add Bleach/Chlorine…………………….2 Strain It Through A Cloth……………….3 Use A Water Filter………………….…...4 Solar Disinfection……………………...5 Let It Stand And Settle………………...6
[243]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
(Rank in priority 1,2,3) With 1 being the most important Water filter: Do they pass water through sand/gravel or charcoal/carbon or ceramic to clean it.
Don’t Know………………………………99 Other: __________________________96 (Specify)
RESUME IN CASE OF SKIP FROM Q 11
13. How do you store drinking water? Record up to a Maximum of 3 Responses (Rank in priority 1,2,3) With 1 being the most important Containers include: bucket, jerry can, jerkin, bottle, drum
In Containers……………………………..1 Roof Tank ……………….………………2 No Water Stored………………………….3 Don’t Know……………………………99 Other: __________________________96 (Specify)
….Go to 16 ….Go to 16 ….Go to 16 ….Go to 16
14. If in container, ask permission to see the containers and observe are they:
Narrow Mouthed……………………………...1 Wide Mouthed…………………………...2 Of Both Types…………………………...3 Not Observed…………………….……...4 Not Allowed………………………….……...5
15. If in containers, observe are they covered?
All Are……………………………….………1 Some Are…………………………………...2 None Are…………………….……………...3
RESUME IN CASE OF SKIP FROM Q 8 / Q13
16. Do you have access to water for other purposes? (‘Other
Yes ...........................................................1 No ............................................................2
….Go to 22
[244]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
purposes’ such as cooking, hand washing and bathing)
17. Where is the location of water point for purposes other than drinking?
Inside the House........................................1 Outside the house.....................................2 DK ..........................................................99
18. 6 What is the main source of water for other purposes of your household? Record up to a Maximum of 3 Responses (Rank in priority 1,2,3) With 1 being the most important (Such as cooking, hand washing and bathing) HandPump: Note in Others 96
Piped Water Into Dwelling (House)...............….1 Piped Water To Yard/Plot…….....……...2 Public Tap/Standpipe………..…….......3 Tube Well/Borehole………........…...4 Protected Dug Well….……................…....5 Unprotected Dug Well………................….6 Protected Spring……………….......……7 Unprotected Spring ………….........….8 Rainwater Collection………….............9 Bottled Water……...……………….... ….10 Cart With Small Tank/Drum…….....…….…...11 Tanker-Truck ………………….…….....12 Surface Water………………….....…..13 DK……………...................……………..99 Other___________________________96 (Specify)
19. Who usually goes to fetch water for your household? RECORD ALL MENTIONED (RANK in priority)
Adult Woman………………………...…………….1 Adult Man………………..………………2 Female Child (Under 18 Years)……..……3 Male Child (Under 18 Years)……........………….4 DK…………………......................……99
20. How long does it take to go there, get water, and come back?
Less Than 15 Minutes…………………1 15 Minutes To 30 Minutes………………2 30 Minutes To 1 Hour……………………3 More Than 1
[245]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
Probe: Try to probe the amount of time spent on socialization that should be excluded in the time value
Hour……………….………………..4 DK…………………..…………………..99
21. How many trips a day are made on average to collect water?
No. of Trips __________________________
RESUME IN CASE OF SKIP FROM Q 16
22. Are there any Water User Committees/CBOs in your villages that you use to claim/access WASH services?
Yes……………………………………...1 No………………………………………….2 DK………………………………….…….99 If ‘Yes’ What Services do you access, Water .......................................................1 Health & Hygiene Promotion Services.................2 Sanitation Services..............................................3 If ‘Yes’ Name of Organisation________________________
23. Are you paying for water for uses other than drinking?
Yes ..................................................................1 No.............................................................2 If ‘Yes’ To Whom Do You Pay _____________________ If ‘Yes’ Fixed Operation & Maintenance Cost.................1 Water Usage Based Cost...................................2 If ‘Yes’ How Much: PKR per month
Health & Hygiene Awareness Raising And Practice Of Hand Washing At Critical Moments
24. Are you aware of Yes …………………………………………..1
[246]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
the need to achieve Open Defecation Free status in your village?
No ……………………………………..……….2 DK …………………………………………...99
25. 24a
Have you received any health and hygiene related message in the last 6 months?
Yes …………………………………………..1 No …………………………………..……….2 DK ……………………………………...99
….Go to 30 ….Go to 30
26. 19 Where did you get the message related to health, hygiene and sanitation? Record up to a Maximum of 3 Responses (Rank in priority 1,2,3) With 1 being the most important
LHWs………………………………………..1 Hospital/BHU……………….……………2 Health Staff……………………………....……3 Village Pharmacy/Shopkeeper……………………4 CBOs…………………………………..…….5 Religious Leaders……………………………..…...6 Friend/Neighbour………………..……..…7 Awareness Raising Session………………..….....8 Radio……………………………………..…9 TV……………………………………..…...10 Print Media/Material.………………..…...11 Folk Media/Theater…………………….…12 Mothers/Youth Groups…………..…..…..13 School Children………………………..……..….14 Don’t Know.…………….………………99 Others ________________________ 96 Specify
27. Which source of information do you prefer more? RECORD THREE MOST PREFERRED SOURCES (Rank in priority 1,2,3) With 1 being the most important
LHWs………………………………………...1 Hospital/BHU…………………….……….…2 Health Staff……………………………………….…3 Village Pharmacy/Shopkeeper………………....…4 CBOs……………………………….……..5 Religious Leaders……………….………….….…...6 Friend/Neighbor……………….…….…7 Awareness Raising Session……………...8 Radio………………………………………9 TV……………………………………....10
[247]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
Print Media/Material.………….………...11 Folk Media/Theater……………………..…12 Mothers/Youth Groups………….…….....13 School Children……………..……….14 Don’t Know.…………………..….………99 Others ________________________ 96 Specify
28. 21 What health & hygiene related topics can you remember? RECORD ALL MENTIONED (Rank in priority 1,2,3) With 1 being the most important
Personal Hygiene……..…………..…………….…1 Clean And Safe Water....…………..……………..2 Water Born Diseases………….…………..3 Environmental & Domestic Hygiene…......4 Safe Disposal Of Human Excreta.... …...5 Solid Waste (Garbage) Disposal………..……….6 Do Not Know………………….….…....99 Other: _________________________ 96 (Specify)
29. 22 Are you practicing any of the messages you received? If “Yes”, please specify any three messages.
Yes ………………………………….………..1 No ……………………………………….2 DK …………………………………………...99 1.
2.
3.
RESUME IN CASE OF SKIP FROM Q 25
30. 26 Does your household have a separate/designated place for hand washing?
Yes………………………………….……….1 No……………………………………………2 DK………………………………………...99
31. Is soap available in your house? RECORD ALL MENTIONED Note: In Case of NOT AVAILABLE then this should be the only
Anywhere In The Dwelling....……………………….1 Near Hand Washing Place.…………………………2 Near Latrine……………………..………3 Inside Latrine..………… …………..……...4 Not Available…………..………………..5 Other: __________________________96 (Specify)
[248]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
entry.
32. 27 (With Women Respondents Only) Ask to see the place used most often for hand washing and latrine to observe which of the above responses with regards to availability of soap are true: (Observe above responses only with Women Respondents)
Anywhere In The Dwelling....……………………….1 Near Hand Washing Place.……… ………2 Near Latrine………………………………3 Inside Latrine..……… …………………...4 Not Available………..…………………..5
33. 28 When do you usually wash your hand with soap? RECORD ALL MENTIONED Note: Number of. Critical Moments
Before Eating………………………….........1 After Eating………….....…………………2 Before Praying……...…..……………….3 Before Breastfeeding A Child…………… .4 Before Feeding A Child ……………………5 Before Cooking Or Preparing Food………6 After Defecation………………..…………...7 After Urination……………………………….8 After Cleaning A Child That Has Defecated/Changing A Child’s Nappy……9 When Your Hands Are Dirty………………………10 After Cleaning The Toilet Or Potty……………….11 DK……………………………..……….99 Other _______________________________ 96 Specify
34. Can household members with physical disabilities use the hand washing station?
Yes …………………………………………….1 No……………………………………....2
35. 29 Can you name the diseases caused by using
Diarrhoea………………………………….1 Cholera…………………………………...2 Hepatitis………………………………..…3
[249]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
unsafe water? RECORD ALL MENTIONED (Rank in priority 1,2,3) With 1 being the most important
Skin Disease…………………….…….4 Don’t Know…………………………99 Other: _________________________96 (Specify)
36. 30 In the past two weeks has any child (less than 5 years) in your household suffered from diarrhea?
Yes………………………………………………1 No………………………………………………..2 DK………………………………………………..99 Was the child, Male ………………………………….…………1 Female …………………………………………2
37. 61 What did you do the last time your youngest child passed stools? MARK SINGLE RESPONSE ONLY
Put/Rinsed Into Latrine………………..…………..…1 Child Used Latrine……………………………………2 Thrown Into Garbage…………………….…………..3 Buried In Yard ………………………………………..4 Not Disposed Of/Left On The Ground……………..5 DK…………………….………………………………99 Other___________________________ ______ 96 Specify
Access To & Use Of Sanitary Facilities And Disposal Of Human Excreta
38. Do you have a latrine?
Yes …………………………………………………1 No…………………………………………………...2 How Many _____________________
….Go to 40
39. Where is the latrine located? In relation to their
Inside the house.....………………………………1 Outside the house….…………………………….2
[250]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
specific house/property (not to be mixed with communal latrine)
RESUME IN CASE OF SKIP FROM Q 38
40. 31 Where do you defecate?
Latrine………………………………………...…...1 Open……………………………………………….2 Both……………………………………….……….3
…Go to 42
41. 33 Why do you practice Open Defecation? Record up to a Maximum of 3 Responses (Rank in priority 1,2,3) With 1 being the most important
No Latrine………………………..……….…..…….1 Latrines Are Dirty……………………….………….2 No Privacy……………………………….….………3 Latrine Far From House……………………..……4 Insufficient Numbers of Latrines………..………..5 Lack of Water for Defecation……………………..6 DK………………………………………………….99 Others________________________________96 Specify
42. Where do your household members defecate?
Latrine………………………………………...…...1 Open……………………………………………….2 Both……………………………………….……….3
….Go to 45
43. 35 Which household members practice open defecation (OD) more? RECORD ALL MENTIONED (Rank in priority 1,2,3) With 1 being the
Men………………………………………………...1 Women…………………………………………….2 Boys (5 Years And Above)…………….………..3 Girls (5 Years And Above)……………..……..…4 Boys (Under 5 Years)…………………..……..…5 Girls (Under 5
[251]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
most important
Years)………………….…………6 DK……………………………………………….….99 Other_________________________________96 Specify
44. 33 Why are your household members practicing Open Defecation? Record up to a Maximum of 3 Responses (Rank in priority 1,2,3) With 1 being the most important
No Latrine…………………………………....…….1 Latrines Are Dirty………………………………….2 No Privacy…………..……………………..………3 Latrine Far From House………..…………...……4 Insufficient Numbers of Latrines……………..…..5 Lack of Water for Defecation…………………....6 DK………………………………………………….99 Others_______________________________96 Specify
RESUME IN CASE OF SKIP FROM Q 42
45. 34 Is there any designated place for open defecation?
Yes………………………………………………...1 No………………………………………………….2 DK…………………………………………………99 If Yes, Inside House ……………………………………1 Outside House ………………………………….2 Other ______________________________96
46. 36 Is it safe to go for open defecation? RECORD ALL MENTIONED (Mark for all
Men YES………………………………………………...1 NO………………………………………………….2 Women
[252]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
types of family members)
YES………………………………………………...3 NO………………………………………………….4 Children YES………………………………………………...5 NO………………………………………………….6 Household Members with Physical Disability YES…………………………………………………7 NO…………………………………………………..8 DK…………………………………………………99
47. 41 What kind of latrine facility do members of your household usually use? Probe if “flush” or “pour flush” probe: Where does it flush to? Also observe the latrine Hanging Latrine: Where the latrine is built directly over body of water (river, stream) VIP Latrine (A pit latrine with ventilation in the form of a pipe from the pit to the outside of the roof taking the smell/flies outside) Composting
Flush/Pour Flush To: Piped Sewer System ……..……………………1 Septic Tank ………………,.…………………....2 Pit Latrine………………..…………………..…..3 Elsewhere……………..……………………..….4 Unknown Place/Not Sure/DK Where…………5 Ventilated Improved Pit Latrine………………....6 Pit Latrine With Slab……………………………...7 Pit Latrine Without Slab/Open Pit……….……...8 Composting Toilet……………………………..…9 Bucket …………………………………………....10 Hanging Toilet/Hanging Latrine…………….….11 Public/Communal Latrine…………………….…12 No Facilities Or Bush Or Field…………………13 Other ________________________________96
[253]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
Toilet: A latrine where little or no water is used, and sawdust, peat moss (khus) may be mixed with excreta to help it compost)
(Specify)
48. 42 Which members of your immediate household use the latrine in the house? RECORD ALL MENTIONED
Male Adults……………………………….………....1 Female Adults ......................................................2 Male Children 5 Years Or Above .........................3 Female Children 5 Years Or Above......................4 Household Members with Physical Disability……5
49. 43 When did you construct the latrine in your house?
In the last year ……………………………………..1 In the last three years ……………………………..2 In the last 5 years…………………………………..3 DK………………………………………………….99
50. 44 Did you receive help from any organization for latrine construction in any of the following? RECORD ALL MENTIONED (Rank in priority 1,2,3) With 1 being the most important
Need Identification….……………………………..1 Supply Facilitation (Design, Material)……………2 Loan ………..…..…………………………………..3 Cost Share Basis…………………………………..4 Demo Latrine……………………………………….5 Subsidy……………………………………………..6 Trained Skilled Labour……………………………7 No External Support………………………………8
[254]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
Other ________________________________96 DK ………………………………………………...99 Specify In case of facilitation, Name Of Organisation:_____________________________ How many latrines did you receive support for: ____
51. 51 Did you receive help from any organization for latrine construction in any of the following?
IGNORE REDUNDANT QUESTION
Need Identification….……………………………..1 Supply Facilitation (Design, Material)……………2 Loan ………..…..…………………………………..3 Cost Share Basis…………………………………..4 Communal Water Facility ……………………….5 Subsidy……………………………………………..6 Trained Skilled Labour……………………………7 No External Support………………………………8 Other ________________________________96 DK ………………………………………………...99 Specify In case of facilitation, Name Of Organisation:_____________________________ Type of Water Facility: ______________________
[255]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
52. 46 Did you easily get the required supplies necessary for construction of latrines?
Yes…………………………………………………1 No…………………………………………………..2 DK………………………………………………….99
53. 47 From where did you get the supplies?
General hardware shop in Yes 1 No 2
the same village Shops in the nearby village Yes 1 No 2 Shops in the nearby city Yes 1 No 2
54. 48 Did you get any of the latrine supplies/material on loan?
Yes…………………………………………………1 No…………………………………………………..2 Dk………………………………………………….99
55. 49 Are trained skilled masons/labour available in your village that can construct a latrine?
Yes………………………………………………….1 No………………………………..…………………2 DK…………………………………………………99
56. 50 Is your household latrine cleaned regularly?
Yes ……..………………………………….……….1 No……..…………………………………………….2 DK…………………………………………………..99 Other_________________________________96 Specify
….Skip to 58 ….Skip to 58
57. 51 How frequently is your latrine cleared in a week?
____________________________
RESUME IN CASE OF SKIP FROM Q 56
58. 52 Who usually cleans the latrine in your house?
Boys………………………………………………..1 Girls………………………………..…………
[256]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
RECORD ALL MENTIONED
…….2 Women…………………………………………….3 Men…………………………………………………4 Dk………………………………………………….99 Other______________________________ 96 Specify
59. 53 Is your latrine linked to drainage? CHECK: In Case of YES confirm with response for Q38. Observe Where Possible
Yes……………………………………………………1 No……………………………………………………..2 If YES, Out in the open………………………………………1 In to an Open Drain..………………………………..2 Underground Drain …………………………………3 Observation ____________________________
….Go to 61
60. 54 Are you aware if your latrine’s underground drain is linked to any sort of waste water treatment system?
Yes………………………………………1 No…………………………………………..2 DK……………………………………….99 If YES What Type? ______________________________
RESUME IN CASE OF SKIP FROM Q 59
61. Have you received any awareness message from any organization about the hazards of open defecation in the last One Year?
Yes……………………………………1 No…………………………………..2 If Yes Name Of Organization-----------------------
62. 37 What are the consequences/ hazards of open defecation? RECORD ALL
Dirty Surroundings……………………………..…1 Diseases…….…………………………..…2 Environmental Problems……………………..…..3 Physical Problems From Not Being Able To
[257]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
MENTIONED
Go Out During The Day (Women)…………………….….4 DK………………………………………99 Other: ________________________96 (Specify)
63. 60a
Are you aware of any design option for constructing a low cost latrine?
Yes……………………………………….1 No……………………………………...2 If Yes What type of Latrine ________________ If Yes How did you get to know this option____________________________
64. 38 Given the consequences/ hazards of open defecation, would you be willing to construct a latrine for your household?
Yes……………………………………….1 No……………….……………………………2 DK…………………………………………99
….Go to 66 ….Go to 66
65. How would you bear the construction cost?
Self-finance……………………..…………1 Loan…….…………………………….……2 Cost-Share Basis………..…………………..……3 Other: ___________________________96 (Specify)
RESUME IN CASE OF SKIP FROM Q 64
66. Are there any communal latrines available? Communal Latrines : These are not household based latrines and are located at a common place for the use of several households, they are also usually
Yes……………………………………….1 No…………………………………………2 DK…………………………..…………...99
….Go to 70 ….Go to 70
[258]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
more than one in number, e.g are latrines in schools, mosques..
67. 60 Do you or your household members use communal latrines?
Men Yes……………………………………...1 No………………………………………….2 Women Yes……………………………………...3 No………………………………………….4 Girls Yes…………………………….……….5 No…………………………………………6 Boys Yes………………………………………..7 No…………………………………………8 DK…………………………………………99
….Go to 70 ….Go to 70 ….Go to 70 ….Go to 70 ….Go to 70
68. 60 Are there separate communal latrines for men and women?
Men Yes…………………………………………...1 No………………………………………….2 Women Yes………………………………………...3 No……………………………………….4 DK…………………………………………99
69. Where are these communal latrines located?
Mosque…………………………………..…1 School………………………..……..……2 DK………………………………………99 Other__________________ ______ 96 Specify
RESUME IN CASE OF SKIP FROM Q 66-67)
70. 62 What do you do with your household garbage?
Open Pit………………………….………..1 Closed Pit………………………………….2 Anywhere………………………………….3 Burning ………………………………...….4 Garbage Collector……………………...…5
[259]
NO QUESTIONS AND FILTERS
CODING CATEGORIES SKIP
Communal Waste Disposal Ground…….6 DK……………………………….…..99 Other _______________________ 96 Specify
Thank you for taking the time to meet with me and answer these questions. The information you provided will be kept strictly confidential and will be used with anonymity
Do you have any questions or comments for us?
OBSERVATION BASED QUESTIONS (with women respondents only check household for the following questions) Q. 14 , Q 15 and Q 32
[260]
Public School Sanitation Survey Questionnaire Sanitation Programme at Scale in Pakistan (SPSP) – Rural through Pakistan
Approach to Total Sanitation (PATS) Evaluation Feb 2014
Introduction to the Interviewee (Informed Consent):
I am _________________________, a representative of an independent consulting entity i.e. the AAN Associates, who has been contracted for the Evaluation of UNICEF’s Sanitation Programme at Scale (SPSP) in Pakistan using the Pakistan Approach to Total Sanitation (PATS). This programme aimed to improve sanitation practices, using Community-led Total Sanitation (CLTS) and School-led Total Sanitation (SLTS) approaches under the overarching PATS strategy.
I would like to ask about health, hygiene and sanitation related aspects/services at schools and about children’s knowledge and practices. This information will help Unicef and partners to assess projects achievements and performance in health, hygiene and sanitation related services for the children. The survey takes around 30 minutes to complete.
Whatever information you provide, will be kept strictly confidential and dealt with anonymity.
Participation in this survey is voluntary and you can choose not to answer any individual questions. However, I hope you will participate in this survey since your views are important.
At this time, do you want to ask me anything about the survey?
RESPONDENT AGREES TO BE INTERVIEWED
RESPONDENT DOES NOT AGREE TO BE INTERVIEWED END
INSTRUCTIONS:
RESPONDENT: QUESTIONNAIRE IS TO BE ADMINISTERED WITH SCHOOL
PRINCIPAL/SENIOR TEACHER
MULTIPLE CHOICE QUESTIONS REQUIRE FURTHER PROBING AND ARE ALSO
HIGHLIGHTED WITH CAPTION OF “RECORD ALL MENTIONED”.
RESPONSE: FOR EACH QUESTION CIRCLE THE APPROPRIATE RESPONSE/S.
OBSERVATION WALK: AFTER THE INTERVIEW TAKE A WALK AROUND THE SCHOOL AND
RECORD OBSERVATIONS.
Edited/Verified by: Approved By:
District Supervisor Provincial Coordinator
Signature: Signature:
Date: Date:
GENERAL INFORMATION
[261]
RECORD NUMBER: (001-TEAM XX)
1 Date of visit ____/____/_____
2 Level of school Primary Secondary/High Secondary
3 Type of School Boys Girls Coed
4 Number of students Boys:__________ Girls:____________
5 Number of students with physical disabilities
Boys: __________ Girls:_________
6 Number of Teachers Male: __________ Female:_________
7 Name of the school _______________________________
8 Province _______________________________
9 District _______________________________
10 Tehsil _______________________________
11 Union Council _______________________________
12 Village _______________________________
13 Distance from Community
(in minutes by walk)
____________min ___________ max
Health and Hygiene
No Question Response Skip Pattern
14 Do the classrooms have the following? RECORD ALL MENTIONED For observation visit one random class room. (Ventilation – Class should not smell musty, or be suffocative)
Windows 1 Ventilation 2 Dust Bin 3
15
What is the condition of the classroom floor? For observation visit one random class room.
Earth 1 Concrete 2 Other ____________96 and Clean 1 Unclean 2
[262]
16 Does the school dispose (solid waste) garbage?
Yes ……………………1 No……………………..2 DK………………..…..99
Go to 18 Go to 18
17 How frequently does the school dispose (solid waste) garbage? And how?
Daily 1 Weekly 2 Irregular 3 No disposal 99 Other ____________ 96 Incinerate 1 Collection Service 2 Thrown/Piled outside of school premises 3 Other____________ 99
18 Does the school have any cleaning staff?
Full time 1 Part time 2 None 3
19
During the last six month did the school/any other organization organize health and hygiene awareness activity?
School Yes 1 No 2
Organization Yes 1 No 2 Name of Organization ___________________
If ‘No’ in both cases Goto 21
20 What topics were covered by the awareness raising exercise? Circle all mentioned
Personal Hygiene…...….1 Clean And Safe Water…2 Water Born Diseases…..3 Environmental& Domestic Hygiene….……………….4 Safe Disposal Of Human Excreta......................….5 Solid Waste (Garbage) Disposal………………….6 Do Not Know…………..99 Other: _____________96
OBSERVATION ONLY
21 Is the school compound free from solid wastes?
Yes 1 No 2
22
Does the school have flowing liquid waste/puddles of stagnant water?
Yes 1 No 2
WATER SUPPLY
[263]
23
Is water available in the school for the following purposes? Mention all and circle each response
Drinking 1 Hand washing 2 Latrine 3 Not Available 4
Go to 25
24 What is the water source? RECORD ALL MENTIONED
Piped Water Into School 1 Piped Water To Yard/Plot 2 Public Tap/Standpipe 3 Tube Well/Borehole 4 Protected Dug Well 5 Unprotected Dug Well 6 Protected Spring 7 Unprotected Spring 8 Rainwater Collection 9 Bottled Water 10 Cart With Small Tank/Drum 11 Tanker-Truck 12 Surface Water 13 Don’t Know 99 Other___________________ 96 (Specify)
25
What type of water storage does the school have? RECORD ALL MENTIONED
Overhead Tank 1 Underground Tank 2 Open Drum 3 Covered Drum 4 Other ____________ 96
26 Are drinking points available for children?
Yes 1 No 2
27
Are hand washing points available for children’s use in the school?
Yes 1 No 2
28
What do children do when school water point is not working?
For drinking:
For hand washing __________________________ __________________________
OBSERVATION ONLY
29 Is soap available at the hand washing point?
Yes 1 No 2
[264]
30 Is the area around the hand washing facility clean?
Yes 1 No 2
31
Is the area around the drinking source clean?
Yes 1 No 2
SANITATION
32
Is a latrine available in the school compound?
Yes 1 No 2
If No Goto 45
33 What is the total number of latrines for the following?
Teachers _______ Boys _______ Girls _______
34 Is the latrine in usable condition?
Yes 1 No 2 DK 99
35 What is the type of school latrine? Hanging Latrine: Where the latrine is built directly over body of water (river, stream)
Flush/Pour Flush To: Piped Sewer System 1 Septic Tank 2 Pit Latrine 3 Elsewhere 4 Ventilated Improved Pit Latrine 5 Pit Latrine With Slab 6 Pit Latrine W/O Slab/Open Pit 7 Composting Toilet 8 Bucket 9 Hanging Toilet/ Latrine 10 No Facilities/Bush/Field 11 Other___________________ 96 Hanging Latrine: Where the latrine is built directly over body of water (river, stream)
36 What type of floor does the latrine have?
Concrete 1 Slab 2 Earth 3
37 What is the structure of the latrine?
Walled 1 Has Door 2 Ceiling 3 Other _____________96
38 Does the latrine have a drain pipe?
Yes 1 No 2 DK 99
39 Is water available in the latrines for washing
Piped/Tap 1 Bucket 2
[265]
purposes?
Other ______________96
40 Is container (mug, pot) for ablution (Wazu) available?
Yes 1 No 2 DK 99
41 What do children do when latrines are not functioning/breakdown?
Open Defecate In School 1
Outside school 2 Go to home 3 DK 99 Other__________ 96
42 OBSERVATION ONLY
43 Is excreta visible around the latrine? Enumerator to check latrine facility
Yes 1 No 2
44 Was there excreta inside the latrine? Enumerator to check latrine facility
Yes 1 No 2
45 Are there children that defecate or urinate in the open?
Defecate Yes 1 No 2 Urinate Yes 1 No 2
Go to 47 Go to 47
46 Which students defecate in the open?
Boys (5 Years And Above)……..1 Girls (5 Years And Above)…..…2 Boys (Under 5 Years)………..…3 Girls (Under 5 Years)…...………4 DK……………………………….99 Other____________________96 Specify
47 Is soap available? Inside the latrine 1 Near Latrine 2 Not Available 3
48
Are children with physical disability able to use the hand washing facility on their own?
Yes 1 No 2 DK 99
49 Are children with physical disabilities able to use the
Yes 1 No 2
[266]
latrine on their own?
DK 99
50. OBSERVATIONS _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________
24. Annex 13 SPSP Outputs - Target Vs. Actual
No. Output Indicators Target
Actual
Difference
As of
30-Apr
A. Demand for sanitation to achieve open defecation free status created in targeted communities Target Actual
1 Boys and Girls reached with sanitation messages in Schools. 103,000 176,022 171%
2 Men, Women, Boys & Girls reached with sanitation messages 776,000 1,200,529 155%
3 People reached with sanitation messages through electronic/print media. 970,000 540,786 56%
4 People reached with sanitation messages through community events (theatres, puppet shows) 291,000 195,450 67%
5 No. of training events conducted for school teachers (men, women) on facilitating SLTS 23 84 365%
6 No. of WASH clubs formed in schools and strengthened with planned follow up capacity development sessions 692 881 127%
7 No. of events conducted by school wash clubs (boys and girls) as per plan 886 1,433 162%
8 No. of training events conducted for social organizers on participatory tools and IEC material for facilitating PATS 8 85 1063%
9 No. of social organizers successful using participatory tools and IEC material for facilitating PATS in villages according to developed criteria 230 911 396%
10 No. of training events conducted for CRPs on conducting CLTS in villages 46 67 146%
11 No. of CRPs successfully facilitating CLTS in villages according to developed criteria 1,385 3,471 251%
12 No. of VSCs existing/formed and trained 1,385 2,994 216%
13 No. of villages/clusters with developed Community Action Plan (CAP) 1,385 1,878 136%
14 No. of CAPs developed with prioritized need for action by community on Hygiene Behaviours 1,385 2,233 161%
B. Sanitation demand sustained with supply side interventions
15 No. of masons trained on sanitation construction and using their learnt skills 692 1,250 181%
16 No. of latrines (other than demo latrines) constructed by masons trained under PATS programme 13,857 20,040 145%
17 No. of complete demonstration latrines constructed for extremely vulnerable families as per agreed criteria with community (maximum 3%) 4,157 5,212 125%
[267]
18 No. of demonstration latrines constructed through materials provision/vouchers for vulnerable families as per agreed criteria with community (max 7%) 9,700 8,581 88%
19 No. of entrepreneurs/manufacturer trained and producing low-cost sanitation solutions 692 469 68%
20 % of trained entrepreneurs doing sanitation related business 80% 199 24930%
21 No. of facilitated sanitation marts doing profitable business 346 82 24%
22 No. of lending institutions successfully approached to give loans for sanitation facilities 4 8 200%
23 No. of households received sanitation loan from some lending institution facilitated by PATS programme 140 0 0%
24 No. of communal water facilities constructed as incentives/rewards 620 552 89%
25 No. of communal sanitation facilities constructed as incentives/rewards 159 88 55%
26 No. of house sanitation facilities constructed as incentives/rewards 11,179 1,202 11%
27 No. of house water facilities constructed as incentives/rewards 11,262 1,383 12%
28 No. of critical community infrastructure constructed as incentives/rewards 158 2 1%
29 No. of health and hygiene packages given as incentives/rewards 242,000 253,330 105%
C. Improved Health and Hygiene behaviors promoted with the target community
30 No. of people reached with WASH BCC messages through electronic/print media. 970,000 609,530 63%
31 No. of people reached with WASH BCC messages through community events (theaters, puppet shows) 183,300 165,092 90%
32 No. of school children (boys and girls) reached with messages through SLTS activities. 63,941 154,526 242%
33 No. of people (men, women, boys, girls) reached with WASH BCC messages through CRPs 291,000 846,444 291%
D. Linkages developed with duty bearers/public institutions to support PATS
34 No. of capacity needs assessments (CNA) conducted for relevant public sector stake holders 14 6 43%
35 No. of capacity development plans developed for relevant public sector stake holders by Implementing Partner 14 6 43%
36 No. of capacity development events undertaken for relevant public sector stake holders by Implementing Partner 14 15 107%
37 No. of government officials and political leaders participated in capacity development and key events of the project. 185 230 124%
38 No. of advocacy initiatives undertaken with duty bearers/political leadership to support PATS 14 17 121%
39 No. of MoUs signed by implementing partner with relevant public institutions responsible for sanitation 14 17 121%
40 No. of villages jointly selected for PATS programme by relevant Government duty bearers with IP 1,385 1,791 129%
41 No. of ODF communities received rewards/incentives by government 39 62 159%
E. Villages supported in attaining Total Sanitation status
42 No. of households with new latrines constructed after start of the PATS programme 17,182 33,673 196%
43 No. villages declared as ODF by community 831 1,436 173%
44 No. of villages Verified as ODF by Implementing partner and independent monitors 831 1,250 150%
45 No. of people (men, women, boys, girls) in communities declared as ODF 581,700 711,467 122%
46 No. of people (men, women, boys, girls) in communities verified as ODF 581,700 679,967 117%
[268]
47 No. of model villages with laid sewers, WW collection and treatment system through PATS programme 4 18 450%
48 No. of advocacy initiatives undertaken to promote communal underground drainage system linked WW treatment system and communal SWM system 2 22 1100%
49 No. of ODF villages provided with underground drainage and WW treatment facility by relevant public sector stake holder/duty bearer 2 0 0%
F. PATS programme Monitoring Evaluation and Learning Framework implemented
50 No of Inception workshops conducted in the beginning of project 12 12 100%
51 No. of villages with baseline established in the beginning of project by IP 1,385 7,448 538%
52 No. of joint progress monitoring reports (IP and public institution) 80 39 49%
53 No. of monthly substantive monitoring reviews’ visits conducted for confirming/measuring performance indicators in results framework 80 216 270%
54 No. ‘monthly substantive monitoring review’ reports shared among partners 63 55 87%
55 % of responsive progress reports submitted as per MEAL framework by IP 100% 33 3336%
56 No. of quarterly progress reviews conducted with IP and minutes documented 2 15 750%
57 No. of mid-term review workshop report endorsed by UNICEF 1 4 400%
58 No. of Human interest stories documented by IP and accepted as per UNICEF guidelines 80 78 98%
59 No. of end line survey reports accepted by UNICEF 11 2 18%
60 No. of Lessons sharing workshop conducted and feedback documented 1 1 100%
[269]
25. Annex 14
Reference Group for Evaluation
1. Why a Reference Group?
The draft Regional Evaluation Strategy requires all COs to form a Reference Group (RG) for each evaluation. According to the UNICEF Evaluation guidelines, RG is a small group of specialists and stakeholders related to an intervention to be evaluated, which is formed by the concerned programme in collaboration with the Evaluation Management Committee (EMC).
The RG should ideally consist of members from all key stakeholders, government, implementing partners and one or two technical experts relating to the sector under evaluation. For each evaluation, members of the RG should be selected carefully on the basis of relevance to the sector and intervention. The RG will support the internal EMC in overseeing the entire process of a specific evaluation and assist in ensuring quality, adherence to UNEG standards and making the evaluation useful for all stakeholders.
The RG is an interface between the Programme (focal person/s for evaluation) and the external evaluation team. Its members help the external evaluation team to identify data sources and to access them. They validate the evaluation questions and discuss the conclusions and recommendations. The reference group allows the variety of points of view on the evaluated intervention to be expressed.
2. Roles and Responsibilities Discuss and comment on the terms of reference drawn up by the evaluation
manager (focal person assigned by the programme/EMC). Aggregate and summarise the views of the commission services and act as an
interface between the evaluation team and the services, thereby supplementing bilateral contacts.
Help ensure that the evaluation team has access to and consults all information sources and documentation on activities undertaken.
Help validate the evaluation questions and methodology. Discuss and comment on notes and reports produced by the evaluation team.
Comments by individual members of the reference group are compiled by the focal person managing the evaluation process and subsequently transmitted to the evaluation team.
Assist in feedback of the findings, conclusions and recommendations from the evaluation.
3. Possible Membership Different types of actors can be invited to participate in the Reference Group. For evaluations managed in the country, membership may extend to: UN counterparts (where applicable) Government partners/services Selected development partners Selected civil society organizations
[270]
Experts15 (One or more members of the reference group may be invited as a resource person, for their expertise)
As per UNICEF guidelines, RG should be limited to 08 to 10 members for effective functioning.
4. Process
The Chief/Focal Person/Manager/ managing the evaluation:
Officially announces the launching of the evaluation (as per the IMEP);
Identifies the services, institutions or organisations (key stakeholders) and sends request to appoint a person as a member of the RG;
Request expert/s to join for technical support;
Shares a list of all members of reference group with EMC/PMER; and
Coordinates all activities of RG
5. When and how to involve the Reference Group? The RG is generally involved in the following way16:
Stage Meeting Role
Terms of reference Comments
Engagement of the external evaluation team Comments
Evaluation questions/methodology 1 Validation of ToR
Inception report Comments
Draft/First phase report (desk) 2 Discussion, comments
Field phase debriefing 1-2 Discussion
Draft final report, conclusions and recommendations
2-3 Discussion, comments
6. Opening the evaluation process for wider comments and advice
In certain cases, under the guidance of the RG and its inputs, the evaluation manager/programme may benefit from a wider variety of views through other channels like identification of and consultation with broader stakeholders by the evaluation team in the first stage of the process; participatory workshops, focus groups or expert panel(s) during the evaluation process; or discussion of the draft evaluation report in a widely attended seminar.
15 On engaging experts, further guidelines are available on
http://ec.europa.eu/europeaid/evaluation/methodology/methods/mth_exp_en.htm 16 For details on how to engage the Reference Group, detailed guidelines are available on
http://ec.europa.eu/europeaid/evaluation/methodology/methods/mth_stg_mee_en.htm