acf pakistan drm (rne) final evaluation
TRANSCRIPT
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
1/34
Photo NIaz Murtaza
Funded by Royal Norwegian Embassy in Islamabad
Niaz Murtaza, January 2012
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
2/34
ACF Pakistan RNE DRM Project Evaluation, 2012
1
TABLE OF CONTENTS
Executive Summary
2
Chapter 1: Brief Background
Emergency Context in Pakistan
Overview of the ProjectProject Details
4445
Chapter 2: Methodology
Evaluation QuestionsEvaluation Approach
Evaluation Constraints
777
8
Chapter 3: Findings
TargetingProject DesignProject ImplementationProject MonitoringProject SustainabilityCross-Cutting Issues
99
1316192022
Chapter 4: Conclusions and Recommendations
ConclusionsRecommendations
252526
Appendix
Damage in KP from 2010 FloodsScoring Matrix
AcronymsList of interviews
List of documents reviewedGood practice case study
2929293131
3232
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
3/34
ACF Pakistan RNE DRM Project Evaluation, 2012
2
XECUTIVE SUMMARYOver the last decade, Pakistan has experienced large-scale internal displacement
caused by a range of natural and human-made disasters. Structural poverty,
inappropriate development, rapid urbanization, inadequate infrastructures,
increased deterioration of the environment - deforestation, increased human
settlements in hazard prone areas, etc. have increased the vulnerability to disasters. Thus, even
a moderate flooding could have devastating effects in the future. An ACF DRM assessment
identified a lack of awareness amongst stakeholders concerning DRM, and thus a need for
capacity building. Thus, ACF carried out a DRM project funded by the Royal Norwegian
Embassy during 2011-12 to reduce morbidity and mortality risks by improving community
resilience to natural disasters in Nowsherra, Charsadda, Mardan and Lower Dir regions of KPK
province in northern Pakistan which are vulnerable to both serious floods and earthquakes.
At the end of 2012, ACF commissioned an evaluation to evaluate the impact and approach of
ACFs RNE funded DRM project. ACF subscribes to the Development Assistance Committee(DAC) criteria for evaluation: Impact, Sustainability, Coherence, Coverage, Relevance /
Appropriateness, Effectiveness and Efficiency. ACF also promotes systematic analysis of the
monitoring system and cross cutting issues (gender, HIV/AIDS etc). These dimensions served
as the main evaluation criteria. The tools included documents review, interviews with ACF
international and national staff, government officials and partner staff and household interviews
(with 204 community members), Focus Group discussions and transect walks in 18 villages.
Conclusions and Recommendations
Overall, the project is highly relevant to Pakistan given its high disaster vulnerability and is a
well-conceived one with an excellent and comprehensive mix of hardware and softwareactivities. Many of the project activities have also been implemented well though others do have
some room for improvement. The main findings can be summarized as follows: i) Targeting of
the most vulnerable is excellent in Charsadda and Nowshera but is less accurate in Mardan and
Dir; ii) ACF has set-up village committees to enhance the sustainability of project activities but
the committees must be linked adequately with other NGOs and the government to enhance
sustainability, iii) The hardware activities should focus more on those which provide more
structural and pro-active protection against floods, iv) Spreading the message to the whole
communities in villages has proven to be a challenge due to their large sizes, v) A sizeable
proportion of the project activities got delayed to the last month due to government permission
delays, procurement delays and staff training, vi) there is a need to enhance the technical
capacity of the team in software DRM activities, vii) monitoring and accountability mechanism
need to be strengthened. Thus, the main recommendations are as follows:
I. Targeting: It is recommended to continue work in Dir, Charsadda and Nowshera but to
reconsider working in Mardan due to its relatively low vulnerability and to undertake a more
thorough needs assessment in Dir to be able to target the most vulnerable village there. ACF
E
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
4/34
ACF Pakistan RNE DRM Project Evaluation, 2012
3
is also advised to focus on districts highly vulnerable to major floods based on the following
three criteria: River and drainage channels mapping; Damage incurred during 2010 floods and
UNDP One DRM project list of most vulnerable districts. Consider Sindh for DRM work too
since it has recently become the most repeatedly struck province in Pakistan.
II. Number of villages: Review the possibility of increasing the number of villages in the nextphase by increasing the number of field staff, covering each village with only one hardware
construction activity and linking with other NGOs and government agencies to cover some of
the construction work that ACF cannot afford in its budget.
III.
Selection of hardware activities: Use the prevention-mitigation-capacity-evacuation-
rescue-relief continuum of DRM activities presented in Chapter 3 to select hardware activities.
The contingency stocks are an important hardware input given their relatively low cost and the
fact that they cover three categories in the continuum even though they are the three bottom-
most categories. Select other hardware construction activities from the highest possible
category of this continuum which is relevant and affordable.
IV.
Village committee sustainability: Enhance the sustainability village committees by
providing them training on CBO management issues, having them adopted by other agencies
working long-term in the area (e.g., Concern), and linking nearby committees with each other
for mutual support and learning during crisis and normal times
V.
Community DRM awareness-raising: Follow up more closely with committees about their
community DRM awareness-raising session schedules and also look at the possibility of
training young educated and unemployed persons in each UC to conduct sessions in all
targeted villages in that UC for a small honorarium
VI. Monitoring and accountability: Develop formal monitoring plans which clearly specify the
role for all relevant staff in the projects monitoring, including the frequency, modality, project
dimension and indicators for each relevant ACF staff and how the information coming from the
monitoring done by each staff will be analyzed and summarized Also institute strong and
responsive accountability measures in future projects. Build sufficient time in projects for
government approvals and procurements.
VII. External linkages: ACF should aim to increase government involvement in the project. ACF
could get its village committees registered, and advocate with the DDMAs to treat NGO village
committees as valid structures any government future grass-roots DRM initiatives. It should
sign an MOU with government at time of starting the project which mentions the roles,
responsibilities and inputs that both sides will provide. ACF should also enhance its profile in
coordination sectors like the DRR Forum and the UNDP DRM project and obtaining greater
technical inputs from them. Enhance technical inputs on DRM work by having CPDM provide
some degree of field-based support to ACF staff on software issues, e.g., by developing the
software package in one model village.
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
5/34
ACF Pakistan RNE DRM Project Evaluation, 2012
4
HAPTER
1
: BACKGROUND
This chapter provides an overview of the emergency context in Pakistan, and
the specific project implemented by ACF with Royal Norwegian Embassy
funding which is the subject of evaluation in this report. The chapter serves the
purpose of placing the whole evaluation in its proper context and in familiarizingthe reader with the project constraints and scope.
1.
Emergency Context in Pakistan
Over the last decade, Pakistan has experienced large-scale internal displacement caused by a
range of natural and human-made disasters. Large-scale displacement occurred in Pakistan in
2010 after the worst flooding to hit Pakistan in memory affected 20 million people, forcing over 7
million people from their homes. Although most of the flood IDPs returned to their home areas
soon after floodwaters receded, most were living in the open for prolonged periods as over 1.9
million houses were damaged or destroyed. The crisis in 2010 was only the latest in Pakistan. It
had been preceded by dislocation of population following clashes between rival militant and
sectarian groups in the tribal areas; military operations against extremist militants and
insurgents; generalized violence and violations of human rights; a devastating earthquake in
2005; and annual floods across the country. Structural poverty, inappropriate development,
rapid urbanization, inadequate infrastructures, increased deterioration of the environment -
deforestation, increased human settlements in hazard prone areas, etc. have increased the
vulnerability to disasters. Flooding recurred in 2011 and 2012, which combined with incomplete
recovery in the 2010 flood-affected districts. Thus, even a moderate flooding could have
devastating effects in the future.
2.Overview of the ProjectAn ACF DRM assessment in 2010-11 showed that community and provincial DRM plans do not
exist (except for symbolic or outdated documents). It identified a lack of awareness amongst
stakeholders concerning DRM, and thus a need for capacity building. It also recognized a need
for structural mitigation to reinforce infrastructure against the damage caused by a disaster, a
need for improved management of agricultural land as well as a need for community level DRM
activities. Thus, ACF carried out a DRM project funded by the Royal Norwegian Embassy during
2011-12 to reduce morbidity and mortality risks by improving community resilience to natural
disasters. With a focus towards infrastructure that is highly affected by disasters, yet vital after a
disaster occurs, as well as early warning systems which are imperative for reducing loss of lifeand environmental degradation stemming from natural hazards, ACF aimed to develop local
capacity and introduce risk mitigation in the intervention areas. The project was a multi-
disciplinary initiative that aimed to involve a variety of stakeholders including ministries, local
authorities, disaster management agencies, NGOs, the academic sector and communities. The
direct beneficiaries of the project were the vulnerable populations affected by natural and man-
made disasters in Nowsherra, Charsadda, Mardan and Lower Dir regions of KPK province in
C
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
6/34
ACF Pakistan RNE DRM Project Evaluation, 2012
5
northern Pakistan which are vulnerable to both serious floods and earthquakes. The basic
premise of the project was that communities generally understand local realities and contexts
better than outsiders. If at-risk groups are involved in all stages of the disaster risk
management, a stronger foundation will be created for the development of sustainable
programs for risk reduction. Through the use of participatory techniques for data collection and
analysis, hazard analysis, trainings, ACF aimed to ensure full participation.
3.Project Details
Project General Objective
To reduce vulnerability and improve resilience to disasters in four districts of KPK, Pakistan
Specific Objectives
To enable communities to prepare for and mitigate the impacts of disasters using participatory
methods aimed at strengthening the self-reliance capacities of the population and reinforcing
or adapting infrastructure
Results
R.1- The ability to recognize hazard events and to cope with their effects is improved
R.2- The capacity to reduce the effect of hazards on communities and services is improved
R.3- The understanding/application of DRM at community and institutional level is improved
Programme Activities
R.1- The ability to recognize hazard events and to cope with their effects is improved
80% of intervention communities have functional local early warning systems
80% of target community can claim access to contingency stocks
90% of target communities are covered by disaster management plans.
R.2- The capacity to reduce the effect of hazard event on communities is improved
Facilities serving 60 communities are structurally strengthened to resist hazard events
90% of intervention communities have identified and equipped evacuation sites
R.3-The understanding/application of DRM at community/institutional level is improved
80% of intervention communities have disaster management units
240 individuals from local or district authorities receive DRM training
25500 individuals in the communities receive DRM Training
Specific Activities
Installation of Early Warning Systems in 48 Villages
Formation of 48 early warning groups in the community
Training of 48 early warning groups in the community
Development of emergency response plans and contingency stocks in 48 villages
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
7/34
ACF Pakistan RNE DRM Project Evaluation, 2012
6
Development of disaster preparedness coping mechanisms in 54 villages
Conducting of PCVA assessments/community risk and vulnerability mapping
Preparation of village disaster management plan in 54 villages
Signing Memorandum of understanding with 48 communities
Rehabilitation of 14 Flood affected small scale irrigation channels for agriculture
Construction of 5 Retaining and Flood protection walls
Rehabilitation of 13 hazard proof water supply schemes
water quality testing (Bacteriological and chemical testing)
Rehabilitation/reconstruction of water and sanitation facilities at evacuation centers
Rehabilitation of culverts at 15 evacuation routes
Rehabilitation of 13 evacuation roads
Formation of 48 Village Disaster Management Units within the community
Training of 48 DMU's on CBDRM
Training and sensitization of 240 local authorities and local institutions in CBDRM
Development of Contingency Stock at District Level
Community training on CBDRM
Awareness sessions on DRM at Boys & Girl Schools
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
8/34
ACF Pakistan RNE DRM Project Evaluation, 2012
7
HAPTER
2
: METHODOLOGY
At the end of 2012, ACF commissioned an evaluation of the RNE funded
DRM project activities in Khyber Pakhtunkhwa (KP). ACF subscribes to the
Development Assistance Committee (DAC) criteria for evaluation: Impact,Sustainability, Coherence, Coverage, Relevance / Appropriateness,
Effectiveness and Efficiency. ACF also promotes systematic analysis of the monitoring system
and cross cutting issues (gender, HIV/AIDS etc). These dimensions served as the main
evaluation criteria.
1.Evaluation QuestionsThe detailed evaluation questions are provided in chapter 3 and were grouped as follows:
Targeting
Project DesignProject Implementation
Project Monitoring
Project Sustainability
Cross cutting Issues
2.Evaluation ApproachReview of Relevant Documents
Prior to the field work, the evaluator reviewed the relevant documents, including those related to
this project, ACFs general work in Pakistan and DRM globally and documents related to DRM
and emergencies in Pakistan in order to gain a better understanding of the project and itscontext and to help develop the evaluation methodology and instruments. Sufficient
documentation was available.
Initial Briefing
Prior to the field work, the evaluator participated in a teleconference with ACF New York HQ.
Indirect Information
Interviews were conducted with local authorities in Lower Dir, Mardan and Charsadda; project
expatriate and national staff in Islamabad and the districts; donor representatives and the
Director CDPM University of Peshawar.
Field activities
Household interviews with community members and Focus Group Discussions (FGDs) with
village committee members based on participatory evaluation methods were utilized. The
villages visited were selected randomly from lists provided by ACF while the community
members for household interviews were selected based on convenience sampling. Random
sampling for household interviews was not attempted since the project provided community-
C
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
9/34
ACF Pakistan RNE DRM Project Evaluation, 2012
8
level services rather than individual household level services and as such there were no project
beneficiary lists from which people could be randomly selected. In each village, transect walks
were also conducted to the source of disaster vulnerability of each village (e.g., rivers near the
village) to get a better understanding of the disaster profile of the village and the manner in
which ACF work addressed the source of vulnerability. During the transect walks, the hardware
work constructed by ACF in each village were also visited to get a better understanding of themanner in which ACF work addressed the source of vulnerability of the village.
Village Sampling Details
Villages FGDs HH interviews
Charsadda 5 5 60
Mardan 3 3 36
Nowshera 5 5 60
Lower Dir 5 5 48
Total 18 18 204
3.Evaluation ConstraintsA major evaluation constraint was the budget available for the evaluation which meant that while
the evaluation was conducted by an external evaluator, he was assisted in interviewing females
by two ACF female staff members from other ACF projects. In order to analyze the impact of the
participation of ACF staff in the data collection on the neutrality and objectivity of the findings,
the field data is presented separately by males (interviewed by external evaluator) and females
(interviewed by ACF staff). As seen later in chapter 3, except for one section of the household
questionnaire, the differences in the responses given to the external evaluator and internal ACF
staff were not found to be unusually different. However, for the future, it would be advisable for
ACF to recruit independent females for evaluations wherever possible even though the external
evaluator did accept the ACF proposition in this case. The budget implications of this strategy
could be managed by reducing the number of field visit days, which the external evaluator found
to be on the high side compared with other evaluations that he has conducted. Thus, the
additional villages visited did not necessarily add to the richness of the data collected and the
number of days in each district could easily have been reduced by one without affecting the
quality of data collection. Another constraint was the security situation in Pakistan as a result of
which the evaluator could not visit Peshawar due to the sudden deterioration of the security
situation during the evaluation. Field work on two particular days was also scaled back due to
security incidents. However, given the fact that the external evaluator is originally from Pakistanand speaks the national language, the impact of security incidents on the evaluation activities
was minimized. Finally, timing was also a constraint as the evaluation was conducted near the
end of the year when some of the staff members were on leave, ACF was also focused on
finishing project activities and the Project Manager had finished his contract. However, despite
all these constraints, the evaluator succeeded in collecting adequate amounts of quality
information, mainly because of his strong background in conducing evaluations in Pakistan.
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
10/34
ACF Pakistan RNE DRM Project Evaluation, 2012
9
HAPTER3: FINDINGSThis chapter provides the findings according to each question included in the
TORs under the headings mentioned in the last chapter. However, the
sequence of the questions and their placement under the headings has been
changed in some places to enhance readability. The findings under eachquestion are based on triangulation of information from the various sources of information
mentioned in the last chapter.
1.
Targeting
This section focuses on the following questions
a) Did the communities selected for the project classify as being most vulnerable andhighly exposed to disaster risk, and did they meet the criteria established in theproject design? Could the PCVA process have been in any way improved to better
achieve this?The evaluator visited 18 of the 60 villages targeted by the ACF for this project. All the villages in
all four districts were found to be vulnerable and those in Charsadda and Nowshera were also
among the most vulnerable villages in KP province highly exposed to disaster risk in the opinion
of the evaluator based on the numerous evaluations that he has conducted in Pakistan after the
2010 floods. The worst-affected villages were those where the floods destroyed most houses in
the village and most of the families in the villages were displaced for weeks or months, finally
returning to live in their villages in tents or makeshift houses for prolonged periods until they
could construct or received houses. Thus, the difference across the districts can be clearly seen
in the response of individual households to the following questions across the four districts:
Difference in disaster vulnerability across ACF districts
Charsadda Nowshera Mardan L. Dir
When was your village last affected by a major
disaster? (% identifying 2010 floods)
100% 100% 100% 100%
Was your family displaced in the disaster?
(% responding yes)
96% 92% 62% 10%
If yes, for how long? (no. of months) 2.1 1.2 0.7 0.4
Was your house damaged during this disaster?(% responding yes)
96% 92% 60% 40%
In all districts, the 2010 floods were identified by people as the most devastating disaster inliving history. FGDs revealed that the 2010 floods were viewed as more damaging than the
2005 earthquakes in the villages which experienced both mega-disasters. However, there is
significant difference in the damages experienced across the four districts. While 90%+ of the
families were displaced in Charsadda and Nowshera were displaced during the floods, only
10% of the families in Dir were displaced while 62% were displaced in Mardan. The period of
displacement ranged from a high of 2.1 months in Charsadda to 0.4 months in Dir. Finally, only
C
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
11/34
ACF Pakistan RNE DRM Project Evaluation, 2012
10
40% of the families reported house damage in Dir compared with 90%+ in Charsadda and
Nowshera. Furthermore, FGDs revealed that even where houses were damaged in Dir, the
damage was relatively minor, consisting of cracks or a wall partially collapsed compared with
whole houses being swept away in Charsadda and Nowshera. However, villages in Dir did
suffer crop and infrastructure damage. Even so, it would be useful to conduct more thorough
assessments in Dir to see if there are more vulnerable areas for future work. Thus, the evaluatorwould recommend the adoption of formal and clear targeting criteria so that the most vulnerable
villages can be targeted more consistently across all project areas. ACF currently has targeting
criteria which include need, direct access, absence of other agencies and security. However,
there is a need to operationalize the need criteria further for the KP context. The following
sections provide the basis for developing such operationalized needs criteria.
International NGOs can generally intervene with emergency response only in cases of major
disasters because of the shortage of funds and based on the premise that communities and
local authorities can deal with minor emergencies on their own. Thus, it is recommended that
even for DRM work, international NGOs should focus on natural hazards which are capable of
producing major disasters and on geographical areas where these natural hazards can be
expected to produce major emergencies with a view to reducing the needs for major emergency
response by INGOs subsequently as a result of this DRM work. Villagers and ACF staff in each
area mentioned a wide range of natural hazards to which communities are exposed, including
earthquakes, floods, snowfall, landslides, seasonal diseases and drought. However, it is also
clear that within this long list, it is only earthquakes, droughts and floods that are capable of
producing major disaster globally. Snowfalls, landslides and seasonal diseases almost never
produce large-scale disasters which would justify emergency response by international NGOs.
Moreover, even droughts are unlikely to produce major emergencies in the context of KP
province. Thus, the starting point of formal targeting criteria for ACF should be to concentrate on
areas which are highly prone to earthquakes and floods in KP. Even among these two, it isclearly floods which are more amenable to DRM work due to their more predictable nature,
especially DRM work by ACF given the fact that earthquake-proofing generally requires
structural improvements in shelter construction, which is not a core ACF sector. Thus, ACF
should primarily focus on areas which are vulnerable to major floods and then incorporate
earthquake DRM concerns if those flood-prone areas are also vulnerable to earthquakes.
With floods identified as the most important natural hazard to focus on for KP, the next task
would be to identify the most flood-prone areas in KP. Unfortunately, ACF is constrained in
doing so by the fact that the Pakistani government has yet not developed rigorous, technology-
based hazard mapping. Thus, it would have to rely on less rigorous sources of information. The
starting point for the future would be to look at where the major rivers flow in KP. This wouldimmediately reveal the fact that Nowshera and Charsadda host all three major rivers in KP
(Indus, Kabul and Swat), while Mardan hosts at most major drainage channels (e.g., the Kalpani
drainage channel) while Dir hosts a couple of smaller rivers. It is the location of rivers which
explains the vast differences in the responses above. All the villages in Charsadda and
Nowshera were on major rivers banks, those in Mardan were generally next to the Kalpani while
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
12/34
ACF Pakistan RNE DRM Project Evaluation, 2012
11
those in Dir were not near any major river or even drainage channels. Another source of
information for identifying the most vulnerable districts, UCs and village would be to look at
which areas incurred the most damage during the 2010 floods. Appendix A, which shows the
damaged caused by the 2010 floods in KP, reveals that Charsadda, Nowshera, Dir and Mardan
were number 1, 2, 8 and 20 among KPs 24 districts in terms of number of affected people.
More disaggregated information at the UC and village levels may be available with thegovernment. This district ranking reveals that while Charsadda and Nowshera are prime targets
for future DRM work, Mardan seems to be a low priority district since very few people were
affected in the first place. Clearly, Dir is a very isolated area and one which is probably more
poverty-stricken than the other three districts. Figures also reveal that it incurred a high degree
of loss. While Dir has a relatively large number of people officially recognized as being affected
(i.e., eligible for government assistance programs, such as the Watan cards), it seems that the
Dir team has not been able to focus on some of them due to security issues as some of the
areas identified by the Dir government authorities as highly affected (Lal Qila and Samarbagh)
were perceived by the Dir team as unsecure. Thus, it would be useful to conduct more in-depth
needs and security assessments there for future work. Finally, the UNDPs one DRM project
has identified 30 most vulnerable districts in Pakistan. This list could also serve to guide ACFs
geographical targeting. Thus, the PCVA process could be strengthened by first weeding out the
less vulnerable villages by macro-level criteria and focusing the PCVA process on the most
needy villages to begin with.
Recommended needs criteria for future DRM geographical targeting
Basis for initial targeting should be natural hazard, with floods being the main priority
Subsequent targeting should be based on the districts, UCs and villages most flood-prone
with the following being the main sources of information:
-River and drainage channels mapping
-Damage incurred during 2010 floods
-UNDP One DRM project list of most vulnerable districts
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
13/34
ACF Pakistan RNE DRM Project Evaluation, 2012
12
Vulnerability Zones and ACF Project Sites in 4 KP Districts
The NGO Code of Conduct asserts that agencies will distribute assistance based on objective
needs. The evaluator strongly feels that NGOs should have strong information within their files
to demonstrate that they are doing so for each project. It would also be advisable for ACF to
ensure that it does so for each project. Currently, while the targeting is excellent in Charsadda
and Nowshera, even for those districts, the corresponding information to reveal that these really
are the most vulnerable villages is not immediately available in project files. Having a map
immediately available in its files which shows that all the villages in Nowshera and Charsadda
are next to major villages along with information about the damage caused by the 2010 floods in
various districts, UCs and villages will help ACF in clearly demonstrating its adherence to this
NGO code of conduct principle. Currently, ACF has the maps with its project areas shown nextto rivers in Nowshera and Charsadda (see maps below) but not the information about 2010
damages. As can be seen in the maps below, the project areas in Dir and Mardan are not next
to major rivers. Similarly, if some areas are being left out due to security reasons, as in Dir, it
would be good to have a formal security report by its professional security team and a
subsequent SMT decision to back up this decision. If some areas are being left out because
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
14/34
ACF Pakistan RNE DRM Project Evaluation, 2012
13
other agencies are already present there, it would be good to have on file minutes of meetings
of coordination bodies where such coordination among NGOs normally takes place.
b) Did the project strike the right balance between number of villages covered and theamount of work done in each village? If not would it have been better to cover morevillages with less work, or fewer villages with a greater amount of work?
While detailed appropriateness of the software and hardware work done by ACF will be
discussed in the next section (project design), in order to answer this question, it can be
asserted briefly here that the evaluator finds the particular mix of such activities to be highly
appropriate and well-selected, which means that it would not be advisable to drop any of them
in order to increase the number of villages. At the same time, given Pakistans large size and
high vulnerability, the number of most vulnerable villages is very high compared with the 60
villages covered in this project and probably runs into hundreds if not thousands. At the same
time, the number of agencies undertaking DRM work is small. Thus, there is a clear need to
increase the number of villages from the present 60 in a future similar project without completely
dropping any category or even sub-category of work. This obviously poses a dilemma. As faras the evaluator can see, the main bottleneck to increasing the number of villages is the number
of front-line field staff who work at the village level. Fortunately, such staff does no cost much
and an additional staff member could perhaps be hired for as little as 12,000-15,000 dollars per
year, which could easily be absorbed by some minor reconfiguration of the overall budget. The
other bottleneck is the cost of hardware work which is much higher. However, the evaluator also
noticed that in some villages more than one hardware activity was being implemented. By
keeping hardware activity down to one activity per village, it may be possible to cover more
villages. Also, ACF could overcome the hardware-related constraint and increase the number of
villages by trying to leverage its work by getting the government and other NGOs to cover some
of the hardware activities which may be beyondACFs budget.
2.Project Designa) Were the objectives, results and activities selected in the implementation of this
program relevant for the identified population needs? Was the participatory approachused appropriate an effective in identifying and selecting specific interventions? Wasthere sufficient consultation and participation of beneficiaries in the design, targeting,implementation and post-implementation stages? Did the activities match with whatpeople wanted?
The starting point of ACFs work for this project is the Participatory Community Vulnerability
Analysis (PCVA) exercise which aims to view each villages vulnerability and possible steps to
reduce it through the eyes of the community by using participatory methods to understand themain hazards, community strengths and weaknesses, and possible remedial measures for
strengthening the communitys resilience. The use of such an approach has helped ACF to
adopt a highly participatory process and identify the main components of the project in light of
community perspectives. While many of the project components, such as early warning
systems, contingency stocks and village plans, are obviously based on industry standards for
DRM work, their specific form in different villages has depended on community perspectives. To
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
15/34
ACF Pakistan RNE DRM Project Evaluation, 2012
14
begin with, the identification and development of the village hazard profile and community
resilience and weaknesses is largely based on community perspectives. Furthermore, the
village committees, which were the main lynchpin for project implementation, were selected by
gathering villagers and having them select responsible, trustworthy people who represented all
segments of the community. Furthermore, the village disaster plans were based largely on
community perspectives. The specific contents of the early warning systems, evacuation sitesand contingency stocks were based on community participation. Finally, the hardware work
done in villages was largely selected by villagers within the constraints of the project budget.
b) What are the main problems (negative or unexpected impacts from the interventions)that the population identified concerning the implementation of the activities andwhat solutions were identified by the project team?
The responses to the following questions reveal the high degree to which communities felt
involved in the project. Satisfaction was very high across all districts, though slightly lower in Dir,
and both sexes. The main problems identified by the committees related to the difficulty in
covering the whole communities with DRM awarenessraising given the large sizes of some of
the villages. A couple of villages also mentioned that ACF staff sometimes came late tomeetings, a problem created often due to the security situation in the area because of which
ACF security staff asked ACF program staff not to proceed until the security situation cleared.
However, there were no complains with the basic contents and approach of the program.
Community satisfaction with ACF project approach
Percentage saying Yes Char-
sada
Now-
shera
Mar-
dan
Lwr
Dir
Male Fem
ale
Were you consulted about the types of services and
their content?
98 96 92 84 90 95
Did agency staff always treat you according to yourcultural norms? 95 95 97 96 95 99
Were services provided keeping in mind your work
schedule?
96 97 97 88 95 94
Were the services timely for you in light of your
needs?
97 96 97 81 99 92
c) Did the project strike a good balance between hardware and software? If not, definewhat the balance should have been? Was the scope of hardware activities sufficient?If not, what other activities could have been added? Was the scope of softwareactivities sufficient? If not, what other activities could have been added?
The main hardware activities within the project consisted of the provision of contingency stock
for evacuation, rescue and relief activities and the construction of latrines, evacuation paths,
drainage and irrigation channels and flood protection walls to reduce the risk of disasters. The
main software activities were the setting up and training of village committees, development of
village disaster plans and early warning systems, awareness-raising about DRM among
communities and schools, training on DRM for construction workers and training of government
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
16/34
ACF Pakistan RNE DRM Project Evaluation, 2012
15
officials. These activities can be better compared with the help of a conceptual framework to
evaluate and compare the value and impact of different DRM activities. As shown in the table
below, DRM activities can be classified along a continuum consisting of six categories with each
higher category representing a more structural and stronger form of disaster risk reduction than
those below it. Prevention activities include efforts to reduce the occurrence of destructive
hazards, e.g., through reforestation and riverbed dredging, and are obviously the most effectiveform of DRM. However, such activities are generally done by governments rather than NGOs.
Given that even with high prevention efforts, there will still be some hazards occurrence, the
next best option is mitigation, i.e., redirecting hazards away from communities through flood
protection walls, drainage channels etc. Given that even with high mitigation efforts, there will
still be some hazards reaching communities, the next option is to move communities out of the
way of the hazards through evacuation before the hazard reaches them. The next option is
rescue efforts for those who could not be evacuated in time. Finally, the provision of timely relief
activities to both evacuated and rescued people reduces the risk of epidemics. In addition,
building the capacities of communities can also enhance their resilience to disasters.
Continuum of DRM Activities
DRM categories Purpose Examples Related ACF
activities
Prevention To reduce the
occurrence of
destructive hazards
Global climate change
advocacy; riverbed
dredging; reforestation
None
Mitigation To build buffers
between hazards and
communities
Drainage channels,
flood protection walls
Drainage channels,
flood protection walls
Communitycapacity-building To enhance theresilience of
communities by
strengthening their
socio-economic status
Livelihoods work,nutrition work, CBO
strengthening
Irrigation channels;VDMUs; community
awareness-raising;
Village disaster plans
Evacuation
preparedness
To help communities
escape approaching
disasters
Early warning systems;
evacuation paths and
sites;
EWS, evacuation
paths and sites
selection, contingency
stock items (e.g.,
megaphones)
Rescue
preparedness
To help rescue people
caught by hazard
Rescue tool and
equipment stocks
Contingency stock
items (e.g., life
jackets)
Relief preparedness To provide life-saving
services to evacuated
and rescued people
Watsan, health,
shelter, NFIs stocks
Evacuation center
latrines; contingency
stock items (e.g.,
water jerry cans)
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
17/34
ACF Pakistan RNE DRM Project Evaluation, 2012
16
With the understandable exception of prevention activities, which are generally out of the range
of NGO technical and financial capacities, ACF work is covering all the remaining categories of
DRM work. This reflects the comprehensive nature of the DRM package that ACF is providing.
On the software side, activities like village committee and disaster plan development, early
warning system and evacuation site identification do not cost much and can also be seen asmandatory components of DRM work. Thus, those activities should all be retained. The real
issue is about hardware activities since they cost much more. The above framework can help in
making decisions about specific hardware work. Thus, the objective in each village should be to
select activities which fall in the highest possible category above while being relevant to the
village and affordable within the ACF budget. So for example, flood protection walls, drainage
channels and irrigation channels are much more durable forms of DRM work than activities such
as evacuation paths and latrines in evacuation centers since the former reduce the chances of
communities getting affected by disasters in the first place while the latter are about helping
communities once they have been struck by hazards. Flood protection walls tend to be
expensive and should preferably be built where benefiting large or several villages os that their
per beneficiary cost is closer to other mitigation activities. More specifically, the evacuation
paths built will not be helpful in major disasters as they will become fully submerged. Within
community capacity-building, especially those related to livelihoods strengthening, activities
which benefit the whole community, such as irrigation channels, are more beneficial than
activities which target individual families, such as provision of agricultural inputs to individual
households. Finally, ACFs contingency stocks packages are also a very useful input even
though they target the lower categories for the simple reason that they have a low cost but still
cover three DRM categories (evacuation, rescue and relief categories). In summary, the
software activities and contingency stocks should be mandatory components given their low
cost and wide-ranging coverage. Among the remaining hardware activities, those targeting
mitigation and community capacity-building are more useful than evacuation paths and latrines.That said, it is recognized that the latter may be the best affordable option for some villages.
3.
PROJECT IMPLEMENTATION
a) Evaluate the impact of the DRM training and disaster plans e.g. impact on capacity oftrained population to be better prepared through the training and preparedness plans.Evaluate the effectiveness of the Early Warning system that was put in place.
DRM village committee and training: In all the villages visited in the four districts, the
evaluators found that both men and women village DRM committees had been formed and hadreceived training from ACF on DRM activities. The village committees seemed representatives
of all sections of the community in each village and seemed highly engaged in the DRM work
initiated by ACF. As a result of their DRM training, the committee members generally seemed
familiar with DRM concepts and fully geared up to play an active role in keeping their villages
safer during future disasters. They were highly appreciative of the DRM training and seemed
well aware about things like the values of early warning system, contingency stock and
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
18/34
ACF Pakistan RNE DRM Project Evaluation, 2012
17
evacuation sites. The committee members demonstrated ownership about the work done by
ACF and expressed a willingness to continue it after ACFs withdrawal.
...We can vouch that ACF has
done excellent work with us,better than any other NGO or
government agency and we
are very happy with its work
and find no problems in it.
Committee members in a
Charsadda village
Village disaster and contingency plans: Village disaster plans were seen hanging on the
walls in all the villages visited in Mardan, Charsadda and Nowshera although there was some
variation in the level of details contained by the plans across the villages. Thus, it would be
useful for ACF to develop a clear outline for the topics to be covered by such plans and ensure
that all villages complete all the topics. In Dir, the team had not yet engaged the villagers in any
village to complete such plans by the time of the evaluation which was undertaken during the
last two weeks of the project. The Dir team felt that they would still be able to complete the
plans in all the villages by December 31. However, clearly, even if they did so, it would seem to
be a rush, last moment job with little follow-up possible, which would affect the quality of the
whole exercise.
Early warning system: The early warning systems planned by ACF include linking the village
committees with governmental sources of information (by giving both parties the telephone
numbers of each other) about approaching hazards and also providing them with equipment,
such as megaphones and battery-operated loudspeakers for communicating the information
received from external sources within the villages. Most of the village committees had been
provided with the telephone numbers of relevant local authorities, such as police stations and
Civil Defence offices. However, the village committees had not been registered with government
departments so that those departments could proactively provide information to villagers. This
issue will be discussed in more detail under the sustainability section. Finally, due to some
delays in procurement, the equipment for intra-village dissemination of early warning systems
was being delivered in villages around the time of the evaluation. However, even so, such
systems would still be up and running before the 2013 flood season in July-August. However,
the success of ACFs EWS will of course depend on villages getting timely and accurate early
warning from government departments. Unfortunately, the technical and management capacity
of governments departments to provide such information is very low in Pakistan at the moment,
which is a major challenge for ACF. The final chapter will look at this issue in more detail.
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
19/34
ACF Pakistan RNE DRM Project Evaluation, 2012
18
Evacuation sites selection: In almost all villages, the committees had selected evacuation
sites in the village. Those sites generally consisted of schools, mosques or high, open places
within the village which were already being used as evacuation sites by villagers. In many of
these sites, ACF has built latrines. However, it would also be useful for ACF to help each village
select evacuation sites outside the village as well in case the floods are so severe that they
threaten to submerge the whole village, including the internal evacuation sites.
DRM construction training: These trainings had not been delivered yet in any district by the
time of the evaluation because of the delays in the project caused by late government
permission and procurement delays. According to subsequent information sent by ACF, these
trainings were completed before the end of the project.
Community DRM awareness-raising: These 3-hour sessions were supposed to be carried
out by the village committee members once they were trained by ACF. Such sessions were held
by committee in many villages. However, their efficacy was affected by two major reasons.
Firstly, in most of the villages, the committee members and in fact people in general did not
have the immediate capacity to become good trainers based on the 3-day training that they
received. Secondly, the population in some of the villages ran up to 3-4,000 persons. Thus,
assuming that in one session, the committee trained up to 100 villagers, it would still take it
around 20-25 sessions to train all adults and children above the age of 5. This obviously is not
an easy task for people who have full-time vocations and only work as volunteers on the
committee. These community trainings were supposed to be the main conduit for imparting
knowledge to the community members on DRM concepts and about the village disaster plans,
early warning systems, contingency stocks etc. As a result of these constraints, understanding
within the larger community about DRM concepts and the village DRM activities was found to be
variable across villages, districts and gender as shown in the table below which is based on
household interviews. The table reveals that while familiarity with the village committees and thehardware work is very high (90+) across all districts and both sexes. However, familiarity within
the general community is much lower and highly uneven across the districts and sexes.
Familiarity in Dir is lower than in the other three districts. The most notable difference is across
the two sexes, with women reporting much greater familiarity with project activities than men,
which in some ways is counter-intuitive since usually Pakistani men have more opportunity to
participate in project activities due to the cultural conservatism. This difference may partially be
due to the fact that women are more village-bound and hence easier to attract to awareness-
raising sessions and meetings. However, the fact that women were interviewed by internal ACF
staff while men were interviewed by the external evaluator could also have been a reason for
the higher positive responses reported by women. In any case, even if women are as well
informed as reported below, increasing the awareness among men would be crucial since
women enjoy much less freedom of movement and initiative in Pakistan during disasters and in
normal times. As mentioned earlier, spreading awareness across such large, illiterate and busy
communities is an enormous structural challenge faced not only by ACF but all agencies. Some
related recommendations are provided in the last chapter.
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
20/34
ACF Pakistan RNE DRM Project Evaluation, 2012
19
Community familiarity with ACF DRM project components
Are you familiar with:
(Percentage responding yes)
Char-
sadda
Now-
shera
Mar-
dan
Lwr
Dir
Male Female
1. DRM village committee 98 98 97 95 94 100
2. Early warning system 75 61 73 50 38 94
3. Village disaster plan 94 74 73 30 48 85
4. DRM contingency stock 64 58 56 2 21 68
5. Evacuation sites 89 85 61 52 55 100
6. DRM training 95 91 71 73 69 94
7. DRM construction work 95 98 85 93 95 92
Overall, the communities generally felt that the project activities have benefited them in a variety
of ways, including increasing their self-reliance, ability to deal with crisis and physical dangers,security of property knowledge about DRM and access to water and sanitation facilities. The
differences in responses across the four districts and the two sexes were minor with respect to
these issues. Communities mainly perceived these improvements due to their familiarity with the
establishment of the village committee, which they felt was a huge boon for the whole village,
and the visible DRM construction work that had occurred in all the villages and to a lesser extent
the other software activities with which their familiarity was lower as seen earlier.
Community perceptions about project impact
Has ACF DRM work
increased your:
(Percentage responding yes)
Char-
sadda
Now-
shera
Mar-
dan
Lwr
Dir
Male Female
Ability to stand on own feet 96 98 97 92 94 98
Ability to deal with crisis 98 98 100 90 94 98
Safety from physical dangers 98 98 97 78 94 91
Security of property 98 95 97 80 93 90
Knowledge about DRM 92 88 78 75 69 100
Access to water and
sanitation services
96 98 97 95 95 98
4.
Project Monitoring
a) How suitable and effective were the M&E systems in place through the duration of theproject and how could these have been improved? What systems were put in place toensure that outputs provided were of the highest quality possible and wereacceptable to beneficiaries?
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
21/34
ACF Pakistan RNE DRM Project Evaluation, 2012
20
ACF staff is regularly monitoring field activities, starting from field-level staff all the way up to the
Country Director. However, it would be useful for ACF to have formal monitoring plans for each
project which clearly specify the role for all relevant staff, starting from field-level staff all the way
up to the Country Director in the projects monitoring. This would include the frequency,
modality, project dimension and indicators for each relevant ACF staff. More importantly, the
plan would also specify how the information coming from the monitoring done by each staff willbe analyzed and summarized (preferably through a software-based monitoring application) and
then fed to progressively higher levels of the ACF Pakistan team, terminating with the Country
Coordination Team and what decisions and follow-up each level would undertake to ensure that
the project achieves its objectives on time. In this way, the team could constantly monitor the
progress on the common program dimensions (e.g., effectiveness, efficiency etc) that eventually
will be part of the project evaluation TORs (instead of being informed about shortcomings by
external evaluators) and take remedial timely action to ensure high project quality. While project
monitoring is clearly occurring, it is also true that there have been schedule slippages along
some project components, especially in Dir. The presence of such a formal plan would have
made it much easier to ensure timely achievement of objectives.
Another missing piece in ensuring quality and timely achievement of objectives is the institution
of strong accountability measures, which can enhance the ability of the ACF team to gain
information about project achievement beyond that coming from its own physical monitoring.
ACF Pakistan has now set up a Performance, Quality and Accountability unit in Islamabad and
plans to incorporate accountability mechanisms in future projects.
5.Project Sustainability
a) Were the construction material and masonry techniques applied appropriate (werethey disaster-resilient?) What alternatives would have been better?
Since there was no technical person on the evaluation team, the analysis of technical
construction quality undertaken as part of the evaluation was limited and was based on three
dimensions. Firstly, the external evaluator observed the construction quality visually for the
following common construction problems in all villages and only minor, occasional problems
were found:
Are there any cracks or plaster falls in the construction?
Have the fixtures and fittings (e.g., doors and windows) been fixed properly?
Dampness on walls/floor?Is the toilet design ok?
Does the foundation look straight?
Are any hinges and bolts loose?
Secondly, the evaluator reviewed the construction-related documents. ACF has qualified
engineers within its DRM teams who had developed detailed Bills of Quantities for each
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
22/34
ACF Pakistan RNE DRM Project Evaluation, 2012
21
construction work. The evaluator requested the ACF team to share sample documentation for
some sites to show whether the BOQ material was actually delivered and used in the
construction and the required documentation from Nowshera, Charsadda and Mardan was
reviewed and found to be in order. Finally, the evaluator also checked with village committee
members about the technical quality of work who in some cases did have some background in
the small-scale construction work that ACF undertook as part of this project and they expresseda high degree of satisfaction. Thus, overall, the construction quality seems sturdy, disaster-
resilient and best possible option under given conditions, though it would have been better for a
qualified technical person to physically observe the work and review construction documents. It
would also be good for ACF to undertake standard construction tests, e.g., brick quality, wood
quality, sand quality and retained weight of crush tests.
b) Evaluate whether the maintenance requirements of the structures were within the
capacity of the beneficiary users, if ACF did enough to reinforce these local
capacities. Evaluate the willingness of communities to maintain the infrastructure and
propose how community ownership could have been improved or reinforced.
The main mechanism used by ACF to ensure maintenance of both the software and hardware
work is the setting up of village committees. While in a few villages, the committees already
existed, in most villages the committees set up by ACF were the first experience of communities
with such committees. Committee members seemed motivated and keen to maintain the
committees even after ACFs withdrawal while communities saw the committees as an
extremely helpful form of social capital developed for them by ACF. So, the chances seem high
that most committees will survive even after the withdrawal of ACF and will maintain the
software and hardware work. However, given the high migration in some areas in KP, turnover
within committees could be high and it would be a test of their endurance whether villagers keep
refilling the vacancies. In this regard, some additional steps by ACF could increase the chancesof committee continuation. Firstly, while ACF has trained committees on DRM issues, it would
also be useful to give them some basic training on management, CBO operations and
development issues. Second, it would also be advisable for ACF to see whether there are other
international or local NGOs working long-term in these areas which could adopt the committees
after ACFs withdrawal. Third, it would also be helpful for ACF to link committees of nearby
villages with each other for mutual support, information sharing and learning during floods and
even normal times. Finally, it would be important to link these committees more strongly with
government structures, as discussed in the next section.
c) Evaluate whether links between VDMUs and authorities are sufficiently reinforced.
Evaluate the impact of the DRM training for local and district authorities. What werethe main issues/weaknesses and how could this have been improved? Were the local
disaster authorities (PDMA and DDMU) sufficiently involved at all levels of the
project? If not, in what aspects of the project could their involvement have been
improved?
ACF organized DRM trainings from relevant district-level departments in all four districts though
some of the training in Mardan was still outstanding at the time of the evaluation. The evaluator
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
23/34
ACF Pakistan RNE DRM Project Evaluation, 2012
22
interviewed district staff in three of the districts (except Nowshera) and all of them expressed a
high degree of appreciation and satisfaction with the training. With respect to involvement of
disaster authorities, all NGOs have to take official permissions from government authorities and
to that extent the government authorities were involved in the project. However, government
departments in Pakistan tend to be highly bureaucratic and it is not easy to get them involved to
any greater extent, beyond granting permissions, in NGO projects. Nevertheless, suchinvolvement would be crucial for the success of ACFs DRM project more so than for other
watsan or food security projects. Viewed so, it is felt that ACF should have built upon the good
will created by the government trainings and expended greater effort in increasing the
involvement of government departments in the project and enhancing the linkages between
them and its village committees in various ways despite the bureaucratic nature of Pakistani
government departments. To begin with, ACF could have got its village committees registered
with the Social Services Departments in district governments to enhance the chances of the
survival of these committees after ACFs withdrawal. Secondly, ACF could have advocated with
the DDMUs, preferably in coordination with other DRM NGOs) to treat the village committees
set up by ACF as valid structures for disseminating early warning information, and involving in
any government future grass-roots DRM initiatives. Thirdly, ACF could have followed up on the
trainings by trying to have some practical linkages between government departments and the
village committees, e.g., by arranging mock drills between the Civil Defence Departments
(which has grass-roots presence through a cadre of community-based volunteers), and
encouraging the government to initiate small-scale mitigation measures in ACF villages which it
could not afford itself.
6.Cross-Cutting Issues
a) How realistic and appropriate was the budget of the project? Was it respected?The table above provides information on the original budget, actual % expenditure by October
2012 and expected % expenditure by project end. This information reveals that almost all major
cost budget lines are being respected and will be fully utilized by the end of the project with an
overall burn rate of 97%. However, there were some budget issues due to the exchange rate
fluctuations given that three currencies (Nok, USD and PKR) were involved. Overall, ACF lost
budget of around $100,000 due to the depreciation of the USD against the Nok despite the fact
that the PKR has also depreciated significantly against the USD. According to the ACF Pakistan
Finance team, this loss was covered by a budget modification where district-level contingency
stock items were dropped. Overall, the Finance team felt that the DRM team had done good
budget management which helped in the budget being respected.
ACF DRM Project Budget utilization
DESCRIPTION
Budget
(Nok)
SPENT
oct 12)
Estimated
Spent , 31-12-
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
24/34
ACF Pakistan RNE DRM Project Evaluation, 2012
23
2012
Personnel 475,953 84% 98%
Expatriates 238,076 87% 98%
Local staff 237,877 82% 97%
Communication, Visibility, Information 4,900 4% 104%
Equipment Procurement 17,164 100% 100%
Equipment Hire 64,445 89% 108%
External Evaluation 20,000 0% 50%
Project Costs 597,098 76% 97%
Construction mitigation 458,165 83% 97%
Disaster preparedness in community 67,380 47% 125%
Capacity building 71,553 62% 69%
Other Costs 107,772 88% 97%
TOTAL DIRECT COSTS 1,287,333 80% 97%
Indirect Costs (9%) 115,860 80% 97%
TOTAL COSTS 1,403,193 80% 97%
b) Were ACFs rules on procurement adhered to? Have concerns relating to a zerotolerance on corruption policy been adequately observed in the projectimplementation?
ACF has a whistle-blower policy and an anti-corruption policy to reduce the chances of
corruption in the projects. Moreover, the approval limits of the Country Office and field offices
are much lower than that seen by the evaluator in other agencies with a view to reducing the
chances of corruption. The ACF Pakistan Finance team also diligently works to minimize thechances of corruption by having a zero-tolerance policy against corruption so much so that a
driver was fired for a few hundred rupees violation. The team also calls additional suppliers by
telephone even where the logistics and field teams have sent in the three mandatory quotations
in order to further minimize the chances of fraud. All these steps ensure strong financial due
diligence and the evaluator did not come across any reports of corruption within villages or field
offices. However, this due diligence did slow up procurement to some extent with some of the
delays mentioned earlier being related to delays in procurement, which in turn were also
exacerbated by the lack of clarity among staff on procurement rules. ACF held a team workshop
recently to smoothen out these issues.
c) What measures were taken to ensure a gender balance at all stages of the project
(assessment, design and implementation)? How could this process have been
improved?
KP is an extremely conservative environment even by Pakistani standards where womens
rights are low and it is very difficult for NGOs to work with women to enhance their rights due to
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
25/34
ACF Pakistan RNE DRM Project Evaluation, 2012
24
the significant threat of violent backlash by militants who in recent weeks have killed scores of
female teachers and aid workers in KP. Within this background, ACF has still succeeded in
involving women in the project. Thus, all villages had separate womens committees who were
also provided DRM training like the male committees. Overall, womens understanding of the
project activities seemed higher than that of men, as mentioned earlier under the Community
DRM awareness-raising section. In many villages in Dir (which is more conservative thanCharsadda etc), women had formed the committees without the knowledge of men who did not
approve of such activities. In some Dir villages, ACF did not work with women because it was
not given access to women by the community men. However, given the severely conservative
culture and the serious threat of violent backlash, there are severe limitations on the extent to
which the involvement of women could be increased by ACF or other NGOs. There are greater
opportunities to do so in the districts other than Dir. Thus, men in these villages admitted that
they initially had some misgivings about allowing women to participate in the project as they had
this impression that foreign NGOs spread vulgarity among women and encourage them to rebel
against the local cultures. However, they also said that these misgivings had been largely
removed after the start of the project due to its positive impact and the fact that they saw ACF
staff constantly respecting local cultural values..
d) Evaluate the added value to the project from the partnership with the Centre forDisaster Preparedness and Management (CDPM), and whether the involvement ofCDPM was maximised.
The CPDM is the only institution focused exclusively on DRM activities in Pakistan. ACF
requisitioned the CPDM to undertake the trainings for its own staff, the government departments
and the village committees. The trainings were highly appreciated by all the stakeholders and as
such the engagement with the CPDM seems like an excellent move by ACF to enhance and
work with local capacity. However, there could have been scope to increase the utilization of
CPDM services further since DRM is such a new area for ACF staff in Pakistan. While thetraining was certainly good, given the limited initial background of ACF staff in DRM work, it may
have been useful to involve the CPDM at the field level to a limited extent, e.g., by having them
undertake the software component in one model village and monitoring ACFs work in an
additional 1-2 villages initially and providing feedback. This suggestion is based on the
observation that the ACF staff, especially in Dir, needed more capacity and support in
implementing the software component which was very new to them.
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
26/34
ACF Pakistan RNE DRM Project Evaluation, 2012
25
HAPTER 4: CONCLUSIONS ANDRECOMMENDATIONSThis chapter summarizes the main findings and conclusions from the last
chapter and provides recommendations for enhancing the quality of the project
further in the future along each dimension of improvement identified in the
conclusions section. The emphasis is on providing recommendations which
would be feasible within the difficult working environment faced by ACF in Pakistan.
1.Conclusions
a) Did the program reach the intended results and objectives? If not, what were the mainconstraints faced by the organization and were they a consequence of poor quality of
the initial assessment, poor project design or poor implementation? Did the projectsufficiently identify and manage risk related to the project activities?
Overall project achiovements
Results and related indicators Status at time of
evaluation
The ability to recognize hazards and cope with them is improved
80% of communities have functional local early warning systems Work in progress at time
of evaluation; ACF later
confirmed completion by
end of project though
quality would have beenbetter with more spaced
out completion
80% of target community can claim access to contingency stocks
90% of communities are covered by disaster management plans.
The capacity to reduce the effect of hazards on communities is improved
Facilities serving 60 communities are structurally strengthened to
resist hazard events
Done
90% of communities have identified and equipped evacuation sites Identified everywhere but
not all equipped due to
government permission
issues
The understanding of DRM at community/institutional level is improved
80% of intervention communities have disaster management units 100% have units
240 individuals from local or district authorities receive DRM training Higher number trained
25500 individuals in the communities receive DRM Training Trained but variable
absorption of concepts
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
27/34
ACF Pakistan RNE DRM Project Evaluation, 2012
26
The table above provides the status on the three results and corresponding indicators at the
time of the evaluation field visits. Thus, the project is likely to achieve its objectives by
December 31, 2012 though the quality of the work could have been higher if so much work was
not squeezed into the last weeks of the project. These delays happened due to procurement
and government permission issues. Overall, the project is a very well-conceived one and aims
to fill a critical need in enhancing resilience within KP. Many components of the project havebeen implemented very well, such as the setting up of the village committees and their training
and the completion of the construction activities. However, other components did have room for
improvement. These shortcomings were caused by a wide range of external and internal
constraints faced by ACF, including high insecurity, governmental bureaucracy, shortage of
skilled staff, stringent procurement rules, low literacy levels within communities and the
conservative culture in the country. Among these, ACF has managed security risks well and has
comprehensive and efficient security systems in place. However, more attention should have
been devoted to dealing with the risks associated with governmental bureaucracy, shortage of
skilled staff (both quantity and skill level), stringent procurement rules, low literacy levels within
communities and the conservative culture in the country. While the communities did not identify
major areas of improvements (beyond the standard requests for more hardware services in the
future), the evaluator himself has identified some. The next section provides recommendations
for improving the quality of the project further in these areas in the future.
2.RecommendationsTargeting: Targeting of the most vulnerable is excellent in Charsadda and Nowshera but is
less accurate in Mardan and Dir. Mardan has structural issues as very few people were affected
during the 2010 floods. In Lower Dir, a large number of people were affected, so the problem is
more about ensuring more thoroughly that ACF is focusing on the most vulnerable areas. It is
recommended to undertake a more thorough needs and security assessment in Dir. In general,
ACF is advised to focus on districts and villages which are vulnerable to major floods. Toidentify such areas, the following three criteria are suggested: River and drainage channels
mapping; Damage incurred during 2010 floods and UNDP One DRM project list of most
vulnerable districts. Consider Sindh for DRM work too since it has recently become the most
repeatedly struck province in Pakistan.
Number of villages: Review the possibility of increasing the number of villages in the next
phase by increasing the number of field staff, covering each village with only one hardware
construction activity and linking with other NGOs and government agencies to cover some of
the construction work that ACF cannot afford in its budget.
Selection of hardware activities: Use the prevention-mitigation-capacity-evacuation-
rescue-relief continuum of DRM activities presented in Chapter 3 to select hardware activities.
The contingency stocks are an important hardware input given their relatively low cost and the
fact that they cover three categories in the continuum even though they are the three bottom-
most categories. Select other hardware construction activities from the highest possible
category of this continuum which is relevant and affordable. For example, wherever possible
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
28/34
ACF Pakistan RNE DRM Project Evaluation, 2012
27
prioritize drainage channels, irrigation channels and protection walls over evacuation routes and
latrines. Focus on community-level activities, .e.g., irrigation channels, rather than individual
household-level activities like provision of agricultural outputs given that the basic purpose of
the project is community joint work which may be undermined by helping certain families with
individual-level inputs and ignoring other families.
Village committee sustainability: Enhance the sustainability village committees by
providing them training on CBO management issues, having them adopted by other agencies
working long-term in the area (e.g., Concern/IRSP), linking nearby committees with each other
for mutual support and learning during crisis and normal times, getting committees registered
with the Departments of Social Welfare and getting them recognized by DDMAs as valid
structures for providing early warning information and inclusion in the governments DRM work
in coordination with other agencies.
Community DRM awareness-raising: Reaching to the larger community has proved
challenging due to the large size of the communities, busy schedules and low literacy levels.
This challenge could be reduced by having communities develop detailed community
awareness-raising schedules and following up more closely with committees about the
schedules and also looking at the possibility of training young educated, unemployed persons in
each UC to conduct sessions in all targeted villages in that UC for a small honorarium
Village disaster plans: Develop a clear formal outline of what each plan should contain and
then ensure that all villages closely follow and complete the plan according to this list to ensure
consistency across villages
Evacuation sites: Beyond the within-village evacuation sites identified now, also help
communities identify evacuation sites outside villages which could be accessed in case thewhole village gets submerged.
Monitoring and accountability: Develop formal monitoring plans which clearly specify the
role for all relevant staff in the projects monitoring, including the frequency, modality, project
dimension and indicators for each relevant ACF staff. More importantly, the plan would also
specify how the information coming from the monitoring done by each staff will be analyzed and
summarized (preferably through a software-based monitoring application) and then fed to
progressively higher levels of the ACF Pakistan team, terminating with the Country Coordination
Team and what decisions and follow-up each level would undertake to ensure that the project
achieves its objectives on time. The presence of such a formal plan would make it much easier
to ensure timely achievement of objectives and avoid slippages. Also institute strong and
responsive accountability measures in future projects.
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
29/34
ACF Pakistan RNE DRM Project Evaluation, 2012
28
Technical capacity/CPDM: Enhance technical inputs on DRM work by having CPDM provide
some degree of field-based support to ACF staff on software issues, e.g., by developing the
software package in one model village.
Government linkages: ACF should build upon the goodwill created by the government
trainings and expend greater effort in increasing the involvement of government departments inthe project. To begin with, ACF could get its village committees registered with the Social
Services Departments. Secondly, ACF should advocate with the DDMAs, preferably in
coordination with other DRM NGOs) to treat NGO village committees as valid structures for
disseminating early warning information, and involving in any government future grass-roots
DRM initiatives. Thirdly, ACF could try to have some practical linkages between government
departments and the village committees, e.g., by arranging mock drills between the Civil
Defense Departments, and encouraging the government to initiate small-scale mitigation
measures in ACF villages which it could not afford itself.
Coordination with other stakeholders on DRM: Having successfully completed one DRM
project, ACF should enhance its profile in this sector by increasing its visibility and inputs in
coordination sectors like the DRR Forum and the UNDP DRM project and obtaining greater
technical inputs from these forums.
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
30/34
ACF Pakistan RNE DRM Project Evaluation, 2012
29
APPENDIX
1.
Damage in KP from 2010 Floods
KPK - Flood Affected Population
S #
District
Total_Population
Total_UCs
Affected_UCs
Affected_Population
HH_Damage
Aftec_pop_pec
1
Abbottabad
1169904 51
0
2 Bannu 971930 49 49 54,473 7,782 6
3 Battagram 451340 20 9 10,416 1,488 2
4 Bunair 738496 27 24 5,614 802 1
5 Charsadda 1492939 49 34 502,732 71,819 34
6 Chitral 465075 24 12 69,164 9,881 15
7 D.I.Khan 1234706 47 26 394,608 56,373 32
8 Dir Lower 1131676 37 7 180,686 25,812 16
9 Dir Upper 753313 28 14 210,498 30,071 28
10 Hangu 459275 19 19 45,841 6,549 10
11
Haripur
1007960
45
42
56,646
8,092
6
12
Karak
622919 21
21
50,935
7,276
8
13 Kohat 774318 32 32 38,716 5,531 5
14 Kohistan 684004 38 38 464,333 66,333 68
15
Lakki
715139 33
26
28,092
4,013
4
16 Malakand 660046 28 6 45,086 6,441 7
17 Mansehra 1431208 59 12 22,870 3,267 2
18 Mardan 2123149 75 43 19,992 2,856 1
19 Nowshera 1174961 47 27 499,818 71,403 43
20 Peshawar 2860402 92 16 237,068 33,867 8
21 Shangla 632670 28 7 83,649 11,950 13
22
Swabi
1494583
56
11
15,389
2,198
1
23 Swat 1834756 65 42 634,654 90,665 35
24 Tank 349373 16 16 148,890 21,270 43
Grand Total 25234142 986 533 3820170 545739 16%
2.
Scoring Matrix
Criteria Rating (1
low, 5 high)
Rationale
Impact 4 The data in chapter 3 from household interviews clearlyshows that over 90% of the respondents felt that the project
helped improve their resilience due to the committees set upand the construction work. These aspects include:
Ability to stand on own feet
Ability to deal with crisis
Safety from physical dangers
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
31/34
ACF Pakistan RNE DRM Project Evaluation, 2012
30
Security of property
Knowledge about DRM
Access to water and sanitation services
However, impact could be increased further by enhancing
understanding within the larger community about otheraspects of the project through more effective communityawareness-raising sessions. As shown in chapter 3,understanding among men about other project componentslike EWS, contingency stopcock etc is variable.
Sustainability 3 The committees set up are the main mechanism forensuring project sustainability. However, there is scope toimprove sustainability further by:
linking communities with other agencies, governmentdepartments and with each other andthrough further training of the committees on CBOmanagement issues
Coherence 4.5 This is the strongest point of the project since thecombination of hardware and software activities coveralmost all categories of the DRM activities continuumpresented in chapter 3. Coherence could be improvedfurther by concentrating more on the higher categories ofthe continuum in terms of hardware construction work, e.g.,by doing more irrigation and drainage channels andprotection walls than evacuation routes and latrineswherever feasible. The project is also in line with thePakistan governments Disaster Management strategy andthe grass-roots DRM approach of the DEC-funded NGOs,
whose DRM project the evaluator reviewed recently.Coverage 3.5 The villages in Charsadda and Nowshera are all clearlyamong the most vulnerable villages in KP. However, there isa need to enhance the coverage of the most vulnerablevillages further.by using the following criteria:
Basis for initial targeting should be natural hazard, with
floods being the most priority natural hazard to focus on
Subsequent detailed targeting should be based on
collecting information about which districts, UCs and
villages are the most flood-prone with the following beingthe main sources of information:
-River and drainage channels mapping
-Damage incurred during 2010 floods
-UNDP One DRM project list of vulnerable districts
Relevance 4.0 Relevance is again a very strong point of the project giventhat Pakistan has become increasingly vulnerable to major
-
7/24/2019 Acf Pakistan Drm (Rne) Final Evaluation
32/34
ACF Pakistan RNE DRM Project Evaluation, 2012
31
disasters in recent years and very few agencies,government or NGO, are doing much active work in DRMthough some of the project areas, e.g., Mardan, are not veryhighly vulnerable.
Effectiveness 3.5 Some project components have been implemented highlyeffectively, including hardware construction work and manysoftware components in Nowshera, Charsadda and Mardan.However, there were also much delay in completing manyproject activities, especially in Dir which reduced theeffectiveness of the project.
Efficiency 4 ACF has managed to stay within budget limits till the endthrough efficient project financial management achievedthrough very close and regular coordination between theFinance department and the Project Manager. The budgetitself reflects efficiency as it increases the resilience of morethan 50,000 persons at relatively low cost.
3.AcronymsBOQ Bill of Quantities
CBDRM Community-based Disaster Risk Management
CBO Community-based Organization
CDPM Center for Disaster Prevention and Management
DAC Development Assistance Committee
DDMA District Disaster Management Authority
DRM Disaster Risk Management
EWS Early Warning System
FGD Focus Group DiscussionIDPs Internally Displaced Persons
KPK Khyber-Pakhtunkhwa Province
NDMA National Disaster Management Authority
PCVA Participatory Community Vulnerability Analysis
RNE Royal Norwegian Embassy
UC Union Council
UNDP United Nations Development Program
VDMU Village Disaster Management Unit
4.List of InterviewsNoor-Ul Amin District Disaster Management Officer, Lower DirAbigail Chatagnon ACF Field Coordinator KP- Paki