reducing vulnerability through disaster risk management

This report is commissioned by Action Against Hunger | ACF International. The comments contained herein reflect the opinions of the Evaluator only.

Photo © NIaz Murtaza

Reducing Vulnerability through Disaster Risk Management (DRM) in Khyber Pakhtunkhwa (KPK), Pakistan

External Evaluation

Funded by Royal Norwegian Embassy in Islamabad

Niaz Murtaza, January 2012

ACF Pakistan RNE DRM Project Evaluation, 2012

1

TABLE OF CONTENTS

Executive Summary 2

Chapter 1: Brief Background Emergency Context in Pakistan

Overview of the Project Project Details

4 4 4 5

Chapter 2: Methodology

Evaluation Questions Evaluation Approach Evaluation Constraints

7 7 7 8

Chapter 3: Findings Targeting Project Design Project Implementation Project Monitoring Project Sustainability Cross-Cutting Issues

9 9

13 16 19 20 22

Chapter 4: Conclusions and Recommendations Conclusions Recommendations

25 25 26

Appendix Damage in KP from 2010 Floods Scoring Matrix Acronyms List of interviews List of documents reviewed Good practice case study

29 29 29 31 31 32 32


2

XECUTIVE SUMMARY Over the last decade, Pakistan has experienced large-scale internal displacement

caused by a range of natural and human-made disasters. Structural poverty,

inappropriate development, rapid urbanization, inadequate infrastructures,

increased deterioration of the environment - deforestation, increased human

settlements in hazard prone areas, etc. have increased the vulnerability to disasters. Thus, even

a moderate flooding could have devastating effects in the future. An ACF DRM assessment

identified a lack of awareness amongst stakeholders concerning DRM, and thus a need for

capacity building. Thus, ACF carried out a DRM project funded by the Royal Norwegian

Embassy during 2011-12 to reduce morbidity and mortality risks by improving community

resilience to natural disasters in Nowsherra, Charsadda, Mardan and Lower Dir regions of KPK

province in northern Pakistan which are vulnerable to both serious floods and earthquakes.

At the end of 2012, ACF commissioned an evaluation to evaluate the impact and approach of

ACF’s RNE funded DRM project. ACF subscribes to the Development Assistance Committee

(DAC) criteria for evaluation: Impact, Sustainability, Coherence, Coverage, Relevance /

Appropriateness, Effectiveness and Efficiency. ACF also promotes systematic analysis of the

monitoring system and cross cutting issues (gender, HIV/AIDS etc). These dimensions served

as the main evaluation criteria. The tools included documents review, interviews with ACF

international and national staff, government officials and partner staff and household interviews

(with 204 community members), Focus Group discussions and transect walks in 18 villages.

Conclusions and Recommendations

Overall, the project is highly relevant to Pakistan given its high disaster vulnerability and is a

well-conceived one with an excellent and comprehensive mix of hardware and software

activities. Many of the project activities have also been implemented well though others do have

some room for improvement. The main findings can be summarized as follows: i) Targeting of

the most vulnerable is excellent in Charsadda and Nowshera but is less accurate in Mardan and

Dir; ii) ACF has set-up village committees to enhance the sustainability of project activities but

the committees must be linked adequately with other NGOs and the government to enhance

sustainability, iii) The hardware activities should focus more on those which provide more

structural and pro-active protection against floods, iv) Spreading the message to the whole

communities in villages has proven to be a challenge due to their large sizes, v) A sizeable

proportion of the project activities got delayed to the last month due to government permission

delays, procurement delays and staff training, vi) there is a need to enhance the technical

capacity of the team in software DRM activities, vii) monitoring and accountability mechanism

need to be strengthened. Thus, the main recommendations are as follows:

I. Targeting: It is recommended to continue work in Dir, Charsadda and Nowshera but to

reconsider working in Mardan due to its relatively low vulnerability and to undertake a more

thorough needs assessment in Dir to be able to target the most vulnerable village there. ACF

E


3

is also advised to focus on districts highly vulnerable to major floods based on the following

three criteria: River and drainage channels mapping; Damage incurred during 2010 floods and

UNDP One DRM project list of most vulnerable districts. Consider Sindh for DRM work too

since it has recently become the most repeatedly struck province in Pakistan.

II. Number of villages: Review the possibility of increasing the number of villages in the next

phase by increasing the number of field staff, covering each village with only one hardware

construction activity and linking with other NGOs and government agencies to cover some of

the construction work that ACF cannot afford in its budget.

III. Selection of hardware activities: Use the prevention-mitigation-capacity-evacuation-

rescue-relief continuum of DRM activities presented in Chapter 3 to select hardware activities.

The contingency stocks are an important hardware input given their relatively low cost and the

fact that they cover three categories in the continuum even though they are the three bottom-

most categories. Select other hardware construction activities from the highest possible

category of this continuum which is relevant and affordable.

IV. Village committee sustainability: Enhance the sustainability village committees by

providing them training on CBO management issues, having them adopted by other agencies

working long-term in the area (e.g., Concern), and linking nearby committees with each other

for mutual support and learning during crisis and normal times

V. Community DRM awareness-raising: Follow up more closely with committees about their

community DRM awareness-raising session schedules and also look at the possibility of

training young educated and unemployed persons in each UC to conduct sessions in all

targeted villages in that UC for a small honorarium

VI. Monitoring and accountability: Develop formal monitoring plans which clearly specify the

role for all relevant staff in the project’s monitoring, including the frequency, modality, project

dimension and indicators for each relevant ACF staff and how the information coming from the

monitoring done by each staff will be analyzed and summarized Also institute strong and

responsive accountability measures in future projects. Build sufficient time in projects for

government approvals and procurements.

VII. External linkages: ACF should aim to increase government involvement in the project. ACF

could get its village committees registered, and advocate with the DDMAs to treat NGO village

committees as valid structures any government future grass-roots DRM initiatives. It should

sign an MOU with government at time of starting the project which mentions the roles,

responsibilities and inputs that both sides will provide. ACF should also enhance its profile in

coordination sectors like the DRR Forum and the UNDP DRM project and obtaining greater

technical inputs from them. Enhance technical inputs on DRM work by having CPDM provide

some degree of field-based support to ACF staff on software issues, e.g., by developing the

software package in one model village.


4

HAPTER 1: BACKGROUND This chapter provides an overview of the emergency context in Pakistan, and

the specific project implemented by ACF with Royal Norwegian Embassy

funding which is the subject of evaluation in this report. The chapter serves the

purpose of placing the whole evaluation in its proper context and in familiarizing

the reader with the project constraints and scope.

1. Emergency Context in Pakistan

Over the last decade, Pakistan has experienced large-scale internal displacement caused by a

range of natural and human-made disasters. Large-scale displacement occurred in Pakistan in

2010 after the worst flooding to hit Pakistan in memory affected 20 million people, forcing over 7

million people from their homes. Although most of the flood IDPs returned to their home areas

soon after floodwaters receded, most were living in the open for prolonged periods as over 1.9

million houses were damaged or destroyed. The crisis in 2010 was only the latest in Pakistan. It

had been preceded by dislocation of population following clashes between rival militant and

sectarian groups in the tribal areas; military operations against extremist militants and

insurgents; generalized violence and violations of human rights; a devastating earthquake in

2005; and annual floods across the country. Structural poverty, inappropriate development,

rapid urbanization, inadequate infrastructures, increased deterioration of the environment -

deforestation, increased human settlements in hazard prone areas, etc. have increased the

vulnerability to disasters. Flooding recurred in 2011 and 2012, which combined with incomplete

recovery in the 2010 flood-affected districts. Thus, even a moderate flooding could have

devastating effects in the future.

2. Overview of the Project

An ACF DRM assessment in 2010-11 showed that community and provincial DRM plans do not

exist (except for symbolic or outdated documents). It identified a lack of awareness amongst

stakeholders concerning DRM, and thus a need for capacity building. It also recognized a need

for structural mitigation to reinforce infrastructure against the damage caused by a disaster, a

need for improved management of agricultural land as well as a need for community level DRM

activities. Thus, ACF carried out a DRM project funded by the Royal Norwegian Embassy during

2011-12 to reduce morbidity and mortality risks by improving community resilience to natural

disasters. With a focus towards infrastructure that is highly affected by disasters, yet vital after a

disaster occurs, as well as early warning systems which are imperative for reducing loss of life

and environmental degradation stemming from natural hazards, ACF aimed to develop local

capacity and introduce risk mitigation in the intervention areas. The project was a multi-

disciplinary initiative that aimed to involve a variety of stakeholders including ministries, local

authorities, disaster management agencies, NGOs, the academic sector and communities. The

direct beneficiaries of the project were the vulnerable populations affected by natural and man-

made disasters in Nowsherra, Charsadda, Mardan and Lower Dir regions of KPK province in

C


5

northern Pakistan which are vulnerable to both serious floods and earthquakes. The basic

premise of the project was that communities generally understand local realities and contexts

better than outsiders. If ‘at-risk’ groups are involved in all stages of the disaster risk

management, a stronger foundation will be created for the development of sustainable

programs for risk reduction. Through the use of participatory techniques for data collection and

analysis, hazard analysis, trainings, ACF aimed to ensure full participation.

3. Project Details

Project General Objective

To reduce vulnerability and improve resilience to disasters in four districts of KPK, Pakistan

Specific Objectives

To enable communities to prepare for and mitigate the impacts of disasters using participatory

methods aimed at strengthening the self-reliance capacities of the population and reinforcing

or adapting infrastructure

Results

R.1- The ability to recognize hazard events and to cope with their effects is improved

R.2- The capacity to reduce the effect of hazards on communities and services is improved

R.3- The understanding/application of DRM at community and institutional level is improved

Programme Activities

R.1- The ability to recognize hazard events and to cope with their effects is improved

80% of intervention communities have functional local early warning systems

80% of target community can claim access to contingency stocks

90% of target communities are covered by disaster management plans.

R.2- The capacity to reduce the effect of hazard event on communities is improved

Facilities serving 60 communities are structurally strengthened to resist hazard events

90% of intervention communities have identified and equipped evacuation sites

R.3-The understanding/application of DRM at community/institutional level is improved

80% of intervention communities have disaster management units

240 individuals from local or district authorities receive DRM training

25500 individuals in the communities receive DRM Training

Specific Activities

Installation of Early Warning Systems in 48 Villages

Formation of 48 early warning groups in the community

Training of 48 early warning groups in the community

Development of emergency response plans and contingency stocks in 48 villages


6

Development of disaster preparedness coping mechanisms in 54 villages

Conducting of PCVA assessments/community risk and vulnerability mapping

Preparation of village disaster management plan in 54 villages

Signing Memorandum of understanding with 48 communities

Rehabilitation of 14 Flood affected small scale irrigation channels for agriculture

Construction of 5 Retaining and Flood protection walls

Rehabilitation of 13 hazard proof water supply schemes

water quality testing (Bacteriological and chemical testing)

Rehabilitation/reconstruction of water and sanitation facilities at evacuation centers

Rehabilitation of culverts at 15 evacuation routes

Rehabilitation of 13 evacuation roads

Formation of 48 Village Disaster Management Units within the community

Training of 48 DMU's on CBDRM

Training and sensitization of 240 local authorities and local institutions in CBDRM

Development of Contingency Stock at District Level

Community training on CBDRM

Awareness sessions on DRM at Boys & Girl Schools


7

HAPTER 2: METHODOLOGY

At the end of 2012, ACF commissioned an evaluation of the RNE funded

DRM project activities in Khyber Pakhtunkhwa (KP). ACF subscribes to the

Development Assistance Committee (DAC) criteria for evaluation: Impact,

Sustainability, Coherence, Coverage, Relevance / Appropriateness,

Effectiveness and Efficiency. ACF also promotes systematic analysis of the monitoring system

and cross cutting issues (gender, HIV/AIDS etc). These dimensions served as the main

evaluation criteria.

1. Evaluation Questions

The detailed evaluation questions are provided in chapter 3 and were grouped as follows:

Targeting

Project Design

Project Implementation

Project Monitoring

Project Sustainability

Cross cutting Issues

2. Evaluation Approach

Review of Relevant Documents

Prior to the field work, the evaluator reviewed the relevant documents, including those related to

this project, ACF’s general work in Pakistan and DRM globally and documents related to DRM

and emergencies in Pakistan in order to gain a better understanding of the project and its

context and to help develop the evaluation methodology and instruments. Sufficient

documentation was available.

Initial Briefing

Prior to the field work, the evaluator participated in a teleconference with ACF New York HQ.

Indirect Information

Interviews were conducted with local authorities in Lower Dir, Mardan and Charsadda; project

expatriate and national staff in Islamabad and the districts; donor representatives and the

Director CDPM University of Peshawar.

Field activities

Household interviews with community members and Focus Group Discussions (FGDs) with

village committee members based on participatory evaluation methods were utilized. The

villages visited were selected randomly from lists provided by ACF while the community

members for household interviews were selected based on convenience sampling. Random

sampling for household interviews was not attempted since the project provided community-

C


8

level services rather than individual household level services and as such there were no project

beneficiary lists from which people could be randomly selected. In each village, transect walks

were also conducted to the source of disaster vulnerability of each village (e.g., rivers near the

village) to get a better understanding of the disaster profile of the village and the manner in

which ACF work addressed the source of vulnerability. During the transect walks, the hardware

work constructed by ACF in each village were also visited to get a better understanding of the

manner in which ACF work addressed the source of vulnerability of the village.

Village Sampling Details

Villages FGDs HH interviews

Charsadda 5 5 60

Mardan 3 3 36

Nowshera 5 5 60

Lower Dir 5 5 48

Total 18 18 204

3. Evaluation Constraints

A major evaluation constraint was the budget available for the evaluation which meant that while

the evaluation was conducted by an external evaluator, he was assisted in interviewing females

by two ACF female staff members from other ACF projects. In order to analyze the impact of the

participation of ACF staff in the data collection on the neutrality and objectivity of the findings,

the field data is presented separately by males (interviewed by external evaluator) and females

(interviewed by ACF staff). As seen later in chapter 3, except for one section of the household

questionnaire, the differences in the responses given to the external evaluator and internal ACF

staff were not found to be unusually different. However, for the future, it would be advisable for

ACF to recruit independent females for evaluations wherever possible even though the external

evaluator did accept the ACF proposition in this case. The budget implications of this strategy

could be managed by reducing the number of field visit days, which the external evaluator found

to be on the high side compared with other evaluations that he has conducted. Thus, the

additional villages visited did not necessarily add to the richness of the data collected and the

number of days in each district could easily have been reduced by one without affecting the

quality of data collection. Another constraint was the security situation in Pakistan as a result of

which the evaluator could not visit Peshawar due to the sudden deterioration of the security

situation during the evaluation. Field work on two particular days was also scaled back due to

security incidents. However, given the fact that the external evaluator is originally from Pakistan

and speaks the national language, the impact of security incidents on the evaluation activities

was minimized. Finally, timing was also a constraint as the evaluation was conducted near the

end of the year when some of the staff members were on leave, ACF was also focused on

finishing project activities and the Project Manager had finished his contract. However, despite

all these constraints, the evaluator succeeded in collecting adequate amounts of quality

information, mainly because of his strong background in conducing evaluations in Pakistan.


9

HAPTER 3: FINDINGS This chapter provides the findings according to each question included in the

TORs under the headings mentioned in the last chapter. However, the

sequence of the questions and their placement under the headings has been

changed in some places to enhance readability. The findings under each

question are based on triangulation of information from the various sources of information

mentioned in the last chapter.

1. Targeting This section focuses on the following questions

a) Did the communities selected for the project classify as being most vulnerable and

highly exposed to disaster risk, and did they meet the criteria established in the project design? Could the PCVA process have been in any way improved to better achieve this?

The evaluator visited 18 of the 60 villages targeted by the ACF for this project. All the villages in

all four districts were found to be vulnerable and those in Charsadda and Nowshera were also

among the most vulnerable villages in KP province highly exposed to disaster risk in the opinion

of the evaluator based on the numerous evaluations that he has conducted in Pakistan after the

2010 floods. The worst-affected villages were those where the floods destroyed most houses in

the village and most of the families in the villages were displaced for weeks or months, finally

returning to live in their villages in tents or makeshift houses for prolonged periods until they

could construct or received houses. Thus, the difference across the districts can be clearly seen

in the response of individual households to the following questions across the four districts:

Difference in disaster vulnerability across ACF districts

Charsadda Nowshera Mardan L. Dir

When was your village last affected by a major

disaster? (% identifying 2010 floods)

100% 100% 100% 100%

Was your family displaced in the disaster?

(% responding yes)

96% 92% 62% 10%

If yes, for how long? (no. of months) 2.1 1.2 0.7 0.4

Was your house damaged during this disaster? (% responding yes)

96% 92% 60% 40%

In all districts, the 2010 floods were identified by people as the most devastating disaster in

living history. FGDs revealed that the 2010 floods were viewed as more damaging than the

2005 earthquakes in the villages which experienced both mega-disasters. However, there is

significant difference in the damages experienced across the four districts. While 90%+ of the

families were displaced in Charsadda and Nowshera were displaced during the floods, only

10% of the families in Dir were displaced while 62% were displaced in Mardan. The period of

displacement ranged from a high of 2.1 months in Charsadda to 0.4 months in Dir. Finally, only

C


10

40% of the families reported house damage in Dir compared with 90%+ in Charsadda and

Nowshera. Furthermore, FGDs revealed that even where houses were damaged in Dir, the

damage was relatively minor, consisting of cracks or a wall partially collapsed compared with

whole houses being swept away in Charsadda and Nowshera. However, villages in Dir did

suffer crop and infrastructure damage. Even so, it would be useful to conduct more thorough

assessments in Dir to see if there are more vulnerable areas for future work. Thus, the evaluator

would recommend the adoption of formal and clear targeting criteria so that the most vulnerable

villages can be targeted more consistently across all project areas. ACF currently has targeting

criteria which include need, direct access, absence of other agencies and security. However,

there is a need to operationalize the need criteria further for the KP context. The following

sections provide the basis for developing such operationalized needs criteria.

International NGOs can generally intervene with emergency response only in cases of major

disasters because of the shortage of funds and based on the premise that communities and

local authorities can deal with minor emergencies on their own. Thus, it is recommended that

even for DRM work, international NGOs should focus on natural hazards which are capable of

producing major disasters and on geographical areas where these natural hazards can be

expected to produce major emergencies with a view to reducing the needs for major emergency

response by INGOs subsequently as a result of this DRM work. Villagers and ACF staff in each

area mentioned a wide range of natural hazards to which communities are exposed, including

earthquakes, floods, snowfall, landslides, seasonal diseases and drought. However, it is also

clear that within this long list, it is only earthquakes, droughts and floods that are capable of

producing major disaster globally. Snowfalls, landslides and seasonal diseases almost never

produce large-scale disasters which would justify emergency response by international NGOs.

Moreover, even droughts are unlikely to produce major emergencies in the context of KP

province. Thus, the starting point of formal targeting criteria for ACF should be to concentrate on

areas which are highly prone to earthquakes and floods in KP. Even among these two, it is

clearly floods which are more amenable to DRM work due to their more predictable nature,

especially DRM work by ACF given the fact that earthquake-proofing generally requires

structural improvements in shelter construction, which is not a core ACF sector. Thus, ACF

should primarily focus on areas which are vulnerable to major floods and then incorporate

earthquake DRM concerns if those flood-prone areas are also vulnerable to earthquakes.

With floods identified as the most important natural hazard to focus on for KP, the next task

would be to identify the most flood-prone areas in KP. Unfortunately, ACF is constrained in

doing so by the fact that the Pakistani government has yet not developed rigorous, technology-

based hazard mapping. Thus, it would have to rely on less rigorous sources of information. The

starting point for the future would be to look at where the major rivers flow in KP. This would

immediately reveal the fact that Nowshera and Charsadda host all three major rivers in KP

(Indus, Kabul and Swat), while Mardan hosts at most major drainage channels (e.g., the Kalpani

drainage channel) while Dir hosts a couple of smaller rivers. It is the location of rivers which

explains the vast differences in the responses above. All the villages in Charsadda and

Nowshera were on major rivers banks, those in Mardan were generally next to the Kalpani while


11

those in Dir were not near any major river or even drainage channels. Another source of

information for identifying the most vulnerable districts, UCs and village would be to look at

which areas incurred the most damage during the 2010 floods. Appendix A, which shows the

damaged caused by the 2010 floods in KP, reveals that Charsadda, Nowshera, Dir and Mardan

were number 1, 2, 8 and 20 among KP’s 24 districts in terms of number of affected people.

More disaggregated information at the UC and village levels may be available with the

government. This district ranking reveals that while Charsadda and Nowshera are prime targets

for future DRM work, Mardan seems to be a low priority district since very few people were

affected in the first place. Clearly, Dir is a very isolated area and one which is probably more

poverty-stricken than the other three districts. Figures also reveal that it incurred a high degree

of loss. While Dir has a relatively large number of people officially recognized as being affected

(i.e., eligible for government assistance programs, such as the Watan cards), it seems that the

Dir team has not been able to focus on some of them due to security issues as some of the

areas identified by the Dir government authorities as highly affected (Lal Qila and Samarbagh)

were perceived by the Dir team as unsecure. Thus, it would be useful to conduct more in-depth

needs and security assessments there for future work. Finally, the UNDP’s one DRM project

has identified 30 most vulnerable districts in Pakistan. This list could also serve to guide ACF’s

geographical targeting. Thus, the PCVA process could be strengthened by first weeding out the

less vulnerable villages by macro-level criteria and focusing the PCVA process on the most

needy villages to begin with.

Recommended needs criteria for future DRM geographical targeting

Basis for initial targeting should be natural hazard, with floods being the main priority

Subsequent targeting should be based on the districts, UCs and villages most flood-prone

with the following being the main sources of information:

-River and drainage channels mapping

-Damage incurred during 2010 floods

-UNDP One DRM project list of most vulnerable districts


12

Vulnerability Zones and ACF Project Sites in 4 KP Districts

The NGO Code of Conduct asserts that agencies will distribute assistance based on objective

needs. The evaluator strongly feels that NGOs should have strong information within their files

to demonstrate that they are doing so for each project. It would also be advisable for ACF to

ensure that it does so for each project. Currently, while the targeting is excellent in Charsadda

and Nowshera, even for those districts, the corresponding information to reveal that these really

are the most vulnerable villages is not immediately available in project files. Having a map

immediately available in its files which shows that all the villages in Nowshera and Charsadda

are next to major villages along with information about the damage caused by the 2010 floods in

various districts, UCs and villages will help ACF in clearly demonstrating its adherence to this

NGO code of conduct principle. Currently, ACF has the maps with its project areas shown next

to rivers in Nowshera and Charsadda (see maps below) but not the information about 2010

damages. As can be seen in the maps below, the project areas in Dir and Mardan are not next

to major rivers. Similarly, if some areas are being left out due to security reasons, as in Dir, it

would be good to have a formal security report by its professional security team and a

subsequent SMT decision to back up this decision. If some areas are being left out because


13

other agencies are already present there, it would be good to have on file minutes of meetings

of coordination bodies where such coordination among NGOs normally takes place.

b) Did the project strike the right balance between number of villages covered and the

amount of work done in each village? If not would it have been better to cover more villages with less work, or fewer villages with a greater amount of work?

While detailed appropriateness of the software and hardware work done by ACF will be

discussed in the next section (project design), in order to answer this question, it can be

asserted briefly here that the evaluator finds the particular mix of such activities to be highly

appropriate and well-selected, which means that it would not be advisable to drop any of them

in order to increase the number of villages. At the same time, given Pakistan’s large size and

high vulnerability, the number of most vulnerable villages is very high compared with the 60

villages covered in this project and probably runs into hundreds if not thousands. At the same

time, the number of agencies undertaking DRM work is small. Thus, there is a clear need to

increase the number of villages from the present 60 in a future similar project without completely

dropping any category or even sub-category of work. This obviously poses a dilemma. As far

as the evaluator can see, the main bottleneck to increasing the number of villages is the number

of front-line field staff who work at the village level. Fortunately, such staff does no cost much

and an additional staff member could perhaps be hired for as little as 12,000-15,000 dollars per

year, which could easily be absorbed by some minor reconfiguration of the overall budget. The

other bottleneck is the cost of hardware work which is much higher. However, the evaluator also

noticed that in some villages more than one hardware activity was being implemented. By

keeping hardware activity down to one activity per village, it may be possible to cover more

villages. Also, ACF could overcome the hardware-related constraint and increase the number of

villages by trying to leverage its work by getting the government and other NGOs to cover some

of the hardware activities which may be beyond ACF’s budget.

2. Project Design a) Were the objectives, results and activities selected in the implementation of this

program relevant for the identified population needs? Was the participatory approach used appropriate an effective in identifying and selecting specific interventions? Was there sufficient consultation and participation of beneficiaries in the design, targeting, implementation and post-implementation stages? Did the activities match with what people wanted?

The starting point of ACF’s work for this project is the Participatory Community Vulnerability

Analysis (PCVA) exercise which aims to view each village’s vulnerability and possible steps to

reduce it through the eyes of the community by using participatory methods to understand the

main hazards, community strengths and weaknesses, and possible remedial measures for

strengthening the community’s resilience. The use of such an approach has helped ACF to

adopt a highly participatory process and identify the main components of the project in light of

community perspectives. While many of the project components, such as early warning

systems, contingency stocks and village plans, are obviously based on industry standards for

DRM work, their specific form in different villages has depended on community perspectives. To


14

begin with, the identification and development of the village hazard profile and community

resilience and weaknesses is largely based on community perspectives. Furthermore, the

village committees, which were the main lynchpin for project implementation, were selected by

gathering villagers and having them select responsible, trustworthy people who represented all

segments of the community. Furthermore, the village disaster plans were based largely on

community perspectives. The specific contents of the early warning systems, evacuation sites

and contingency stocks were based on community participation. Finally, the hardware work

done in villages was largely selected by villagers within the constraints of the project budget.

b) What are the main problems (negative or unexpected impacts from the interventions) that the population identified concerning the implementation of the activities and what solutions were identified by the project team?

The responses to the following questions reveal the high degree to which communities felt

involved in the project. Satisfaction was very high across all districts, though slightly lower in Dir,

and both sexes. The main problems identified by the committees related to the difficulty in

covering the whole communities with DRM awareness—raising given the large sizes of some of

the villages. A couple of villages also mentioned that ACF staff sometimes came late to

meetings, a problem created often due to the security situation in the area because of which

ACF security staff asked ACF program staff not to proceed until the security situation cleared.

However, there were no complains with the basic contents and approach of the program.

Community satisfaction with ACF project approach

Percentage saying Yes Char-

sada

Now-

shera

Mar-

dan

Lwr

Dir

Male Fem

ale

Were you consulted about the types of services and

their content?

98 96 92 84 90 95

Did agency staff always treat you according to your

cultural norms?

95 95 97 96 95 99

Were services provided keeping in mind your work

schedule?

96 97 97 88 95 94

Were the services timely for you in light of your

needs?

97 96 97 81 99 92

c) Did the project strike a good balance between hardware and software? If not, define

what the balance should have been? Was the scope of hardware activities sufficient? If not, what other activities could have been added? Was the scope of software activities sufficient? If not, what other activities could have been added?

The main hardware activities within the project consisted of the provision of contingency stock

for evacuation, rescue and relief activities and the construction of latrines, evacuation paths,

drainage and irrigation channels and flood protection walls to reduce the risk of disasters. The

main software activities were the setting up and training of village committees, development of

village disaster plans and early warning systems, awareness-raising about DRM among

communities and schools, training on DRM for construction workers and training of government


15

officials. These activities can be better compared with the help of a conceptual framework to

evaluate and compare the value and impact of different DRM activities. As shown in the table

below, DRM activities can be classified along a continuum consisting of six categories with each

higher category representing a more structural and stronger form of disaster risk reduction than

those below it. Prevention activities include efforts to reduce the occurrence of destructive

hazards, e.g., through reforestation and riverbed dredging, and are obviously the most effective

form of DRM. However, such activities are generally done by governments rather than NGOs.

Given that even with high prevention efforts, there will still be some hazards occurrence, the

next best option is mitigation, i.e., redirecting hazards away from communities through flood

protection walls, drainage channels etc. Given that even with high mitigation efforts, there will

still be some hazards reaching communities, the next option is to move communities out of the

way of the hazards through evacuation before the hazard reaches them. The next option is

rescue efforts for those who could not be evacuated in time. Finally, the provision of timely relief

activities to both evacuated and rescued people reduces the risk of epidemics. In addition,

building the capacities of communities can also enhance their resilience to disasters.

Continuum of DRM Activities

DRM categories Purpose Examples Related ACF

activities

Prevention To reduce the

occurrence of

destructive hazards

Global climate change

advocacy; riverbed

dredging; reforestation

None

Mitigation To build buffers

between hazards and

communities

Drainage channels,

flood protection walls

Drainage channels,

flood protection walls

Community

capacity-building

To enhance the

resilience of

communities by

strengthening their

socio-economic status

Livelihoods work,

nutrition work, CBO

strengthening

Irrigation channels;

VDMUs; community

awareness-raising;

Village disaster plans

Evacuation

preparedness

To help communities

escape approaching

disasters

Early warning systems;

evacuation paths and

sites;

EWS, evacuation

paths and sites

selection, contingency

stock items (e.g.,

megaphones)

Rescue

preparedness

To help rescue people

caught by hazard

Rescue tool and

equipment stocks

Contingency stock

items (e.g., life

jackets)

Relief preparedness To provide life-saving

services to evacuated

and rescued people

Watsan, health,

shelter, NFIs stocks

Evacuation center

latrines; contingency

stock items (e.g.,

water jerry cans)


16

With the understandable exception of prevention activities, which are generally out of the range

of NGO technical and financial capacities, ACF work is covering all the remaining categories of

DRM work. This reflects the comprehensive nature of the DRM package that ACF is providing.

On the software side, activities like village committee and disaster plan development, early

warning system and evacuation site identification do not cost much and can also be seen as

mandatory components of DRM work. Thus, those activities should all be retained. The real

issue is about hardware activities since they cost much more. The above framework can help in

making decisions about specific hardware work. Thus, the objective in each village should be to

select activities which fall in the highest possible category above while being relevant to the

village and affordable within the ACF budget. So for example, flood protection walls, drainage

channels and irrigation channels are much more durable forms of DRM work than activities such

as evacuation paths and latrines in evacuation centers since the former reduce the chances of

communities getting affected by disasters in the first place while the latter are about helping

communities once they have been struck by hazards. Flood protection walls tend to be

expensive and should preferably be built where benefiting large or several villages os that their

per beneficiary cost is closer to other mitigation activities. More specifically, the evacuation

paths built will not be helpful in major disasters as they will become fully submerged. Within

community capacity-building, especially those related to livelihoods strengthening, activities

which benefit the whole community, such as irrigation channels, are more beneficial than

activities which target individual families, such as provision of agricultural inputs to individual

households. Finally, ACF’s contingency stocks packages are also a very useful input even

though they target the lower categories for the simple reason that they have a low cost but still

cover three DRM categories (evacuation, rescue and relief categories). In summary, the

software activities and contingency stocks should be mandatory components given their low

cost and wide-ranging coverage. Among the remaining hardware activities, those targeting

mitigation and community capacity-building are more useful than evacuation paths and latrines.

That said, it is recognized that the latter may be the best affordable option for some villages.

3. PROJECT IMPLEMENTATION a) Evaluate the impact of the DRM training and disaster plans e.g. impact on capacity of

trained population to be better prepared through the training and preparedness plans. Evaluate the effectiveness of the Early Warning system that was put in place.

DRM village committee and training: In all the villages visited in the four districts, the

evaluators found that both men and women village DRM committees had been formed and had

received training from ACF on DRM activities. The village committees seemed representatives

of all sections of the community in each village and seemed highly engaged in the DRM work

initiated by ACF. As a result of their DRM training, the committee members generally seemed

familiar with DRM concepts and fully geared up to play an active role in keeping their villages

safer during future disasters. They were highly appreciative of the DRM training and seemed

well aware about things like the values of early warning system, contingency stock and


17

evacuation sites. The committee members demonstrated ownership about the work done by

ACF and expressed a willingness to continue it after ACF’s withdrawal.

...We can vouch that ACF has

done excellent work with us,

better than any other NGO or

government agency and we

are very happy with its work

and find no problems in it.

Committee members in a

Charsadda village

Village disaster and contingency plans: Village disaster plans were seen hanging on the

walls in all the villages visited in Mardan, Charsadda and Nowshera although there was some

variation in the level of details contained by the plans across the villages. Thus, it would be

useful for ACF to develop a clear outline for the topics to be covered by such plans and ensure

that all villages complete all the topics. In Dir, the team had not yet engaged the villagers in any

village to complete such plans by the time of the evaluation which was undertaken during the

last two weeks of the project. The Dir team felt that they would still be able to complete the

plans in all the villages by December 31. However, clearly, even if they did so, it would seem to

be a rush, last moment job with little follow-up possible, which would affect the quality of the

whole exercise.

Early warning system: The early warning systems planned by ACF include linking the village

committees with governmental sources of information (by giving both parties the telephone

numbers of each other) about approaching hazards and also providing them with equipment,

such as megaphones and battery-operated loudspeakers for communicating the information

received from external sources within the villages. Most of the village committees had been

provided with the telephone numbers of relevant local authorities, such as police stations and

Civil Defence offices. However, the village committees had not been registered with government

departments so that those departments could proactively provide information to villagers. This

issue will be discussed in more detail under the sustainability section. Finally, due to some

delays in procurement, the equipment for intra-village dissemination of early warning systems

was being delivered in villages around the time of the evaluation. However, even so, such

systems would still be up and running before the 2013 flood season in July-August. However,

the success of ACF’s EWS will of course depend on villages getting timely and accurate early

warning from government departments. Unfortunately, the technical and management capacity

of governments departments to provide such information is very low in Pakistan at the moment,

which is a major challenge for ACF. The final chapter will look at this issue in more detail.


18

Evacuation sites selection: In almost all villages, the committees had selected evacuation

sites in the village. Those sites generally consisted of schools, mosques or high, open places

within the village which were already being used as evacuation sites by villagers. In many of

these sites, ACF has built latrines. However, it would also be useful for ACF to help each village

select evacuation sites outside the village as well in case the floods are so severe that they

threaten to submerge the whole village, including the internal evacuation sites.

DRM construction training: These trainings had not been delivered yet in any district by the

time of the evaluation because of the delays in the project caused by late government

permission and procurement delays. According to subsequent information sent by ACF, these

trainings were completed before the end of the project.

Community DRM awareness-raising: These 3-hour sessions were supposed to be carried

out by the village committee members once they were trained by ACF. Such sessions were held

by committee in many villages. However, their efficacy was affected by two major reasons.

Firstly, in most of the villages, the committee members and in fact people in general did not

have the immediate capacity to become good trainers based on the 3-day training that they

received. Secondly, the population in some of the villages ran up to 3-4,000 persons. Thus,

assuming that in one session, the committee trained up to 100 villagers, it would still take it

around 20-25 sessions to train all adults and children above the age of 5. This obviously is not

an easy task for people who have full-time vocations and only work as volunteers on the

committee. These community trainings were supposed to be the main conduit for imparting

knowledge to the community members on DRM concepts and about the village disaster plans,

early warning systems, contingency stocks etc. As a result of these constraints, understanding

within the larger community about DRM concepts and the village DRM activities was found to be

variable across villages, districts and gender as shown in the table below which is based on

household interviews. The table reveals that while familiarity with the village committees and the

hardware work is very high (90+) across all districts and both sexes. However, familiarity within

the general community is much lower and highly uneven across the districts and sexes.

Familiarity in Dir is lower than in the other three districts. The most notable difference is across

the two sexes, with women reporting much greater familiarity with project activities than men,

which in some ways is counter-intuitive since usually Pakistani men have more opportunity to

participate in project activities due to the cultural conservatism. This difference may partially be

due to the fact that women are more village-bound and hence easier to attract to awareness-

raising sessions and meetings. However, the fact that women were interviewed by internal ACF

staff while men were interviewed by the external evaluator could also have been a reason for

the higher positive responses reported by women. In any case, even if women are as well

informed as reported below, increasing the awareness among men would be crucial since

women enjoy much less freedom of movement and initiative in Pakistan during disasters and in

normal times. As mentioned earlier, spreading awareness across such large, illiterate and busy

communities is an enormous structural challenge faced not only by ACF but all agencies. Some

related recommendations are provided in the last chapter.


19

Community familiarity with ACF DRM project components

Are you familiar with:

(Percentage responding yes)

Char-

sadda

Now-

shera

Mar-

dan

Lwr

Dir

Male Female

1. DRM village committee 98 98 97 95 94 100

2. Early warning system 75 61 73 50 38 94

3. Village disaster plan 94 74 73 30 48 85

4. DRM contingency stock 64 58 56 2 21 68

5. Evacuation sites 89 85 61 52 55 100

6. DRM training 95 91 71 73 69 94

7. DRM construction work 95 98 85 93 95 92

Overall, the communities generally felt that the project activities have benefited them in a variety

of ways, including increasing their self-reliance, ability to deal with crisis and physical dangers,

security of property knowledge about DRM and access to water and sanitation facilities. The

differences in responses across the four districts and the two sexes were minor with respect to

these issues. Communities mainly perceived these improvements due to their familiarity with the

establishment of the village committee, which they felt was a huge boon for the whole village,

and the visible DRM construction work that had occurred in all the villages and to a lesser extent

the other software activities with which their familiarity was lower as seen earlier.

Community perceptions about project impact

Has ACF DRM work

increased your:

(Percentage responding yes)

Char-

sadda

Now-

shera

Mar-

dan

Lwr

Dir

Male Female

Ability to stand on own feet 96 98 97 92 94 98

Ability to deal with crisis 98 98 100 90 94 98

Safety from physical dangers 98 98 97 78 94 91

Security of property 98 95 97 80 93 90

Knowledge about DRM 92 88 78 75 69 100

Access to water and

sanitation services

96 98 97 95 95 98

4. Project Monitoring a) How suitable and effective were the M&E systems in place through the duration of the

project and how could these have been improved? What systems were put in place to ensure that outputs provided were of the highest quality possible and were acceptable to beneficiaries?


20

ACF staff is regularly monitoring field activities, starting from field-level staff all the way up to the

Country Director. However, it would be useful for ACF to have formal monitoring plans for each

project which clearly specify the role for all relevant staff, starting from field-level staff all the way

up to the Country Director in the project’s monitoring. This would include the frequency,

modality, project dimension and indicators for each relevant ACF staff. More importantly, the

plan would also specify how the information coming from the monitoring done by each staff will

be analyzed and summarized (preferably through a software-based monitoring application) and

then fed to progressively higher levels of the ACF Pakistan team, terminating with the Country

Coordination Team and what decisions and follow-up each level would undertake to ensure that

the project achieves its objectives on time. In this way, the team could constantly monitor the

progress on the common program dimensions (e.g., effectiveness, efficiency etc) that eventually

will be part of the project evaluation TORs (instead of being informed about shortcomings by

external evaluators) and take remedial timely action to ensure high project quality. While project

monitoring is clearly occurring, it is also true that there have been schedule slippages along

some project components, especially in Dir. The presence of such a formal plan would have

made it much easier to ensure timely achievement of objectives.

Another missing piece in ensuring quality and timely achievement of objectives is the institution

of strong accountability measures, which can enhance the ability of the ACF team to gain

information about project achievement beyond that coming from its own physical monitoring.

ACF Pakistan has now set up a Performance, Quality and Accountability unit in Islamabad and

plans to incorporate accountability mechanisms in future projects.

5. Project Sustainability

a) Were the construction material and masonry techniques applied appropriate (were

they disaster-resilient?) What alternatives would have been better?

Since there was no technical person on the evaluation team, the analysis of technical

construction quality undertaken as part of the evaluation was limited and was based on three

dimensions. Firstly, the external evaluator observed the construction quality visually for the

following common construction problems in all villages and only minor, occasional problems

were found:

Are there any cracks or plaster falls in the construction?

Have the fixtures and fittings (e.g., doors and windows) been fixed properly?

Dampness on walls/floor?

Is the toilet design ok?

Does the foundation look straight?

Are any hinges and bolts loose?

Secondly, the evaluator reviewed the construction-related documents. ACF has qualified

engineers within its DRM teams who had developed detailed Bills of Quantities for each


21

construction work. The evaluator requested the ACF team to share sample documentation for

some sites to show whether the BOQ material was actually delivered and used in the

construction and the required documentation from Nowshera, Charsadda and Mardan was

reviewed and found to be in order. Finally, the evaluator also checked with village committee

members about the technical quality of work who in some cases did have some background in

the small-scale construction work that ACF undertook as part of this project and they expressed

a high degree of satisfaction. Thus, overall, the construction quality seems sturdy, disaster-

resilient and best possible option under given conditions, though it would have been better for a

qualified technical person to physically observe the work and review construction documents. It

would also be good for ACF to undertake standard construction tests, e.g., brick quality, wood

quality, sand quality and retained weight of crush tests.

b) Evaluate whether the maintenance requirements of the structures were within the

capacity of the beneficiary users, if ACF did enough to reinforce these local

capacities. Evaluate the willingness of communities to maintain the infrastructure and

propose how community ownership could have been improved or reinforced.

The main mechanism used by ACF to ensure maintenance of both the software and hardware

work is the setting up of village committees. While in a few villages, the committees already

existed, in most villages the committees set up by ACF were the first experience of communities

with such committees. Committee members seemed motivated and keen to maintain the

committees even after ACF’s withdrawal while communities saw the committees as an

extremely helpful form of social capital developed for them by ACF. So, the chances seem high

that most committees will survive even after the withdrawal of ACF and will maintain the

software and hardware work. However, given the high migration in some areas in KP, turnover

within committees could be high and it would be a test of their endurance whether villagers keep

refilling the vacancies. In this regard, some additional steps by ACF could increase the chances

of committee continuation. Firstly, while ACF has trained committees on DRM issues, it would

also be useful to give them some basic training on management, CBO operations and

development issues. Second, it would also be advisable for ACF to see whether there are other

international or local NGOs working long-term in these areas which could adopt the committees

after ACF’s withdrawal. Third, it would also be helpful for ACF to link committees of nearby

villages with each other for mutual support, information sharing and learning during floods and

even normal times. Finally, it would be important to link these committees more strongly with

government structures, as discussed in the next section.

c) Evaluate whether links between VDMUs and authorities are sufficiently reinforced.

Evaluate the impact of the DRM training for local and district authorities. What were

the main issues/weaknesses and how could this have been improved? Were the local

disaster authorities (PDMA and DDMU) sufficiently involved at all levels of the

project? If not, in what aspects of the project could their involvement have been

improved?

ACF organized DRM trainings from relevant district-level departments in all four districts though

some of the training in Mardan was still outstanding at the time of the evaluation. The evaluator


22

interviewed district staff in three of the districts (except Nowshera) and all of them expressed a

high degree of appreciation and satisfaction with the training. With respect to involvement of

disaster authorities, all NGOs have to take official permissions from government authorities and

to that extent the government authorities were involved in the project. However, government

departments in Pakistan tend to be highly bureaucratic and it is not easy to get them involved to

any greater extent, beyond granting permissions, in NGO projects. Nevertheless, such

involvement would be crucial for the success of ACF’s DRM project more so than for other

watsan or food security projects. Viewed so, it is felt that ACF should have built upon the good

will created by the government trainings and expended greater effort in increasing the

involvement of government departments in the project and enhancing the linkages between

them and its village committees in various ways despite the bureaucratic nature of Pakistani

government departments. To begin with, ACF could have got its village committees registered

with the Social Services Departments in district governments to enhance the chances of the

survival of these committees after ACF’s withdrawal. Secondly, ACF could have advocated with

the DDMUs, preferably in coordination with other DRM NGOs) to treat the village committees

set up by ACF as valid structures for disseminating early warning information, and involving in

any government future grass-roots DRM initiatives. Thirdly, ACF could have followed up on the

trainings by trying to have some practical linkages between government departments and the

village committees, e.g., by arranging mock drills between the Civil Defence Departments

(which has grass-roots presence through a cadre of community-based volunteers), and

encouraging the government to initiate small-scale mitigation measures in ACF villages which it

could not afford itself.

6. Cross-Cutting Issues

a) How realistic and appropriate was the budget of the project? Was it respected? The table above provides information on the original budget, actual % expenditure by October

2012 and expected % expenditure by project end. This information reveals that almost all major

cost budget lines are being respected and will be fully utilized by the end of the project with an

overall burn rate of 97%. However, there were some budget issues due to the exchange rate

fluctuations given that three currencies (Nok, USD and PKR) were involved. Overall, ACF lost

budget of around $100,000 due to the depreciation of the USD against the Nok despite the fact

that the PKR has also depreciated significantly against the USD. According to the ACF Pakistan

Finance team, this loss was covered by a budget modification where district-level contingency

stock items were dropped. Overall, the Finance team felt that the DRM team had done good

budget management which helped in the budget being respected.

ACF DRM Project Budget utilization

DESCRIPTION Budget

(Nok)

% SPENT

(oct ’12)

Estimated %

Spent , 31-12-


23

2012

Personnel 475,953 84% 98%

Expatriates 238,076 87% 98%

Local staff 237,877 82% 97%

Communication, Visibility, Information 4,900 4% 104%

Equipment Procurement 17,164 100% 100%

Equipment Hire 64,445 89% 108%

External Evaluation 20,000 0% 50%

Project Costs 597,098 76% 97%

Construction mitigation 458,165 83% 97%

Disaster preparedness in community 67,380 47% 125%

Capacity building 71,553 62% 69%

Other Costs 107,772 88% 97%

TOTAL DIRECT COSTS 1,287,333 80% 97%

Indirect Costs (9%) 115,860 80% 97%

TOTAL COSTS 1,403,193 80% 97%

b) Were ACF’s rules on procurement adhered to? Have concerns relating to a zero tolerance on corruption policy been adequately observed in the project implementation?

ACF has a whistle-blower policy and an anti-corruption policy to reduce the chances of

corruption in the projects. Moreover, the approval limits of the Country Office and field offices

are much lower than that seen by the evaluator in other agencies with a view to reducing the

chances of corruption. The ACF Pakistan Finance team also diligently works to minimize the

chances of corruption by having a zero-tolerance policy against corruption so much so that a

driver was fired for a few hundred rupees violation. The team also calls additional suppliers by

telephone even where the logistics and field teams have sent in the three mandatory quotations

in order to further minimize the chances of fraud. All these steps ensure strong financial due

diligence and the evaluator did not come across any reports of corruption within villages or field

offices. However, this due diligence did slow up procurement to some extent with some of the

delays mentioned earlier being related to delays in procurement, which in turn were also

exacerbated by the lack of clarity among staff on procurement rules. ACF held a team workshop

recently to smoothen out these issues.

c) What measures were taken to ensure a gender balance at all stages of the project

(assessment, design and implementation)? How could this process have been

improved?

KP is an extremely conservative environment even by Pakistani standards where women’s

rights are low and it is very difficult for NGOs to work with women to enhance their rights due to


24

the significant threat of violent backlash by militants who in recent weeks have killed scores of

female teachers and aid workers in KP. Within this background, ACF has still succeeded in

involving women in the project. Thus, all villages had separate women’s committees who were

also provided DRM training like the male committees. Overall, women’s understanding of the

project activities seemed higher than that of men, as mentioned earlier under the “Community

DRM awareness-raising” section. In many villages in Dir (which is more conservative than

Charsadda etc), women had formed the committees without the knowledge of men who did not

approve of such activities. In some Dir villages, ACF did not work with women because it was

not given access to women by the community men. However, given the severely conservative

culture and the serious threat of violent backlash, there are severe limitations on the extent to

which the involvement of women could be increased by ACF or other NGOs. There are greater

opportunities to do so in the districts other than Dir. Thus, men in these villages admitted that

they initially had some misgivings about allowing women to participate in the project as they had

this impression that foreign NGOs spread vulgarity among women and encourage them to rebel

against the local cultures. However, they also said that these misgivings had been largely

removed after the start of the project due to its positive impact and the fact that they saw ACF

staff constantly respecting local cultural values..

d) Evaluate the added value to the project from the partnership with the Centre for

Disaster Preparedness and Management (CDPM), and whether the involvement of CDPM was maximised.

The CPDM is the only institution focused exclusively on DRM activities in Pakistan. ACF

requisitioned the CPDM to undertake the trainings for its own staff, the government departments

and the village committees. The trainings were highly appreciated by all the stakeholders and as

such the engagement with the CPDM seems like an excellent move by ACF to enhance and

work with local capacity. However, there could have been scope to increase the utilization of

CPDM services further since DRM is such a new area for ACF staff in Pakistan. While the

training was certainly good, given the limited initial background of ACF staff in DRM work, it may

have been useful to involve the CPDM at the field level to a limited extent, e.g., by having them

undertake the software component in one model village and monitoring ACF’s work in an

additional 1-2 villages initially and providing feedback. This suggestion is based on the

observation that the ACF staff, especially in Dir, needed more capacity and support in

implementing the software component which was very new to them.


25

HAPTER 4: CONCLUSIONS AND

RECOMMENDATIONS This chapter summarizes the main findings and conclusions from the last

chapter and provides recommendations for enhancing the quality of the project

further in the future along each dimension of improvement identified in the

conclusions section. The emphasis is on providing recommendations which

would be feasible within the difficult working environment faced by ACF in Pakistan.

1. Conclusions

a) Did the program reach the intended results and objectives? If not, what were the main constraints faced by the organization and were they a consequence of poor quality of the initial assessment, poor project design or poor implementation? Did the project sufficiently identify and manage risk related to the project activities?

Overall project achiovements

Results and related indicators Status at time of

evaluation

The ability to recognize hazards and cope with them is improved

80% of communities have functional local early warning systems Work in progress at time

of evaluation; ACF later

confirmed completion by

end of project though

quality would have been

better with more spaced

out completion

80% of target community can claim access to contingency stocks

90% of communities are covered by disaster management plans.

The capacity to reduce the effect of hazards on communities is improved

Facilities serving 60 communities are structurally strengthened to

resist hazard events

Done

90% of communities have identified and equipped evacuation sites Identified everywhere but

not all equipped due to

government permission

issues

The understanding of DRM at community/institutional level is improved

80% of intervention communities have disaster management units 100% have units

240 individuals from local or district authorities receive DRM training Higher number trained

25500 individuals in the communities receive DRM Training Trained but variable

absorption of concepts

C


26

The table above provides the status on the three results and corresponding indicators at the

time of the evaluation field visits. Thus, the project is likely to achieve its objectives by

December 31, 2012 though the quality of the work could have been higher if so much work was

not squeezed into the last weeks of the project. These delays happened due to procurement

and government permission issues. Overall, the project is a very well-conceived one and aims

to fill a critical need in enhancing resilience within KP. Many components of the project have

been implemented very well, such as the setting up of the village committees and their training

and the completion of the construction activities. However, other components did have room for

improvement. These shortcomings were caused by a wide range of external and internal

constraints faced by ACF, including high insecurity, governmental bureaucracy, shortage of

skilled staff, stringent procurement rules, low literacy levels within communities and the

conservative culture in the country. Among these, ACF has managed security risks well and has

comprehensive and efficient security systems in place. However, more attention should have

been devoted to dealing with the risks associated with governmental bureaucracy, shortage of

skilled staff (both quantity and skill level), stringent procurement rules, low literacy levels within

communities and the conservative culture in the country. While the communities did not identify

major areas of improvements (beyond the standard requests for more hardware services in the

future), the evaluator himself has identified some. The next section provides recommendations

for improving the quality of the project further in these areas in the future.

2. Recommendations

Targeting: Targeting of the most vulnerable is excellent in Charsadda and Nowshera but is

less accurate in Mardan and Dir. Mardan has structural issues as very few people were affected

during the 2010 floods. In Lower Dir, a large number of people were affected, so the problem is

more about ensuring more thoroughly that ACF is focusing on the most vulnerable areas. It is

recommended to undertake a more thorough needs and security assessment in Dir. In general,

ACF is advised to focus on districts and villages which are vulnerable to major floods. To

identify such areas, the following three criteria are suggested: River and drainage channels

mapping; Damage incurred during 2010 floods and UNDP One DRM project list of most

vulnerable districts. Consider Sindh for DRM work too since it has recently become the most

repeatedly struck province in Pakistan.

Number of villages: Review the possibility of increasing the number of villages in the next

phase by increasing the number of field staff, covering each village with only one hardware

construction activity and linking with other NGOs and government agencies to cover some of

the construction work that ACF cannot afford in its budget.

Selection of hardware activities: Use the prevention-mitigation-capacity-evacuation-

rescue-relief continuum of DRM activities presented in Chapter 3 to select hardware activities.

The contingency stocks are an important hardware input given their relatively low cost and the

fact that they cover three categories in the continuum even though they are the three bottom-

most categories. Select other hardware construction activities from the highest possible

category of this continuum which is relevant and affordable. For example, wherever possible


27

prioritize drainage channels, irrigation channels and protection walls over evacuation routes and

latrines. Focus on community-level activities, .e.g., irrigation channels, rather than individual

household-level activities like provision of agricultural outputs given that the basic purpose of

the project is community joint work which may be undermined by helping certain families with

individual-level inputs and ignoring other families.

Village committee sustainability: Enhance the sustainability village committees by

providing them training on CBO management issues, having them adopted by other agencies

working long-term in the area (e.g., Concern/IRSP), linking nearby committees with each other

for mutual support and learning during crisis and normal times, getting committees registered

with the Departments of Social Welfare and getting them recognized by DDMAs as valid

structures for providing early warning information and inclusion in the government’s DRM work

in coordination with other agencies.

Community DRM awareness-raising: Reaching to the larger community has proved

challenging due to the large size of the communities, busy schedules and low literacy levels.

This challenge could be reduced by having communities develop detailed community

awareness-raising schedules and following up more closely with committees about the

schedules and also looking at the possibility of training young educated, unemployed persons in

each UC to conduct sessions in all targeted villages in that UC for a small honorarium

Village disaster plans: Develop a clear formal outline of what each plan should contain and

then ensure that all villages closely follow and complete the plan according to this list to ensure

consistency across villages

Evacuation sites: Beyond the within-village evacuation sites identified now, also help

communities identify evacuation sites outside villages which could be accessed in case the

whole village gets submerged.

Monitoring and accountability: Develop formal monitoring plans which clearly specify the

role for all relevant staff in the project’s monitoring, including the frequency, modality, project

dimension and indicators for each relevant ACF staff. More importantly, the plan would also

specify how the information coming from the monitoring done by each staff will be analyzed and

summarized (preferably through a software-based monitoring application) and then fed to

progressively higher levels of the ACF Pakistan team, terminating with the Country Coordination

Team and what decisions and follow-up each level would undertake to ensure that the project

achieves its objectives on time. The presence of such a formal plan would make it much easier

to ensure timely achievement of objectives and avoid slippages. Also institute strong and

responsive accountability measures in future projects.


28

Technical capacity/CPDM: Enhance technical inputs on DRM work by having CPDM provide

some degree of field-based support to ACF staff on software issues, e.g., by developing the

software package in one model village.

Government linkages: ACF should build upon the goodwill created by the government

trainings and expend greater effort in increasing the involvement of government departments in

the project. To begin with, ACF could get its village committees registered with the Social

Services Departments. Secondly, ACF should advocate with the DDMAs, preferably in

coordination with other DRM NGOs) to treat NGO village committees as valid structures for

disseminating early warning information, and involving in any government future grass-roots

DRM initiatives. Thirdly, ACF could try to have some practical linkages between government

departments and the village committees, e.g., by arranging mock drills between the Civil

Defense Departments, and encouraging the government to initiate small-scale mitigation

measures in ACF villages which it could not afford itself.

Coordination with other stakeholders on DRM: Having successfully completed one DRM

project, ACF should enhance its profile in this sector by increasing its visibility and inputs in

coordination sectors like the DRR Forum and the UNDP DRM project and obtaining greater

technical inputs from these forums.


29

APPENDIX 1. Damage in KP from 2010 Floods

KPK - Flood Affected Population

S #

District

Total_Population Total_

UCs

Affected_UCs

Affected_Population

HH_Damage

Aftec_pop_pec

1 Abbottabad 1169904 51 0 2 Bannu 971930 49 49 54,473 7,782 6 3 Battagram 451340 20 9 10,416 1,488 2 4 Bunair 738496 27 24 5,614 802 1 5 Charsadda 1492939 49 34 502,732 71,819 34 6 Chitral 465075 24 12 69,164 9,881 15 7 D.I.Khan 1234706 47 26 394,608 56,373 32 8 Dir Lower 1131676 37 7 180,686 25,812 16 9 Dir Upper 753313 28 14 210,498 30,071 28

10 Hangu 459275 19 19 45,841 6,549 10 11 Haripur 1007960 45 42 56,646 8,092 6 12 Karak 622919 21 21 50,935 7,276 8 13 Kohat 774318 32 32 38,716 5,531 5 14 Kohistan 684004 38 38 464,333 66,333 68 15 Lakki 715139 33 26 28,092 4,013 4 16 Malakand 660046 28 6 45,086 6,441 7 17 Mansehra 1431208 59 12 22,870 3,267 2 18 Mardan 2123149 75 43 19,992 2,856 1 19 Nowshera 1174961 47 27 499,818 71,403 43 20 Peshawar 2860402 92 16 237,068 33,867 8 21 Shangla 632670 28 7 83,649 11,950 13 22 Swabi 1494583 56 11 15,389 2,198 1

23 Swat 1834756 65 42 634,654 90,665 35 24 Tank 349373 16 16 148,890 21,270 43

Grand Total 25234142 986 533 3820170 545739 16%

2. Scoring Matrix

Criteria

Rating (1

low, 5 high)

Rationale

Impact 4 The data in chapter 3 from household interviews clearly shows that over 90% of the respondents felt that the project helped improve their resilience due to the committees set up and the construction work. These aspects include:

Ability to stand on own feet

Ability to deal with crisis

Safety from physical dangers


30

Security of property

Knowledge about DRM

Access to water and sanitation services

However, impact could be increased further by enhancing understanding within the larger community about other aspects of the project through more effective community awareness-raising sessions. As shown in chapter 3, understanding among men about other project components like EWS, contingency stopcock etc is variable.

Sustainability 3 The committees set up are the main mechanism for ensuring project sustainability. However, there is scope to improve sustainability further by:

linking communities with other agencies, government departments and with each other and

through further training of the committees on CBO management issues

Coherence 4.5 This is the strongest point of the project since the combination of hardware and software activities cover almost all categories of the DRM activities continuum presented in chapter 3. Coherence could be improved further by concentrating more on the higher categories of the continuum in terms of hardware construction work, e.g., by doing more irrigation and drainage channels and protection walls than evacuation routes and latrines wherever feasible. The project is also in line with the Pakistan government’s Disaster Management strategy and the grass-roots DRM approach of the DEC-funded NGOs, whose DRM project the evaluator reviewed recently.

Coverage 3.5 The villages in Charsadda and Nowshera are all clearly among the most vulnerable villages in KP. However, there is a need to enhance the coverage of the most vulnerable villages further.by using the following criteria:

Basis for initial targeting should be natural hazard, with

floods being the most priority natural hazard to focus on

Subsequent detailed targeting should be based on

collecting information about which districts, UCs and

villages are the most flood-prone with the following being

the main sources of information:

-River and drainage channels mapping

-Damage incurred during 2010 floods

-UNDP One DRM project list of vulnerable districts

Relevance 4.0 Relevance is again a very strong point of the project given that Pakistan has become increasingly vulnerable to major


31

disasters in recent years and very few agencies, government or NGO, are doing much active work in DRM though some of the project areas, e.g., Mardan, are not very highly vulnerable.

Effectiveness 3.5 Some project components have been implemented highly effectively, including hardware construction work and many software components in Nowshera, Charsadda and Mardan. However, there were also much delay in completing many project activities, especially in Dir which reduced the effectiveness of the project.

Efficiency 4 ACF has managed to stay within budget limits till the end through efficient project financial management achieved through very close and regular coordination between the Finance department and the Project Manager. The budget itself reflects efficiency as it increases the resilience of more than 50,000 persons at relatively low cost.

3. Acronyms

BOQ Bill of Quantities

CBDRM Community-based Disaster Risk Management

CBO Community-based Organization

CDPM Center for Disaster Prevention and Management

DAC Development Assistance Committee

DDMA District Disaster Management Authority

DRM Disaster Risk Management

EWS Early Warning System

FGD Focus Group Discussion

IDPs Internally Displaced Persons

KPK Khyber-Pakhtunkhwa Province

NDMA National Disaster Management Authority

PCVA Participatory Community Vulnerability Analysis

RNE Royal Norwegian Embassy

UC Union Council

UNDP United Nations Development Program

VDMU Village Disaster Management Unit

4. List of Interviews

Noor-Ul Amin District Disaster Management Officer, Lower Dir

Abigail Chatagnon ACF Field Coordinator KP- Pakistan

Waheed-ul Haq Civil Defence Officer, Mardan

Dildar Hussain ACF-Pakistan, Deputy Project Manager, DRM

Dr. Amir N Khan Director, CDPM University of Peshawar

Mr. Zahoor Khan District Liaison Officer Charsada


32

Clement Philat ACF Field Coordinator KP- Pakistan (ex)

Nick Radin ACF-USA, Water and Sanitation Advisor

Zach Watson ACF-Pakistan Water and Sanitation Advisor

Anwar Zeb ACF-Pakistan, Deputy Project Manager, DRM

5. List of Documents Reviewed

ACF Global Guidelines for DRR Mainstreaming

ACF Global DRM for Insecure Environments Briefing Paper

ACF Global DRM Policy

ACF PCVA Practitioner Manual

Bills of Quantities (various) for hardware activities

Deputy Project Managers’ Monthly reports (various)

DRR Forum Pakistan minutes of meetings (various)

PCVA reports for Charsadda, Lower Dir, Mardan and Nowshera

Project Manager Hand-over notes

Project Proposal and interim report, DRM Project

Training reports for government officials and VDMUs

Village Profiles (various)

6. Good practice case study

Village Profiles in ACF Pakistan DRM Project

Innovative Features & Key Characteristics

Based on Participatory Community Vulnerability Analysis (PCVA) exercises, ACF Pakistan

helps community members develop detailed Village Profiles for the villages that it finally selects

to be part of its DRM project. The Village Profiles include the following information:

Overall general village information

Geographic and demographic information

Accessibility and security

Socio-economic situation

Local resources and critical facilities

Other NGOs present

Hazard profile and seasonal risk calendar

DRM capacity analysis

Preparedness and response prospects

DRM needs identified

This is an innovative and useful exercise for the following reasons:

The exercise is led by and based on the perspective and knowledge of the local

community and is thus more likely to be accurate

It helps the community to view its vulnerability in a systematic fashion and also identify


33

its internal strengths as well as the gaps for which it would need external help. This

exercise then helps the community to develop Village Disasters Plans

It helps ACF to gain a better understanding of local hazard patterns and community

strengths and weaknesses. This information can then be used to fine tune the ACF

program in line with local realities.

Practical/Specific Recommendations for Roll Out

In order to ensure that ACF programs around the world get the maximum out of this exercise,

the following preparation and follow-up will be helpful

Ensure that all sections of the community, particularly women and other highly

vulnerable groups participate in the exercise

Ensure that the community has been adequately briefed about basic DRM concepts so

that it can participate more meaningfully in the exercise

Set realistic expectations about what ACF would be able to provide

Encourage local government officials to participate in some such exercises wherever

feasible so that they develop a better understanding of local vulnerabilities and villagers

develop closer linkages with them

Based on the information collected from several villages in a locality, develop overall

hazard profiles of different localities