oxfam measuring the impact of disaster risk reduction learning companion

8/7/2019 oxfam measuring the impact of disaster risk reduction learning companion

http://slidepdf.com/reader/full/oxfam-measuring-the-impact-of-disaster-risk-reduction-learning-companion 1/16

Measuring the Impact of Disaster Risk Reduction:A Learning CompanionOxfam Disaster Risk Reduction and Climate ChangeAdaptation Resources



1. About this Companion

Disaster Risk Reduction (DRR) and Climate Change

Adaptation (CCA) are corporate priorities or Oxam GB.

This Learning Companion aims to support Oxam sta

in developing indicators o disaster resilience1 or any

programmes that aim to reduce the risk o disaster, and

as part o the Monitoring, Evaluation and Learning (MEL)

system.

This Companion is one o a series o documents

providing inormation about DRR and CCA. You should

read the Introductory Companion rst or denitions o

DRR and the key terminology. It assumes a basic level o

knowledge o the Oxam GB MEL system and processes,

so, i you require more inormation on this, please browse

the Programme Quality section o the Intranet or contact

[email protected].

This Companion has been produced in response to

requests rom programme sta or more guidance on

measuring the impact o their work and, in particular, or developing indicators that link to national and international

DRR rameworks. It will help you to develop high-quality

indicators, collect data to measure your eectiveness,

and use the data you gather to inorm programme-level

decision-making.

2. Introduction: What are the issues?

Monitoring, evaluation and learning are crucial in all

Oxam programmes. They help us to learn, understand

and increase our eectiveness, make evidence-based

decisions, and ensure accountability to our donors and

the communities with which we work. Monitoring and

evaluation (M&E) o DRR impact to date has not been

systematically applied across the development sector or a

number o reasons, such as the ollowing:

•M&Etraditionallymeasureschange,butinDRRthere

is a ‘reverse logic’, because its success is measured i

disasters are avoided.

•Vulnerabilitytodisasterismostoftentheresultofthe

interplay between several actors (see the Learning

Companion An Introduction to Disaster Risk Reduction or

more on this). It is unusual or programmes to address all

risk actors aecting a vulnerable community, which means

that it can be challenging to identiy clear programme logicagainst which to measure change.

•Itisoftendifculttoattributecauseandeffecttospecic

measures when there is a wide range o actors working in

a complex environment.

Within Oxam, there are an increasing number o

programmes with exclusively DRR objectives or that

incorporate a DRR approach. As the need to address

the risk o disaster increases, so does the importance o

demonstrating progress, impact and accountability to all

stakeholders in this expanding area o work.

Globally, ve priority areas or reducing vulnerability to

disaster have been identied or states and other actors

in the Hyogo Framework or Action. Developing and

monitoring indicators according to these areas allows

us to monitor our own and others’ progress in these

crucial areas. Specic guidance on indicators o disaster

resilience ollows later, but rst it is important to understand

how these indicators t into the logic o our programme

design and how we use indicators as part o a wider MEL

plan.Learning Objectives

After reading this Companion, you should:

• understand why MEL is important in DRR work;• know what the key elements of a MEL plan are

and have a sense of how you can incorporate

this into your programme work plan;

• understand the role of logic models and logical

frameworks, and the differences between them;

• know what indicators are, the critical role they

play in programme monitoring, and how to write

good indicators for DRR programmes;

• understand how to use the ‘Characteristics of

a Disaster-resilient Community’ in developing

outcome indicators for your programme; and• understand how the development of indicators

ts into the wider Oxfam MEL system.

1

Page opposite: Volunteers receive training in ood preparedness in

Bangladesh and practice how to make a rat using local materials.

Photo: Shailan Parker/Oxam

1 Resilience reers to the ability o a community, society or other

system to continue to unction during or ater a hazard. It is generally

determined by the extent to which the social system is capable o adapting and organising itsel to manage basic unctions during

hazards and to recover aterwards. It is a broader term than capacity,

because it represents the coming together o capacities with the

social, institutional structures that enable their use. (Summarised rom Twigg (2007) Characteristics o a Disaster Resilient Community,

p6, and DFID (2004) Disaster Risk Reduction: A Development

Concern: A Scoping Study, p16.)



22



4. Systematic monitoring against core

indicators

4.1. Developing programme logic2

Good-quality M&E depends on coherent programme

design. Coherent programmes have strong internal

logic which makes clear: what the programme intends

to achieve, how it expects to achieve it, and what

assumptions are being made – both in terms o how theprogramme will be implemented and its theory o change

(i.e. how the programme expects change to happen

within a given context). All o Oxam’s programmes are

based on the ollowing logical chain:

4.1.1. What is the dierence between a logic model and

a logical ramework?

A logic model aims to clariy the overarching logic o

a programme. It can be used in the design phase or during implementation to review the theory o change

or the whole programme or or a specic component o

it. It is usually no more than one page long and can be

expressed as a table with columns or each stage or in

a ree-fow diagram. All programmes in Oxam need to

demonstrate an understanding o their programme logic.

A logical ramework (lograme)3 is a more detailed

document but based on the logic model, generally used as

a programme management tool. Although Oxam does not

require logrames, many donors that und DRR projects, such

as the Department or International Development (DFID) or

the European Commission (EC), do ask or them as part

o project proposal and reporting. Although you will use

the same principles to design a logical chain in a lograme,

donors may have their own terminology or the dierent levels.

3. Developing a MEL plan

Good MEL plans contain three dierent types o

processes – all o which play a key role in improving the

quality o our work. While this Companion ocuses mainly

on the rst type outlined below, all three are essential.

I you would like more inormation on any o these

processes, please contact [email protected].

3

1. Systematic monitoring against core

programme indicators

•Developprogrammelogic.

•Identifykeyindicators(e.g.forDRRtheyare

based on Hyogo Framework priority areas and

using Characteristics o a Disaster Resilient

Community).

•Collectandanalysedatawhichrelatetothe

indicators and will enable assessment o impact

and better decision-making.

2. Space or refection and learning

•Monitoringreviews:(twiceayearindevelopment

programmes, and as appropriate in humanitarian

responses.)

•ThesecanbeusedinstandaloneDRR

programmes and also to consider how

incorporating DRR could improve the quality

o programmes not currently taking a DRR

approach.

3. Rigorous analysis o outcomes and

impact

•AlthoughtheOxfamGBevaluationpolicymakesit

mandatory or only some programmes to carry out

mid-term and nal evaluations, all programmes

should plan or some type o evaluation.

•Evaluationscantakemanydifferentforms.They

can be internal or external, ormative (which help in

developing programmes) or summative (document

impact or learning rom a completed programme).

ACTIVITIES

OUTPUTS

OUTCOMES

IMPACT

Assumptions

Based on the evidence that we have, we

assume that there is a logical progression romone to another. However, these assumptions

need to be tested during the development o

the M&E ramework. It is important to question

whether the outputs will actually lead to the

outcomes, and i so, how they will do so.

3

2 This is only a brie introduction to logic models to illustrate how

DRR indicators relate to this. For ur ther guidance, please see the

MEL Guidance Sheet on Programme Logic Models.

3 For more on logrames, please see the Pick Up and Go module: An

Introduction to Logical Frameworks or contact [email protected].



•Impact indicators are quantitative measures or qualitative

judgements (or both) by which the achievements o

outcomes (or the positive or negative ‘impact’ on the

target group) can be judged. These indicators are

developed at outcome/objective or impact/goal level.

Process and impact indicators can be direct or indirect

(‘proxy’) and quantitative and/or qualitative. Here are a

ew indicators showing what this terminology means inpractice. Further examples are given below in Section 5.3.

•Direct indicators are straightorward and easy to

measure. They are usually a precise, comprehensive

restatement o the respective objective.

Example: % o households possessing a amily

contingency plan developed by their children, where the

head o the household can describe the plan.

• Indirect (proxy) indicators are used when the

achievement o the objective is not directly observable,

measurement would incur very high costs, resources or measuring are not available, or the achievement o the

objective is only measurable a long time ater the activity

has been carried out.

Example: Level o public health systems working

according to minimum standards, suering minimal

damages rom foods, and easily repaired and unctioning

again within three days.

Logrames are also used to document the indicators

at dierent levels (output, outcome and impact) and

means o verication (evidence/data which will be used

to measure progress towards the indicators). Logrames

can be used as a basis or developing a monitoring

ramework.

4.1.2. Testing programme logic through monitoring

Once you have documented your programme logicand started implementing your programme, checking

the validity o the assumptions in the logic model is an

essential part o programme monitoring. This is key to

assessing the programme’s contribution towards change.

4.2. Developing high-quality indicators

4.2.1. What are indicators?4

Indicators are observable or measurable changes due to

the actions or input rom a project. Oxam describes an

indicator as a “quantitative or qualitative actor or variable

that provides a simple and reliable means to measureachievement”.5

There are two dierent types o indicators:

•Process indicators show how the project is progressing,

what is happening, and i the original plan is being

ollowed. They can measure the amount o work being

done as well as the quality and timeliness. These

indicators are developed at output level.

4

4 This is a very brie introduction. There is existing generic guidance on developing good-quality programme indicators; please see the MELGuidance Note on Indicators or contact [email protected].

5 Defnition taken rom Organisation or Economic Co-operation and Development (OECD) Development Assistance Committee (DAC)

A cash or work scheme in Cambodia built a system o canals or ir rigation. A total o 6 canals are dug in 9 villages and the CFW scheme

directly benefts 270 villagers. Photo: Pariphan Uawithya/Oxam.



a. Outputs (called ‘results’ in EC logrames)

Indicators at this level ask what the result of the

activities was. You held a training (activity), and you

have trained some health workers. It is good to make sure

these indicators incorporate measures o quality, quantity

and time (QQT). It is not enough to just train community

workers i they stop working ater the end o the course.

Good example o an output indicator or a DRR programme:

•Numberof villagevolunteerstrainedinrstaidandable

to demonstrate skills during a simulation six months ater

training.

b. Outcomes (or ‘specifc objectives’ or ‘purpose’ in EC/

DFID logrames)

Indicators at this level highlight what happened as a

result of the outputs. Training community workers

is not an end in itsel; they need to bring about some

observable or measurable change that can be attributed

to their presence. Several indicators may be needed

• Quantitative indicators pertain to amounts and answer

the questions: who was it, when, where, how oten and

how much? They are expressed in numbers such as units,

prices, percentages, time, scoring or ranking.

Example: % o trained committee members who can

explain the process o setting up and executing small-

scale mitigation works.

•Qualitative indicators pertain to descriptions and answer questions such as how and why? They are a judgement

and can be expressed as a narrative or by using scoring

or ranking. They are used or gathering people’s opinions,

and monitoring satisaction and changes that have

occurred.

Example: Level o fexibility and appropriateness o

credit schemes to the requency and magnitude o

shocks and the needs o households.

It is most useul to employ a balance o quantitative and

qualitative indicators.

Two o the indicators above start with the words “level

o...”. These are called ‘scoring indicators’. The level o

their achievement needs to be measured, and each level

needs a context and project- or programme-relevant

example. Oten, ve levels are used or simplicity. Some

generic scoring levels, which could be adapted to the

relevant context, are:

1. no evidence o this at all;

2. some evidence o this but very limited;

3. moderate levels o evidence but with signicant scope

or improvements;

4. substantial evidence with recognised limitations; and5. very positive evidence o comprehensive achievements

which are covering the (or example, local area)

eectively.

4.2.2. How do I develop indicators or each level in my

logic model?

As explained above, it is really important to have a small

number o indicators or each level o programme logic.

Oxam programmes oten have good output indicators,

but then do not have indicators to show change at

outcome or impact level. This is a problem because:

•Withoutoutcomeindicators,theassumptionswehavemade in our logic model are never tested. We, thereore,

do not know i our theory o change makes sense. We

only know that activities are being completed, but we do

not know i they lead to the change we want.

•Weonlyhaveanecdotalevidencethatweareresponsible

or positive change. We cannot authoritatively talk about

the change that we have created. This means that we are

not able to be accountable to donors or beneciaries.

Note: You do not need indicators at activity level,

because you report against the activities themselves. At

activity level, you need to employ more o a benchmark

– i.e. certain activities will be completed at/by a certain

time.

5



Milagros Villagas Nima, 17, picks corn in her back yard. She lives in the tiny hamlet of San Martin de Malingas, Peru, which like many places in the

area is prone to ooding and landslides. As well as supporting civil defence committees to be better prepared when disasters strike, Oxfam is also

supporting families to use new technology such as drip irrigation to adapt to the changing climate.” Photo: Gilvan Barreto/Oxfam

6

to give reliable inormation on the achievement o the

specic objective, but their number should be limited to a

level manageable or data collection.

Good example o an outcome indicator or a DRR

programme:

•Percentageofpopulationwhoreachsheltersquickly

(within x hours/minutes) and saely in the event o a

hazard o x strength.

It is important to note that a number o dierent outputs

might contribute to each outcome. So, or example, the

outcome ‘vulnerable communities able to access shelters

quickly and saely’ might be made up o a number o

outputs, such as ‘shelters constructed or retrotted,

‘vulnerable groups trained on evacuation and practising

in regular simulations’ and ‘completion o the small-scale

mitigation works’ that allow sae access to shelters. Only

by measuring the outcome will we know i our outputs

have had the desired eect.

c. Impact (‘principal objective’, ‘overall objective’ or ‘goal’

in EC/DFID logrames)

This is the hardest level at which it is possible to attribute

change to Oxam’s intervention. In some cases, Oxam

may not be able to commit to attributing change at impact

level, and some donors, or example, ECHO, do not

require indicators at this level.

However, in other circumstances, it may be useul tomeasure change towards the goal, even i it is not entirely

possible to attribute the change entirely to Oxam’s

intervention. The Oxam MEL team would encourage this

where possible, because – although there may be other

infuencing actors – it can still provide useul data that

test the assumptions made in our theory o change.

For example, i the goal o the programme is ‘to

contribute to the substantial reduction o disaster

losses caused by cyclones (both in terms o human lie

and the social, economic and environmental assets o

communities) in our disaster-prone areas’, it may be

possible to include an indicator, such as:

People try to cross a ooded road by oot holding their belongings above their heads to keep them dry in Sultanpur village, West Bengal.

Anisa Draboo/Oxam GB

6



•percentagereductioninthenumberofhumanlives

lost during cyclones o x strength within x km o the

programme area by the end o the programme period.

In this instance, it may be possible to use data collected

by the government or other organisations on loss o lie.

Similar indicators could be developed or the social,

economic and environmental assets.

4.2.3. What makes a good indicator?

All indicators must be:

•linkedtoyourprogrammelogicmodelandanimportant

part o your lograme;

•appropriate–youneeddifferentkindsof indicatorsat

dierent stages o the project cycle;

•exible–theyarenot‘setinstone’andcanbechanged

i ound to be impractical or immeasurable i the situation

changes; however, money can be wasted collecting data

against inappropriate indicators, so it is even better to get

it right rst time; and

•SMART: specic, measurable, attainable, relevant, and

time-bound.

• Qualitative indicators should also be SPICED:

subjective, participatory, interpreted (and communicable),

cross-checked (and compared), empowering, and diverse.6

Impact indicators must:

•includetargetvalues;

•helptoanswerthequestion‘howwillweknowiftheresult

has been achieved?’; and

•includedetailsof quantity,qualityandtime.

Make sure:•Yourindicatorsmeasurekeyelementsofvulnerability

and coping capacity. The initial analysis o vulnerability

and risk should highlight which elements are most

important or the communities with which you are working.

Indicators covering attitudes, knowledge, behaviour or

access to resources can help to measure these.

•Yourindicatorsmeasuretheoutcomesofourworkfor

vulnerable and socially excluded people, especially

women, older people, men and women with disabilities

orwhoareaffectedbyHIV/AIDS.Seethesectionon

social exclusion and DRR in the Learning Companion

An Introduction to Disaster Risk Reduction or moreinormation on why this is so important.

•Youselectasmallnumberofindicators.Donotdevelop

similar indicators that require dierent data, and do not

develop too many indicators. This will make it easier to

gather data and inorm decision-making.

•Youknowhowyouwillcollectdata(andfunddata

collection) to measure the indicator beore you put it into

the programme design.

•Youavoidusingvagueandambiguoustermsfor

indicators such as ‘to increase awareness…’, ‘appropriate

use o…’, ‘beneciary involvement…’, ‘increased

participation…’, ‘good understanding…’.

Involvebenefciarieswherepossible

The involvement o beneciaries in developing indicators

and collecting data has a number o benets, including:

•ensuringtheprojectisrelevanttothebeneciaries’needs;

•strengtheningrelationshipsbetweentheprojectandthe

community, which improves implementation;

•ensuringasharedunderstandingof whatsuccesswill

look like in the project, and contributing to downward

accountability;

•improvingourabilitytomeasurechangebyensuringthat

the indicators measure changes that the community eels

are appropriate to the objectives o the project; and•creatinggreaterbeneciaryownershipof theprojectand,

thereore, contributing to its sustainability.

5. Linking Oxam’s impact to global disaster risk

reduction rameworks

It makes sense to link how we measure Oxam’s impact in

achieving DRR to the Hyogo Framework or Action (HFA).

It improves accountability by enabling progress against

HFA priorities to be communicated in a standardised

way across the sector. Likewise, improving how we

measure progress against the HFA enables us to build

the capacity o others, especially governments, to do thesame. The identication o good practice through MEL

linked to the HFA also makes it easier to lobby or the

replication and scale-up o successul interventions. This

section describes two documents which we have used to

produce the tables o example indicators that ollow.

5.1. Characteristics o a Disaster-resilient Community:

A Guidance Note

Characteristics o a Disaster-resilient Community: A

GuidanceNote(CDRC)wasproducedinresponseto

callsbyNGOsforindicatorsfortheHFAthatprioritised

the poorest people and those most vulnerable todisaster.7 Rather than generic indicators, it contains a

comprehensive list o the ‘characteristics’ o disaster

resilience arranged around the same ve thematic areas

as the HFA. These are, thereore, a useul tool when

developing indicators or DRR interventions.

While its ocus is at the level o communities, the

document also highlights the importance o wider

institutional, policy and socio-economic actors in

supporting community-level resilience. For each

component o resilience, it also lists ‘Elements o the

Enabling Environment’. These are less detailed thanthe characteristics but illustrate the local, national and

sometimes international dimensions o the enabling

environment, which need to be addressed to achieve

community resilience. These elements are extremely

useul to consider when developing advocacy strategies.

Oxam programmes have used this guidance note in

various ways in MEL and advocacy.

In addition to its use in identiying indicators or MEL,

it has also been used as a scoping tool to support

governmentsandNGOstoanalysevulnerabilityand

develop programme interventions. By suggesting what a

disaster-resilient community might look like, and setting

out many dierent elements o resilience, it can help us to

identiy which vulnerabilities exist in our target area and

need to be addressed. Examples o and urther advice

7



on how CDRC has been used are available rom the

Adaptation and Risk Reduction (ARR) team in Oxord.

5.2. Oxam DRR indicator matrix

The ARR team in Oxord has reviewed hundreds o

indicators used in Oxam’s programmes around theworld. A large number o good-quality indicators have

been presented in a matrix organised according to the

ve priority areas set out in the HFA, with an additional

section or those measuring ‘mainstreamed’ objectives.

The ull matrix is available on the ARR Intranet pages or

on request by e-mailing [email protected].

5.3. Developing indicators using these tools

This section illustrates how the Characteristics o

a Disaster-resilient Community can be turned into

indicators. Examples are all drawn rom Oxam’s

programmes, as documented in the indicator matrix. Itshould be noted that:

•Indicatorsforonly10ofthecharacteristicsareused

here to demonstrate the process o transorming the

characteristics. There are 167 characteristics in total.

•Whilethecharacteristicsdocumentisaveryuseful

basis or developing indicators or DRR, it is not

a comprehensive checklist. I appropriate to the

programme, indicators that are not covered by the

characteristics can be used.

•Theindicatorsinthiscompaniononlyillustratethe

Characteristics o a Disaster-resilient Community, not the

Elements o an Enabling Environment. However, i you

wish to measure the success o lobbying and advocacy

work, you can use the same process to transorm the

Elements o an Enabling Environment into indicators.

•ThisCompanionfocusesmainlyonoutcomeindicators,

as these are the hardest to write and are most otenmissing rom Oxam’s M&E work.

Make sure programme indicators are context specifc

The characteristics are generic and aim to represent all

contexts. The indicators which we have suggested give an

indication o how generic characteristics can be

transormed into generic indicators. To successully

measure the outcomes o your project or programme, the

indicators suggested below must be modied to refect its

unique context. The process or identiying which indicators

are most useul to your programme and which are easible

to measure is likely to be a crucial part o the programmedevelopment process. The development o indicators is

most eective when a range o programme stakeholders

are involved so that there is agreement about what is

important to measure and how data will be collected.

Remember that indicators that start with the word ‘level’

or ‘extent’ are ‘scoring indicators’: the level o their

achievement needs to be measured, and each level (1–5)

needs a context and project- or programme-relevant

example. The generic ‘scoring levels’ can be ound in

Section 4.2.1.

The Jovos, young volunteers or disaster prevention meet regularly to learn about how they can help people to stay sae rom ooding and rock

alls. They have regularly training courses including frst aid and evacuation skills. Photo: Jane Beesley/Oxam

8

6 More inormation about the defnitions o SMART and SPICED can be ound in Roche C (1999) Impact Assessment or Development

Agencies: Learning to Value Change, Oxord: Oxam GB.

7 Ofcial Guidelines or Reporting on Progress on the Implementation o the Hyogo Framework or Action have been produced but do not havea particular emphasis on social justice. See the section on Measuring Progress in DRR on the UNISDR website or more.



Oxam consulted extensively with women and other vulnerable sections o Darur society to tailor our projects to meet villagers’ precise

needs. Livelihoods and protection teams worked together to identiy low cost items that were urgently needed but which would not increase

the risk o looting and attack. Photo: Adrian McIntyre/Oxam

9



10



Component of Resilience 2: Legal and regulatory

systemsCharacteristic 2.2 Community aware o its rights and the

legal obligations o government and other stakeholders to

provide protection

•Potential generic indicator: % o community members

who can list at least x rights and x legal obligations o

governments and other stakeholders to provide protection

•Example indicator from an Oxfam programme: % o

community members who can detail at least x relevant/

specic legislations/regulations/procedures and their

importance

Component of Resilience 4: Integration with

emergency response and recovery

Characteristic 4.1 Community and other local-level

actors in sustainable development and DRR engage in

joint planning with community and local-level emergency

teams and structures

•Potential generic indicator: % o community and other

local-level actors in sustainable development who attend

joint planning meetings with community and local-level

emergency teams and structures and can list at least x o

their contributions to joint plans•Example indicator from an Oxfam programme: level o

inputs between communities and Parliaments during the

process o the development/review and deliberation o

DRR legal rameworks

Component of Resilience 2: Vulnerability and impact

data and assessment

Characteristic 2.3 Assessment ndings shared,

discussed, understood and agreed among all

stakeholders and ed into community disaster planning

•Potential generic indicator: % o stakeholders who

can explain assessment ndings and how they eed into

community disaster planning

•Example indicator from an Oxfam programme: % o households who can list at least ve main issues rom the

ParticipatoryCapacityandVulnerabilityAnalysis(PCVA)

ndings

1111

Component of Resilience 1: Public awareness,

knowledge and skillsCharacteristic 1.2 Whole community exposed to/taken

part in ongoing awareness campaigns, which are geared

to community needs and capacities (e.g. literacy levels)

• Potential generic indicator : % o community members,

who took part in accessible awareness campaigns,

that are able to describe at least x relevant measures to

approach community needs and capacities

Example indicator from an Oxfam programme: level

o involvement o youth groups in organising DRR

awareness-raising events

Component of Resilience 5: Learning and Research

Characteristic 5.1 Documentation, use and adaptation o

indigenous technical knowledge and coping strategies

• Potential generic indicator: level o documentation, use

and adaptation o indigenous technical knowledge and

coping strategies (e.g. in DRR plans)

• Example indicator from an Oxfam programme: level o

indigenous technical knowledge and coping mechanisms

used to develop awareness-raising materials

Component of Resilience 1: Environmental and

natural resource management

Characteristic 1.2 Adoption o sustainable environmental

management practices that reduce hazard risk

•Potential generic indicator: level o adoption o

sustainable environmental management practices that

reduce hazard risk (e.g. within the parish)

•Example indicator from an Oxfam programme: level o

sustainability o the system in place or the maintenance

o protection works

Component of Resilience 6: Physical protection,

structural and technical measures (including physical

capital)

Characteristic 6.3 Sae locations: community members

and acilities (homes, workplaces, public and social

acilities) not exposed to hazards in high-risk areas within

locality and/or relocated away rom unsae sites

Thematic Area 1: Governance

Hyogo Priority 1: Ensure that DRR is a national

and local priority with a strong institutional basis

or implementation

CDRC Thematic Area 2: Risk Assessment

Hyogo Priority 2: Identiy, assess and monitor

disaster risks and enhance early warning

CDRC Thematic Area 3: Knowledge and Education

Hyogo Priority 3: Use knowledge, innovation

and education to build a culture o saety and

resilience at all levels

CDRC Thematic Area 4: Risk Management and

Vulnerability Reduction

Hyogo Priority 4: Reduce the underlying risk

actors



12

•Potential generic indicator: % o community members

with access to and means to reach acilities (homes,workplaces, public and social acilities) not exposed to

hazards in high-risk areas within locality and/or relocated

away rom unsae sites

•Example indicator from an Oxfam programme: % o

households o most-at-risk amilies willing to move to saer

sites or to strengthen their individual dwelling (resources

permitting)

Component of Resilience 1: Organisational Capacities

and Coordination

Characteristic 1.6 Sucient number o trained and

organisational personnel and community members to

carry out specic relevant tasks (e.g. communication,

search and rescue, rst aid, relie distribution)

• Potential generic indicator: % o committee membersshowing skills in carrying out relevant response tasks

according to minimum standards in a coordinated manner

• Example indicator from an Oxfam programme: %

o (e.g. committees) having a system or managing

their response equipment and options or replacing

consumables, doing essential maintenance, andsupporting basic organisational activities

Component of Resilience 2: Early warning systems

Characteristic 2.2 Early warning system capable o

reachingwholecommunity(viaradio,TV,telephoneand

other communications technologies, and via community

early warning mechanisms such as volunteer networks)

• Potential generic indicator: % o community members

who receive early warning messages rom at least one

source

• Example indicator from an Oxfam programme: level o unctioning o the communications/early warning system

or the transmission o alerts that permits inormation to

reach people in an appropriate and timely manner

Component of Resilience 4: Emergency resources and

infrastructure

Characteristic 4.2: Sae evacuation routes identied,

maintained, and known to community members

• Potential generic indicator: % o sae evacuation routes

that receive regular maintenance, and percentage o

community members able to identiy sae evacuation

routes

• Example indicator from an Oxfam programme: %

o community members who are able to reach shelters

saely and quickly

CDRC Thematic Area 5: Disaster Preparedness

and ResponseHyogo Priority 5: Strengthen disaster

preparedness at all levels

A Civil Protection Committee in Haiti runs through a training exercise which teaches them how to save lives in a ood. Abbie Trayler-Smith/Oxam

12



1313

5.4. Collecting data

For indicators to be useul, data must be collected to

show progress against them. The ollowing rules apply

when thinking about the data that you will need to collect.

•Collecttherightinformation.Mostprogrammesalready

collect too much inormation that they cannot use.

Inormation should only be collected that is essential

or infuencing decision-making and accountability and

demonstrating impact. Developing strong programmelogic and selecting a small number o eective indicators

is the best way to ensure that you only collect the most

relevant data.

•Collecttheinformationattherighttime.Thestartingpoint

or data collection is a baseline that shows the situation

beore the intervention. Data collection intervals will then

depend on the needs o the programme and its donors.

•Usetherighttoolsforthejob.Thereareanumberof

specic tools or collecting data on the elements that

make up disaster risk – i.e. hazards, vulnerability, and

capacity – which make use o existing participatory

methodologies and tools. For more guidance on whattools to use when, please email the Programme Help

Desk: [email protected].

•Planandbudgetfordatacollectioninadvance.

Collecting baseline data and carrying out annual impact

assessments require an investment o sta time and

unds. Make sure this is in your plans rom the beginning.

You will also need to make sure that annual work plans

allocate time or monitoring reviews or ongoing data

collection activities.

6. Space or learning and refection

Programme Monitoring Reviews, Country Learning

Reviews and Regional Learning Reviews can all be

used to strengthen our understanding o the impact

o our programmes on vulnerability and disaster risk.

These provide critical moments to review inormationcollected through monitoring and to ensure that this

inormation eeds into decision-making. I you are working

in a standalone DRR programme, discussions about

vulnerability and disaster are likely to eature in all o your

Monitoring Reviews. However, in programmes where DRR

is not the main objective, it can be really benecial to use

this time and space to think about how to strengthen the

impact on DRR.

7. Rigorous analysis o outcomes and impact:

evaluation

All Oxam programmes should plan to evaluate their impact and learn about how they can improve their work

in uture. The Oxam evaluation policy and accompanying

guidance are available on the Intranet and rom phd@

oxam.org.uk; they exist to help programme teams to plan

and implement evaluations across all thematic areas.

However, more specically, the ollowing have been

identied as specic goals or DRR evaluations:

Joyce Aneno Oywelo, a Public Health Facilitator (PHF) in Amida camp, northern Uganda. Home visits are used to provide support, but also to

collect data on malaria, diarrhoea and other health problems. Geo Sayer/Oxam



14

•totesttheassumptionsmadeduringprogramme

design on the occurrence and severity o hazards and

vulnerabilities;

•tounderstandtheimpact(positiveornegative,intended

or unintended) o the intervention on the vulnerability o

men and women in the target communities, and ensure

beneciary eedback;

•togeneratelearningoninterventionswhichreduceor

contribute to vulnerability, which can be incorporated intouture programme design; and

•toproduceawrittenreport,whichcancontributeto

transparency and accountability and, where relevant,

inorm lobbying activities with local or national governments.

For example, a report demonstrating the eectiveness o

certain activities in achieving a specic goal o the HFA or in

implementing national policy could be eective in lobbying

or additional government unding or this activity.

This is not an exhaustive list, and programme sta are

encouraged to contact their regional programme quality

lead or [email protected] i they would like more supportin thinking through the objectives or an evaluation.

8. Summary o key learning points rom

this Learning Companion

• MEL plans must include monitoring against core

programme indicators, moments for review, and

rigorous analysis of outcomes and impact.

• All good MEL plans are based on clear

programme logic, with high-quality indicatorsfor output and outcome levels. Impact-level

indicators are desirable but not always possible.

• An indicator is an observable or measurable

change which results from the intervention of a

project.

• The CDRC Guidance Note is a valuable input to

help develop a MEL plan that will demonstrate

progress against the Hyogo Framework for

Action.

• A matrix of good DRR indicators is available on

the ARR Intranet pages.

• Developing a small number of relevant indicators

will make it easier to collect the right data.

• Planning and budgeting for data collection

exercises is essential.

• Space to reect on monitoring data and to use

it to inuence decision-making are an essential

part of any MEL plan. Monitoring reviews can be

used to review our progress against indicators

of vulnerability to disasters in all Oxfam’s

programmes, not just standalone DRR work.

• It is vital to plan and budget for evaluations andconsider how learning from evaluations can be

disseminated and incorporated into decision-

making and future programme planning.

9. Further reading

I you are having diculty locating any o these

resources, please email [email protected] or advice.

MEL pages on the Oxam GB Intranet

http://intranet.oxam.org.uk/programme/pm/programme_

cycle_management/mel

Contains inormation about the Oxam GB MEL system

as well as tools and guidance that aim to supportprogramme teams to plan and implement MEL processes,

including developing logic models, identiying indicators,

collecting data including baseline studies, and acilitating

moments or review such as Monitoring Reviews and

evaluations.

BensonCandTwiggJ(2007)‘GuidanceNote13–

Evaluating disaster risk reduction initiatives’, Tools or

Mainstreaming Disaster Risk Reduction: Guidance Notes

or Development Organisations,ProVentionConsortium

This guidance note sets out the main steps in planning

evaluations, collecting and analysing data, and using theresults, and it discusses issues associated with these

activities.

EmergencyCapacityBuildingProject(2007) Impact

Measurement and Accountability in Emergencies: The

Good Enough Guide

This pocket guide presents some tried and tested

methods or putting impact measurement and

accountability into practice throughout the lie o a

project. It is available to download rom the Oxam

website, on CD-Rom or in hard copy.

ProVentionConsortium(2007)DRR and M&E Sourcebook

ThisfollowsonfromtheProVentionConsortiumguidance

note mentioned above and is a more comprehensive

online guide to the M&E issues o DRR. It contains many

practical examples o M&E, as well as links to useul

reerence material online and a bibliography o o-line

publications.

Roche C (1999) Impact Assessment or Development

Agencies: Learning to Value Change, Oxord: Oxam GB

This book shows how and why impact assessment can be

integrated into all stages o development programmes –

rom planning to evaluation.

For more advice on measuring the impact o DRR,

please contact your regional Programme Quality Advisor

or regional DRR or Humanitarian Advisor, or the PPT

Adaptation and Risk Reduction team in Oxord at arr@

oxam.org.uk.



Disaster Risk Reduction and Climate Change Adaptation are corporate priorities or Oxam GB. The Learning Companions are a set

o articles, which provide accessible and practical guidance to Oxam sta wishing to integrate DRR and Climate Change adaptationapproaches into programming. To nd out about other resources on Disaster Risk Reduction and Climate Change Adaptation, and to

give us your eedback on these resources, please contact the Programme Resource Centre. Email: [email protected]

Front picture: Participants get ready to apply the techniques they have learnt during Oxam’s week-long water rescue training, Haiti.Photo: Maite Alvarez/Oxam

Oxfam is a registered charity in England and Wales No 202918 and Scotland SCO 039042. Inhouse 4087

oxfam measuring the impact of disaster risk reduction learning companion

Documents