Impact Assessment Training10/11th September 2013

Oslo

Workshop Outcomes

By the end of these two days you will have: • Developed a common understanding of what is meant by

impact assessment; and how it differs from and complements the processes of monitoring and evaluation.

• Identified key challenges that you face in conducting impact assessments

• Worked with pre-selected case studies to design an impact assessment process

• Considered how the results of impact assessments might be used in learning, reporting and being more accountable to both donors and stakeholders

• Identified ways in which you can take forward learning from this workshop.

The vicious circle

Impact

So what do we mean when we talk about Impact?

Think about an event or a person who made a significant impact on your life (a birth or death of a family member or friend, a relationship, a marriage, an accident)

– What does impact mean in relation to this?

– What concepts or ideas does it include?

Impact

How do you “assess” this impact?

• ... You try to understand the nature of the change that has taken place in you and to determine its significance in your life.

Impact Assessment

“The systematic analysis of significant and/or lasting change – positive or negative, intended or not – in the lives of target groups, brought about by a given action or a series of actions”

Its not that easy... consider Danny..

Why assess impact?

• To understand the implications of our work• To become more accountable to those we work

with (stakeholders)• To support institutional learning and decision

making and improve future work• To contribute to policy development and effective

advocacy• To help demonstrate organisational performance

Key areas of enquiry for impact

Five essential questions:• What has changed?• For whom? • How significant was it? • Will it last? • In what ways did we contribute to

these changes?

The challenge for this workshop

To be able to develop Impact Assessment processes which are:

• Simple and user-friendly• Build on existing structures and systems• Are useful for accountability both

upwards and downwards• Are useful for your organisational

learning

Session 2Relationship between M&E and

Impact Assessment

M&E or Impact Assessment?• How is the assessment of impact different

from the processes of monitoring and evaluation?

• What to do?• When to do it?

ExampleIn a project to build social housing for a local

community, for example: • Monitoring would relate to the purchase of

materials, and building the houses according to plans that have been drawn up.

• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective etc

But...It’s possible to have well-built cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine “settlements”). Thousands of “successful projects and programmes” which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts. Impact Assessment addresses the “So what?” question: as a result of our efforts, what’s changed for whom; and how significant is this for them?

Impact – so what?!Thousands of “successful” projects and programmes make no lasting difference to people’s lives.

Impact Assessment addresses the “So What?” Question

As a result of our efforts, what has changed for whom; and how significant is this for them?

Differences in Brief

Log Frame Approache.g. Skills training programme - a results chain...

Does it always work like this???

Inputs Activities Outputs Outcomes Impact

Funds, expertise

etc

Training

Improved skills

Improved livelihood

s

Less poverty

Small groups’ taskSmall UK based NGO, working through partners

in Ethiopia in EducationGoal: All children have access to free basic

education1. What high level outcomes would ensure that

this goal is achieved?

Small groups’ task• Select one outcome• Discuss what activities/outputs would lead

to the achievement of this outcome• Develop this into a mini project• Develop indicators at each level

• Use different colour post its!

Small groups’ taskLevel Detail Indicators

Impact All children have access to free basic education

Outcome 1

Outputs•....•....•.....

Analysing the resultsTo what extent are you able to assess the

key questions for impact assessment?– What has changed?– For whom?– How significant/lasting are these changes for

different target groups?– What -if anything - did our programme

(project) contribute?

Measuring impact based on results chain

Tends to be limited to assessing what you expect/hope will change...

Good for demonstrating results to donors..Positive impact Negative

impact

Expected xxxx ?

Unexpected ? ?

So...

• So how else to assess impact?• What are the challenges?• What different approaches could we use?

Answers this afternoon!

Session 3Focus on change and how to

assess it

Thinking about Change

• How does change happen?• How do/can development organisations

conceptualise this? • What is their realistic sphere of influence on

changes that do take place?

Change is..

Complex in that many different changes can take place simultaneously in people’s lives

• Continuous in that nothing in society or the environment is ever static

• Variable in pace, scale and/or over the course of any intervention

• Not necessarily lasting or sustainable.

What do we need to know to assess impact?

•Who or what was involved in the change? (e.g. individual actors or state institutions)

•What strategies were used to bring about the change? (e.g. reform, mass mobilization)

•What were the contexts that affected how the change happened? (e.g. urbanization, power relationships)

•What was the process or pathway of change? (e.g. demonstration effects, cumulative progress)

•How were our efforts connected to this?

Organisational response to this?

• Many organisations are developing Theories of Change to better support their understanding of how change happens and their role in the process

Four Key ElementsUnderstand how change happens and your role in

this

Develop Organisation/Prog

ramme change pathway

Assess the impact of your

efforts

Critically reflect on your pathway

and how you thought change

happened

How do they complement other planning and M&E processes?

• Strategic plans?• Log frames?• M&E systems?• Learning loops?

3. ToC doesn’t replace results frameworks - they make them stronger

How change happens

Organisation or Programme

change pathway

Impact assessment

Reflection and adaptation of

ToC

ToC = critical analysis process

Logic Model = Accountability

Framework which is informed by ToC

Summary of some of the differences

34

Logic Models Theories of Change

•Describe component parts of a projects • Make linear links between activities and results•Are used as a management tool

•Situate programme efforts in bigger picture of change •Surface and articulate assumptions•Explain causal pathways of ChangeAre used as a critical analysis process

Examples of Impact Assessment Frameworks:

Capacity Building to partners

Funds for partners

Funds for Capacity Building

Increased partner capacity

Activities proposed by partners

Direct

sphere

of influen

ce

Contributes to:

Long

Term

Impa

ct

Power holders at local and regional

level ensure that all women access rights,

opportunities and services

Marginalised women are

empowered and access rights,

opportunities and services

including:

• Equal access to property and

resources

• Reduction in harmful traditional

practises

• Increased representation in

local, regional and national leadership

positions

•Increased economic empowerment

• Reduction in violence and

exploitation against women?

MRDF

Contributes to:

Theory of Change – Women's Empowerment (MRDF as a stone making ripples in the water)

Contributes To change in:

Marginalised women are supported and

empowered by their families, society and

culture

Marginalised women are organised, active

and influential and play an active role in society

Marginalised women have the confidence,

knowledge and skills to access rights,

opportunities and servicesAccess to

support

Awareness of rights

Access to information

Access to resources

Access to credit

Access to training

Session 4

Approaches and strategies

• Three approaches:– Post programme: Testing logic of log frame ( impact “evaluation”)– Participatory ToC approach which is used to design monitor and assess

efforts– Research: looking back sometime later and assessing changes and their

relation to programme efforts

• Four strategies:– Build into existing M&E– Tracer and tracker studies– Ensure key moments of critical reflection– Commission a retrospective study

Plans for tomorrow

Case Study Task

Day 2

Session 1Case Study Task 1

Reflection on yesterday

Main steps in designing an Impact Assessment

1. Define purpose, approach and scope of the assessment

2. Develop/confirm theory of change and/or dimensions of change

3. Develop a list of areas of enquiry which will help you to explore impact

4. Select tools and methods for gathering and analysing relevant information

5. Decide on reporting strategies6. Make concrete plans and timelines (who, when,

how....)

The challengesTo develop a process which is:• Simple and user-friendly• Build on existing structures and systems• Are useful for accountability both upwards

and downwards• Are useful for your organisational learningAnd the challenges that we identified yesteday

ACCOUNTABILITY TO THE POOR

Case Study Task 1 In small groups:• Share the details of the programme you are

working on (you need to be clear about context, overall goal and top level outcomes)

Case Study Task 1• What will be the main purpose of doing this

impact assessment? Organisational learning? To meet donor demands? Accountability to stakeholders? For advocacy

• Which approach (or combination of approaches?) to Impact Assessment would be most appropriate in this Why?

• What will be the scope and scale of this assessment?

Session 2Case Study Task 2

Case Study Task 2:• What is your realistic “scope of influence”? Which what

areas of impact will you realistically be able to “assess”, and which areas of change will you be able to “illustrate contributions to change”?

• Based on this, what “Dimensions of Change” will you be looking to assess?

• Develop a menu of areas of enquiry which will enable you to set baselines and track progress in relation to impact

•

Being clear about your scope of influence

Sphere of indirect influence (there are other factors/actors

which influence changes you want to see)

Assessment process lessrigorous –

illustrations of impact

Sphere of direct influence

(you are working directly with target groups on

specific changes) Need to assess impact

rigorously

DimensionTask 1

Areas of EnquiryTask 2

Tools/ Sources of Information

Task 3

Direct scope of influence

e.g. Capacity of partners

•Ability to plan and deliver•Levels of technical capacity•Approaches to networking with others• Level of sustainability•Shifts in confidence

Indirect scope of influence

Areas of Enquiry and indicatorsAreas of enquiry indicators

Written neutrally to encourage expected/ unexpected

Designed to test logic of project and expected changes

Can track levels of xx, shifts in xxx, trends, perceptions, Requires open-ended questions

Session 3Case Study Task 3

Case Study Task 3• Propose a range of appropriate methods that

you could use to gather relevant data for both monitoring and assessing impact (including building on or adapting existing tools and mechanisms).

Making the process manageable and robust

• Baselines • Sample size• Triangulation• Selecting the right tools and ways

collecting useful information• Ensuring that you have asked the right

questions in the right way

Baselines• The problems?

• The solutions?– Dedicated desk research and ask others– Plan from the start and use rolling baselines– Reconstructing them

Sample sizeA good sample is one that is sufficiently large

and is unbiased.How to select?

– Mathematical?– Random?– Pragmatic?

Triangulation• You need three perspectives to ensure validity

of information gathered. Could include:– Three ways of checking same info– Asking the same question of three target groups

A word about tools and collecting information

Less is more.. • Be aware of time resources and capacity• Stick to the “old faithfuls” as the basis• Mix of methods is good• Be led by areas of enquiry

• And perhaps the hardest thing of all.. Try to enable people to tell the real story...

Checklist – use for areas of enquiry and selection of tools

In order to assess impact effectively, your areas of enquiry combined with data gathering methods needs to capture this information

1. Have there been any changes?2. How many people were affected (which target groups)?3. How were they affected (and were they affected differently)?4. Were these changes intended?5. How do they compare to baselines (have you got evidence)?6. What can be attributed to your organisational efforts?7. How confident are you in reporting these findings?

Session 4Wrap up and the way forward