impact assessment training 10/11 th september 2013 oslo
TRANSCRIPT
Impact Assessment Training10/11th September 2013
Oslo
Workshop Outcomes
By the end of these two days you will have: • Developed a common understanding of what is meant by
impact assessment; and how it differs from and complements the processes of monitoring and evaluation.
• Identified key challenges that you face in conducting impact assessments
• Worked with pre-selected case studies to design an impact assessment process
• Considered how the results of impact assessments might be used in learning, reporting and being more accountable to both donors and stakeholders
• Identified ways in which you can take forward learning from this workshop.
The vicious circle
Impact
So what do we mean when we talk about Impact?
Think about an event or a person who made a significant impact on your life (a birth or death of a family member or friend, a relationship, a marriage, an accident)
– What does impact mean in relation to this?
– What concepts or ideas does it include?
Impact
How do you “assess” this impact?
• ... You try to understand the nature of the change that has taken place in you and to determine its significance in your life.
Impact Assessment
“The systematic analysis of significant and/or lasting change – positive or negative, intended or not – in the lives of target groups, brought about by a given action or a series of actions”
Its not that easy... consider Danny..
Why assess impact?
• To understand the implications of our work• To become more accountable to those we work
with (stakeholders)• To support institutional learning and decision
making and improve future work• To contribute to policy development and effective
advocacy• To help demonstrate organisational performance
Key areas of enquiry for impact
Five essential questions:• What has changed?• For whom? • How significant was it? • Will it last? • In what ways did we contribute to
these changes?
The challenge for this workshop
To be able to develop Impact Assessment processes which are:
• Simple and user-friendly• Build on existing structures and systems• Are useful for accountability both
upwards and downwards• Are useful for your organisational
learning
Session 2Relationship between M&E and
Impact Assessment
M&E or Impact Assessment?
• How is the assessment of impact different from the processes of monitoring and evaluation?
• What to do?• When to do it?
Example
In a project to build social housing for a local community, for example:
• Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective etc
But...
It’s possible to have well-built cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine “settlements”). Thousands of “successful projects and programmes” which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts. Impact Assessment addresses the “So what?” question: as a result of our efforts, what’s changed for whom; and how significant is this for them?
Impact – so what?!
Thousands of “successful” projects and programmes make no lasting difference to people’s lives.
Impact Assessment addresses the “So What?” Question
As a result of our efforts, what has changed for whom; and how significant is this for them?
Differences in Brief
Log Frame Approache.g. Skills training programme - a results chain...
Does it always work like this???
Inputs Activities Outputs Outcomes Impact
Funds, expertise
etc
Training
Improved skills
Improved livelihood
s
Less poverty
Small groups’ task
Small UK based NGO, working through partners in Ethiopia in Education
Goal: All children have access to free basic education
1. What high level outcomes would ensure that this goal is achieved?
Small groups’ task
• Select one outcome• Discuss what activities/outputs would lead
to the achievement of this outcome• Develop this into a mini project• Develop indicators at each level
• Use different colour post its!
Small groups’ taskLevel Detail Indicators
Impact All children have access to free basic education
Outcome 1
Outputs•....•....•.....
Analysing the results
To what extent are you able to assess the key questions for impact assessment?– What has changed?– For whom?– How significant/lasting are these changes for
different target groups?– What -if anything - did our programme
(project) contribute?
Measuring impact based on results chain
Tends to be limited to assessing what you expect/hope will change...
Good for demonstrating results to donors..Positive impact Negative
impact
Expected xxxx ?
Unexpected ? ?
So...
• So how else to assess impact?• What are the challenges?• What different approaches could we use?
Answers this afternoon!
Session 3Focus on change and how to
assess it
Thinking about Change
• How does change happen?• How do/can development organisations
conceptualise this? • What is their realistic sphere of influence on
changes that do take place?
Change is..
Complex in that many different changes can take place simultaneously in people’s lives
• Continuous in that nothing in society or the environment is ever static
• Variable in pace, scale and/or over the course of any intervention
• Not necessarily lasting or sustainable.
What do we need to know to assess impact?
•Who or what was involved in the change? (e.g. individual actors or state institutions)
•What strategies were used to bring about the change? (e.g. reform, mass mobilization)
•What were the contexts that affected how the change happened? (e.g. urbanization, power relationships)
•What was the process or pathway of change? (e.g. demonstration effects, cumulative progress)
•How were our efforts connected to this?
Organisational response to this?
• Many organisations are developing Theories of Change to better support their understanding of how change happens and their role in the process
Four Key ElementsUnderstand how change happens and your role in
this
Develop Organisation/Prog
ramme change pathway
Assess the impact of your
efforts
Critically reflect on your pathway
and how you thought change
happened
How do they complement other planning and M&E processes?
• Strategic plans?• Log frames?• M&E systems?• Learning loops?
3. ToC doesn’t replace results frameworks - they make them stronger
How change happens
Organisation or Programme
change pathway
Impact assessment
Reflection and adaptation of
ToC
ToC = critical analysis process
Logic Model = Accountability
Framework which is informed by ToC
Summary of some of the differences
34
Logic Models Theories of Change
•Describe component parts of a projects • Make linear links between activities and results•Are used as a management tool
•Situate programme efforts in bigger picture of change •Surface and articulate assumptions•Explain causal pathways of ChangeAre used as a critical analysis process
Examples of Impact Assessment Frameworks:
Capacity Building to partners
Funds for partners
Funds for Capacity Building
Increased partner capacity
Activities proposed by partners
Direct
spher
e of in
fluence
Contributes to:
Long
Term
Impa
ct
Power holders at local and regional
level ensure that all women access rights,
opportunities and services
Marginalised women are
empowered and access rights,
opportunities and services
including:
• Equal access to property and
resources
• Reduction in harmful traditional
practises
• Increased representation in
local, regional and national leadership
positions
•Increased economic empowerment
• Reduction in violence and
exploitation against women?
MRDF
Contributes to:
Theory of Change – Women's Empowerment (MRDF as a stone making ripples in the water)
Contributes To change in:
Marginalised women are supported and
empowered by their families, society and
culture
Marginalised women are organised, active
and influential and play an active role in society
Marginalised women have the confidence,
knowledge and skills to access rights,
opportunities and servicesAccess to
support
Awareness of rights
Access to information
Access to resources
Access to credit
Access to training
Session 4
Approaches and strategies
• Three approaches:– Post programme: Testing logic of log frame ( impact “evaluation”)– Participatory ToC approach which is used to design monitor and assess
efforts– Research: looking back sometime later and assessing changes and their
relation to programme efforts
• Four strategies:– Build into existing M&E– Tracer and tracker studies– Ensure key moments of critical reflection– Commission a retrospective study
Plans for tomorrow
Case Study Task
Day 2
Session 1Case Study Task 1
Reflection on yesterday
Main steps in designing an Impact Assessment
1. Define purpose, approach and scope of the assessment
2. Develop/confirm theory of change and/or dimensions of change
3. Develop a list of areas of enquiry which will help you to explore impact
4. Select tools and methods for gathering and analysing relevant information
5. Decide on reporting strategies
6. Make concrete plans and timelines (who, when, how....)
The challenges
To develop a process which is:• Simple and user-friendly• Build on existing structures and systems• Are useful for accountability both upwards
and downwards• Are useful for your organisational learning
And the challenges that we identified yesteday
ACCOUNTABILITY TO THE POOR
Case Study Task 1
In small groups:• Share the details of the programme you are
working on (you need to be clear about context, overall goal and top level outcomes)
Case Study Task 1
• What will be the main purpose of doing this impact assessment? Organisational learning? To meet donor demands? Accountability to stakeholders? For advocacy
• Which approach (or combination of approaches?) to Impact Assessment would be most appropriate in this Why?
• What will be the scope and scale of this assessment?
Session 2Case Study Task 2
Case Study Task 2:
• What is your realistic “scope of influence”? Which what areas of impact will you realistically be able to “assess”, and which areas of change will you be able to “illustrate contributions to change”?
• Based on this, what “Dimensions of Change” will you be looking to assess?
• Develop a menu of areas of enquiry which will enable you to set baselines and track progress in relation to impact
•
Being clear about your scope of influence
Sphere of indirect influence (there are other factors/actors
which influence changes you want to see)
Assessment process lessrigorous –
illustrations of impact
Sphere of direct influence
(you are working directly with target groups on
specific changes) Need to assess impact
rigorously
DimensionTask 1
Areas of EnquiryTask 2
Tools/ Sources of Information
Task 3
Direct scope of influence
e.g. Capacity of partners
•Ability to plan and deliver•Levels of technical capacity•Approaches to networking with others• Level of sustainability•Shifts in confidence
Indirect scope of influence
Areas of Enquiry and indicators
Areas of enquiry indicators
Written neutrally to encourage expected/ unexpected
Designed to test logic of project and expected changes
Can track levels of xx, shifts in xxx, trends, perceptions, Requires open-ended questions
Session 3Case Study Task 3
Case Study Task 3
• Propose a range of appropriate methods that you could use to gather relevant data for both monitoring and assessing impact (including building on or adapting existing tools and mechanisms).
Making the process manageable and robust
• Baselines • Sample size• Triangulation• Selecting the right tools and ways
collecting useful information• Ensuring that you have asked the right
questions in the right way
Baselines
• The problems?
• The solutions?– Dedicated desk research and ask others– Plan from the start and use rolling baselines– Reconstructing them
Sample size
A good sample is one that is sufficiently large and is unbiased.
How to select?– Mathematical?– Random?– Pragmatic?
Triangulation
• You need three perspectives to ensure validity of information gathered. Could include:– Three ways of checking same info– Asking the same question of three target groups
A word about tools and collecting information
Less is more.. • Be aware of time resources and capacity• Stick to the “old faithfuls” as the basis• Mix of methods is good• Be led by areas of enquiry
• And perhaps the hardest thing of all.. Try to enable people to tell the real story...
Checklist – use for areas of enquiry and selection of tools
In order to assess impact effectively, your areas of enquiry combined with data gathering methods needs to capture this information
1. Have there been any changes?
2. How many people were affected (which target groups)?
3. How were they affected (and were they affected differently)?
4. Were these changes intended?
5. How do they compare to baselines (have you got evidence)?
6. What can be attributed to your organisational efforts?
7. How confident are you in reporting these findings?
Session 4Wrap up and the way forward