outcomemapping.ca monitoring in outcome mapping principles, design & practice steff deprez &...
TRANSCRIPT
outcomemapping.ca
Monitoring in Outcome Mapping Principles, Design & Practice
Steff Deprez & Kaia Ambrose
OM Lab Dar Es Salaam, Tanzania 22-23 Sept, 2014
Monitoring in Outcome Mapping1. Core principles of monitoring in Outcome Mapping
2. Monitoring Design Issues Experiences from practice
3. Monitoring Practice Approaches, tools & instruments
Steff Deprez & Kaia Ambrose (Sept 2014) 2
Outcome Mapping Monitoring• Systematic collection of data on outcomes and
performance• A regular learning & improvement cycle • Credit a program for its contribution to bringing about
change• Encourages the program to challenge itself
Steff Deprez & Kaia Ambrose (Sept 2014) 5
Outcome Mapping Monitoring• Flexibility• Participatory• Evaluative thinking• Organisational & social
learning• Power of self-assessment• Regular face-to-face meetings
Steff Deprez & Kaia Ambrose (Sept 2014) 6
within the broadest development context or sphere of interest
within the program’s
sphere of influence
design boldly
M&E modestly
Steff Deprez & Kaia Ambrose (Sept 2014) 9
Outcome Mapping offers a system/process to gatherdata & encourage reflection on:
1. The progress of external partners towards the achievement of outcomes (progress markers)
2. The internal performance of the program (strategy maps)
3. The program's functioning as an organizational unit (Organisational Practices)
Steff Deprez & Kaia Ambrose (Sept 2014) 10
Monitoring in Outcome Mapping
Implementing team
Boundary partner 1
Boundary partner 2
Boundary partner 3
Beneficiary 1 Beneficiary 2
Beneficiary 3
Intervention Strategies
Outcomes
Organisational Practices
Sphere of interest
Sphere of influence
Sphere of control
Focus of M&E
Viability
Efficiency & Relevancy
Behavioral changes
Steff Deprez & Kaia Ambrose (Sept 2014) 11
3 types of monitoring journals
Implementing team
Boundary partner 1
Boundary partner 2
Boundary partner 3
Beneficiary 1 Beneficiary 2
Beneficiary 3
Intervention Strategies
Outcomes
Organisational Practices
Focus of M&E
viability
Efficiency & Relevancy
Behavioral changes
Outcome Journal
Strategy Journal
PerformanceJournal
OM Journals
Steff Deprez & Kaia Ambrose (Sept 2014) 12
Critical questions?
Programme response
•What should we keep doing?
•What do we need to change in order to improve?
•Are we still working with the right BPs?
•What strategies/practices do we need to add?
•What strategies do we need to end?
•What should be evaluated in more depth?
Steff Deprez & Kaia Ambrose (Sept 2014) 14
Making learning explicit Use of Spaces & Rhythms (*)
Conventional M&E design •Information needs related to the programme framework (objectives, results, outcomes, … + indicators) •Data collection methods•Reporting
Outcome Mapping (as presented in the OM manual)Based on principles of Utilisation-Focused Evaluation
•Focus on monitoring priorities: Who will use it? Purpose?•Use of outcome, strategy and performance journals
(*) introduced by Guijt & Ortiz (2008)
Steff Deprez & Kaia Ambrose (Sept 2014) 16
Assumptions about monitoring (in OM)
• Monitoring process = learning process > reflection and analysis happen automatically
• An actor-centered design leads to a participatory monitoring process
• M&E results will be used• Users have the capacity, time and willingness to participate
or facilitate the monitoring process• Using outcome journals & strategy journals is enough to
pave the way forward • The monitoring process is embedded in organisational or
programme management cycles• …
Steff Deprez & Kaia Ambrose (Sept 2014) 18
Making learning explicit
• For learning to happen > data is not the starting point
• Start from the intended use
• Start with defining the spaces that are crucial for debate, sharing, reflection and decision-making
• Make the monitoring integral to the thinking and doing of the organisation and programme
Steff Deprez & Kaia Ambrose (Sept 2014) 19
Steff Deprez & Kaia Ambrose (Sept 2014) 20
(Source: Seeking Surprise (Guijt, 2008)
Learning-oriented monitoring
Steff Deprez & Kaia Ambrose (Sept 2014) 21
1. BE CLEAR ON PURPOSE, USES & USERS
2. DEFINE ORGANISATIONAL SPACES & RHYTHMS
3. DECIDE ON INFORMATIONNEEDS
Which information is required, for who, at what time/event,
in what form, to do what?
Three core steps in the design of alearning-oriented monitoring system
Steff Deprez & Kaia Ambrose (Sept 2014) 22
E.g. Intended uses of M&E process (Patton)
1. BE CLEAR ON PURPOSE, USES & USERS
Steff Deprez & Kaia Ambrose (Sept 2014) 23
e.g. Wheel of Learning Purposes (Guijt, 2008)
1. BE CLEAR ON PURPOSE, USES & USERS
Steff Deprez & Kaia Ambrose (Sept 2014) 24
Short-term planning
PLANNING
LEARNINGACCOUNTABILITY
Long-term & strategic planning
Programme improvement
Knowledge creation VECO & partners
Negotiation & understanding chain actors
Upward & downward accountability
Programmatic & financial accountability
Evidence building & upscaling
24
e.g. PLA system VECO
1. BE CLEAR ON PURPOSE, USES & USERS
Steff Deprez & Kaia Ambrose (Sept 2014) 25
1. BE CLEAR ON PURPOSE, USES & USERS
2. DEFINE ORGANISATIONAL SPACES & RHYTHMS
3. DECIDE ON INFORMATIONNEEDS
• What are the spaces and rhythms central to planning, learning, accountability, debate, decision-making, … (Guijt & Ortiz, 2007)
• How can we ensure that monitoring is integral to the thinking and doing of the organisation and programme?
Organisational spaces Formal and informal meetings and events which bring organisations and programmes to life
RhythmsPatterns in time, the regular activities or processes which provide a structure-in-time, through which an organisation can direct, mobilise and regulate its efforts, i.e. regular weekly, monthly, annual activities that characterise the tempo of organisational functioning.
When do people interact and share information and make
sense of what is happening?
2. DEFINE ORGANISATIONAL SPACES & RHYTHMS
Steff Deprez & Kaia Ambrose (Sept 2014) 26
Steff Deprez & Kaia Ambrose (Sept 2014) 27
Description of the main spaces & rhythms
EVENT What is thepurpose ofthe event
Time/frequency
Whoparticipates
Whocoordinates
What istheexpectedoutput
Which dataInformationis required
…
…
…
> Group exercise
Steff Deprez & Kaia Ambrose (Sept 2014) 28
1. BE CLEAR ON PURPOSE, USES & USERS
2. DEFINE ORGANISATIONAL SPACES & RHYTHMS
3. DECIDE ON INFORMATIONNEEDS
• Which data & information is required?
• What type of data / information?
• From ‘Nice-to-know’ to ‘Must-know’ information
Steff Deprez & Kaia Ambrose (Sept 2014) 30
Information needs linked to the main spaces & processes
EVENT What is thepurpose ofthe event
Time/frequency
Whoparticipates
Whocoordinates
What istheexpectedoutput
Which dataInformationis required
…
…
…
Masterlist of info needs
General info need
Specific information need? (indicators or questions)
FrequencyBy when?
Who has the info? Or where is the data generated?
Data collection method/approach?
Who will use the info at which event?
Steff Deprez & Kaia Ambrose (Sept 2014) 32
1. BE CLEAR ON PURPOSE, USES & USERS
2. DEFINE ORGANISATIONAL SPACES & RHYTHMS
3. DECIDE ON INFORMATIONNEEDS
•Plan how data is used and analysed > make it usable for action
• Focus on social interaction: sharing, reflection, debate, decision
• Should be well-planned & facilitated > it will not happen by itself
Plan for Sensemaking
How to make sure that your monitoring principles and design is translated in an effective monitoring practice?
1. Creating organisational conditions: motives, means & oppportunities
2. The ‘web of institutionalisation’
> Should be reflected in the Organisational Practices
Steff Deprez & Kaia Ambrose (Sept 2014) 33
Institutionalising a learning-oriented M&E practice
Creating the motives, means & opportunities
Creating Means-Human capacities-Specialist support-Concepts, methods and tools-Budget
Creating Opportunities-Integration in planning and management-Clear M&E plans and responsabilities-Responsive information management system-Trust and respect – speak out, challenge, feedback
Creating Motives-Guiding ideas-Support by management-Develop a learning Culture-Provide incentives
(Steff Deprez, 2009)
Steff Deprez & Kaia Ambrose (Sept 2014) 34
2. The web of institutionalisation (Levy, 2006)
See: THE PROCESS OF INSTITUTIONALISING GENDER IN POLICY AND PLANNING: THE ‘WEB’ OF INSTITUTIONALISATION (Levy, 2006)
Steff Deprez & Kaia Ambrose (Sept 2014) 35
Monitoring Practice in OM
Working with progress markers, boundary partners and organizational practices
Sense-making with boundary partnersOngoing challenges
Thinking through the different aspects of monitoring
– M&E plan (Performance Measurement Framework) – more narrative– Unpack different moments
Working with progress markers
• What is your purpose and use? • What is your monitoring culture?• What resources do you have for monitoring?• What qualitative and quantitative data needs
do you have?
Challenges• Qualitative data collection – informal
interviews, observation (including looking for unexpected – positive and negative)
• Qualitative analysis – looking for patterns and trends
• Critical analysis and sense-making – the need for facilitated, well-constructed (agenda, exercises) spaces and processes
• Usage of information
Evolving Lessons
Monitoring beyond outputsGood Enough (in terms of tools, capacity) and
build from thereM&E – mande – evaluative thinking – - explicit
sense-making spaces
Working with progress markersUse progress markers as a checklist to track progression against pre-defined behavioural changes for a specific partner in a specific period of time; use of scoring (LMH, 1234, colour)Write a qualitative description of change (I.e. every 4-6 months) for each pre-defined PM for a respective periodOther monitoring tools, qual or quan, that are then cross-referenced with pre-defined PMs (new ones added)
Working with progress markers: who?
BP describe their own change - then send to implementing teamImplementing team describes change based on their own observationsMutual reflection process with team and BPsExternal evaluator judges progression in change
Working with progress markers: what?
Every single PM monitored Only PMs that are relevant for a specific periodPMs and / or OCs used to trigger discussion during reflection process; key changes documentedDepth of analysis can varyAcross different BPs (comparison)In combination with SMs (effective intervention?)
Working with progress markers
Using progress markers at an organisational scale (across programmes)
Different geographical regions
Different thematic foci
Different type and bigger numbers of boundary partners
Use of progress markers for whom?Use of general & standard progress markers for each type of BP
-less (or not) useful for the individual projects and their actors
Tailor-made progress markers for individual BPs
+ relevant to guide and steer local projects and their actors
- less useful for higher levels, overload detailed data & aggregation is difficult
Working with progress markers
Sensemaking with BPs
Regular (4-6 months) reflection with staff + BP (one or more), external evaluator, other stakeholdersEmbedded in spaces and rhythms of the programme
Working with strategy maps
Useful for monitoring relevance, effectiveness and efficiency of the supporting activities of the implementing teamUnderstand how you spend time and money within a programme teamGet direct feedback from BPs on the required / requested support
e.g. VECO Indonesia - 9 categories of strategy maps
Every activity that is carried out by the VECO can be linked to one of the 9 support strategies.
Working with organisational practices
Use as is in manualUse of OCs and PMs at the level of the implementing organizationMotives, Means and OpportunitiesWeb of institutionalisation
Reports and Journals
Simplified journal formatsCombined PM and SM reportsIntegrate elements from journals into regular programme reports Integration in MIS, databases, etc.Used as final report, or as report embedded within another reportOM journal as a databaseSM as an activity guide
Three ways of combining LFA & OM
OM+Modifications of OM framework with elements of LFA
LFA+ Modificiations of an LFA with elements of OM
Combined uses OM frameworks that are linked to/ feed into LFA
OM+Additional layer of
objectives (+indicators) to
pinpoint exactly to what BPs are contributing
Some programmes add an ‘ouput’ layer
between strategy maps and outcomes
OM+Additional layer of
objectives (+indicators) + intermediate
tangible results (+indicators) to
clearly state what BPs are contributing
Inte rventi onlogi c
Indi cators S ou rce ofVer ifi cati on A ss umpti ons
G oal /D evel opm en t O bj
R esult s /Outc om es
Act ivi t ies
Pr oje ct Ob ject ive /Pur po se ≈ Specific objectives + indicators
≈ Outcome Challenge + Progress markers
≈ Activities VECO
Actor-centred results: results describing changes in behaviour
LFA+
Logical Framework of a specific project
OM Framework of a specific project
Inte rventi onlogi c
Indi cators S ou rce ofVer ifi cati on A ss umpti ons
G oal /D evel opm en t O bj
R esult s /Outc om es
Act ivi t ies
Pr oje ct Ob ject ive /Pur po se ≈ Specific objectives + indicators
≈ Chain results + indicators
≈ PMs BPs + Activities VECO
Combined use of LFA & OM
OM frameworks are guiding the PM&E of
specific projects
Logical framework of the over-all programme
Synthesis of the different projects