exchange1 “most significant changes”
Post on 18-Jan-2018
219 Views
Preview:
DESCRIPTION
TRANSCRIPT
03/02/2005 MSC @ Exchange 1
“Most Significant Changes”
03/02/2005 MSC @ Exchange 2
Adapted from: Rick Davies - MandENEWS Jessica Dart – Clear Horizon
03/02/2005 MSC @ Exchange 3
“If you knew what was going to happen in advance every day you could do amazing things. You could become insanely wealthy, influence the political process et cetera. Well, it turns out that most people don't even know what happened yesterday in their own business. So, a lot of businesses are discovering they can take tremendous competitive advantage simply by finding out what happened yesterday as soon as possible“ (Steve Jobs, Fortune, 1994:23)
03/02/2005 MSC @ Exchange 4
03/02/2005 MSC @ Exchange 5
Why stories? People tell stories naturally - indigenous
Stories can deal with complexity and context
People remember stories
Stories can carry hard messages /undiscussables
But stories not known for accuracy/truth
03/02/2005 MSC @ Exchange 6
Use of stories in MSC Collection of stories + systematic,
collective interpretation = storytelling can be effectively harnessed for participatory evaluation
Because interpretations tell another story & process has beneficial outcomes for evaluation utilisation
03/02/2005 MSC @ Exchange 7
MSC Form of qualitative, participatory M&E
Based on ‘stories’ of significant change
Developed by Rick Davies 1996 - Bangladesh
Now used in numerous development programs and in the public sector
03/02/2005 MSC @ Exchange 8
MSC
• Creates space for stakeholders to reflect, to make sense of complex changes
• Provides dialogue to help make sense of each other’s values
• Facilitates dynamic dialogue ie. “what do we really want to achieve and how will we produce more of it?”
• Excellent for participatory programs with diverse, complex outcomes, & multiple stakeholders
03/02/2005 MSC @ Exchange 9
Purpose of MSC in M&E Primary purpose to facilitate improvement
by: focusing direction of work towards explicitly
valued directions eg. what do we really want to achieve and how
will we produce more of it? Contributes to summative evaluation:
Information about unexpected outcomes Performance information concerning very best
success stories Can inform criteria used to judge projects
03/02/2005 MSC @ Exchange 10
Qualitative vs quantitative monitoringQuantitative
Focus on measurement Closed questions About ‘proving’ Easy to aggregate Deductive Static Goal displacement can
be a problem
Qualitative
Focus on questioning Open questions About learning Hard to aggregate Inductive Dynamic Goal displacement is
not an issue
03/02/2005 MSC @ Exchange 11
Qualitative monitoring
Can be used in conjunction with conventional output monitoring
Is usually more aimed at learning than accountability
03/02/2005 MSC @ Exchange 12
Indicators & their limits
Four types of possible outcomes:
They are expected They are unexpected
They have an agreed meaning
1. Indicators are useful here
2. Unlikely to be developed here
They have a disputed meaning
3. They might be developed here
4. Indicators cannot be used here
03/02/2005 MSC @ Exchange 13
Goal displacement
Not about learning
Don’t tell you what you don’t know you need to know
Limitations of indicator based monitoring
03/02/2005 MSC @ Exchange 14
How to aggregate complex experience?
Example: a bowl containing 2 oranges, 3 apples, 4
bananas, and 1 mango Summary-by-inclusion
There are 10 pieces of fruit in the bowl Find lowest common denominator = fruit
Cost: Loss of interesting detail Summary-by-selection
The mango is rotten, it will spoil the rest of the fruit. Remove it, please
03/02/2005 MSC @ Exchange 15
Practice Now! Turn to your neighbour and ask them
What was the most significant change that took place as a result of the workshop this week? (Get the details)
Then ask them why they thought this was the most interesting.
Document: description (who, what, where, when) explanation (why is it significant) who documented the story (name, position location, date)
Then let your neighbour ask the same questions to you.
03/02/2005 MSC @ Exchange 16
Then… Come to a decision about which of
the two stories you think is most interesting, and identify why you both think so. You may have a number of reasons.
When asked to, tell a group of 8 people the story you chose, and why you did so
03/02/2005 MSC @ Exchange 17
Then… Come to a decision about which of
the 4 stories you think is most interesting, and identify why you all think so. You may have a number of reasons.
When asked to, tell the large group people the story you choose, and why you did so
03/02/2005 MSC @ Exchange 18
The core of MSC A question:
“In your opinion what was the most significant change that took place in ….over the … months”
[describe the change and explain why you think it is significant]
Re-iteration of the same kind of question “Which of these SC stories do you think is the
most significant of all?” [describe the change and explain why you think it is
significant]
03/02/2005 MSC @ Exchange 19
Explaining MSC The first challenge when introducing
it Can be difficult because it is very
different to conventional methods Make use of direct experience Use metaphors Highlight the key differences Explain in terms of stages
03/02/2005 MSC @ Exchange 20
Using metaphors Organisations as newspapers,
with journalists, sub-editors, editors, senior editors, etc
Stories get passed up the hierarchy, but only a few make it to the front page, and only one to the top of the front page
Organisations as amoeba, sensing positive and negative
experiences and moving to and away from those respectively.
03/02/2005 MSC @ Exchange 21
Organisations as amoeba
03/02/2005 MSC @ Exchange 22
How is MSC different? Participants have a choice about what
sort of information to collect
Uses diverse rather than standard data
Information is analysed by all participants, not simply by a central unit
Subjectivity is used rather than avoided
03/02/2005 MSC @ Exchange 23
MSC vs quantitative monitoringQuantitative Focus on
measurement Closed questions Project out About ‘proving’ Deductive Static Inclusive Central tendencies
MSC Focus on questioning Open questions Context in About learning Inductive Dynamic Selective Outer edges of
experience
03/02/2005 MSC @ Exchange 24
Goals
Goal-based evaluation
change
change
change
change
change
change
viewpoint participants
Extent to which they were achieved
From the view point of the program staff + consultation
Program out Context in
From the viewpoint of the Participants
Program
03/02/2005 MSC @ Exchange 25
Explaining MSC in stages1. Defining Domains of Change2. Define reporting period3. Collecting SC stories4. Selection of collected SC stories5. Feedback of the choices made6. Verification7. Quantification8. Meta-monitoring and secondary analysis9. Re-settings of MSC system
03/02/2005 MSC @ Exchange 26
1. Defining “domains” Opposite of SMART indicators?
Like newspaper sections: sports, finance, leisure, business, etc
Defined by how people use them Examples:
“changes in peoples’ lives” “changes in relationships with our partners” “changes in government policy on HIV/AIDS
03/02/2005 MSC @ Exchange 27
Defining domains… Not essential but
Can help structure the selection process
Can help focus on goals of concern Their use tells us how what goals mean
to participants Options
Open window domain Negative changes domain
03/02/2005 MSC @ Exchange 28
2. Set the reporting period “In your opinion what was the most
significant change that took place in ….over the … months
Period used by NGOs varies from 2 weekly, to monthly, to three monthly, and yearly. Three monthly is most common
Time demands on staff is the main constraint on frequency
03/02/2005 MSC @ Exchange 29
3. Collecting SC stories From those closest to the event’s of
concern. But do not exploit people’s unpaid time
Basic format: Description (who, what, where, when) Explanation (why is it significant) Who documented the story (name,
position location, date) Option: Recommendation
03/02/2005 MSC @ Exchange 30
Collecting SC stories… Reminder: Key parts of the question
“Looking back over the last month…” “…what do you think was…” “…the most significant…” “…change…” “…in the quality of people’s lives…” “…in this community?”
03/02/2005 MSC @ Exchange 31
4. Selecting SC storiesS
A G M
B C D E F H I J K L N O P Q R
03/02/2005 MSC @ Exchange 32
Funder meeting
State meetings
Region 1 Region 2 Region 3 Region 4
Story tellers
feedbackflow ofstories
03/02/2005 MSC @ Exchange 33
Selecting SC stories… Task is to read through and identify the
most significant of all the submitted SC stories. Take one domain at a time
Need to decide who to involve: story providers, their superiors, their peers,..
Need to decide whether to predefine selection criteria, or let them emerge through discussion of SC stories
03/02/2005 MSC @ Exchange 34
Selecting SC stories… Must (not optional)
Document what SC was selected Why it was selected Process used to make the selection
Participants Their preferences
[Subjectivity is made accountable through transparency]
03/02/2005 MSC @ Exchange 35
5. Feedback To immediate providers of SC stories,
on what was selected, why selected and process used
Enables adjustment of focus of MSC next time around
A motivational factor
Weakest point in all M&E systems, including MSC
03/02/2005 MSC @ Exchange 36
6. Verification of SC stories What
Factual content & interpretation of facts Why
Encourages some discipline in reporting Enables elaboration and further learning
When When SC story first enters system When selected as MS of all SC When SC stories are publicly used
03/02/2005 MSC @ Exchange 37
7. Quantification Within the SC story
Number of people, events, etc involved
As once–off follow-up to SC story How many other cases like this known
Within meta-monitoring (see next) How many other SC stories like this
03/02/2005 MSC @ Exchange 38
8. Meta-monitoring and secondary analysis Keep all SC stories on record
Meta-monitoring (Recommended) of Changes in numbers of SC stories, who
provides them, whose SC stories are selected, changes in percentage of negative stories
Secondary analysis (Optional) by Categorising and counting of types of
changes reported, and types of explanations given, at different levels
03/02/2005 MSC @ Exchange 39
03/02/2005 MSC @ Exchange 40
9. Re-setting of MSC process
Frequency of reporting
Definition of domains to use
Who sorts SC stories into domains
Selection process design: participants & process used
Feedback and follow up
03/02/2005 MSC @ Exchange 41
Where to use MSC? Talk to your neighbour, and
identify where you think MSC Would be most useful, and why Would be least useful, and why
Share this view with the whole group, when asked
03/02/2005 MSC @ Exchange 42
Where to use MSC Not as a stand-alone method Alongside indicator based systems To identify unexpected changes To engage people in analysis of
change To involve a wide range of people To focus on outcomes rather than
outputs
03/02/2005 MSC @ Exchange 43
Finding out more about MSC Original MSC paper (n’th version) is at
http://www.mande.co.uk/docs/ccdb.htm
MSC Mailing list is at http://groups.yahoo.com/group/mostsignificantchanges
Rick Davies at rick@mande.co.uk
top related