most significant change (msc) handouts

8/19/2019 Most Significant Change (MSC) Handouts

1/76

The “Most Significant Change”Technique

Nur Hidayati & Maaike Esselink


2/76

“Most Significant Change”

2


3/76

What will we learn?

• Introductions

• Expectations, ground rules

•MSC and M&E

• MSC steps and practical exercises

• Significant change stories analysis

• How to use MSC in M&E system

• Training Evaluation

3


4/76

Assignment 1 - Monitoring and Evaluation –

what is it and how do we do it?

Individual

1.1. What is Monitoring?

1.2. What is Evaluation?1.3. What do you monitor and evaluate in a programme you

are working on?

1.4. Do you experience any challenges in monitoring and/or

evaluating your program using your indicators so far?

Write your answers down and discuss in the class in the plenary

4


5/76

Qualitative vs quantitative monitoring

Quantitative

• Focus on measurement

• Closed questions• About ‘proving’

(accountability)

• Easy to aggregate

• Deductive• Static

• Goal displacement can be aproblem

Qualitative

• Focus on questioning

• Open questions• About learning

• Hard to aggregate

• Inductive

• Dynamic

• Goal displacement is not anissue

5


6/76

Most Significant Change Technique

• Form of qualitative, participatory M&E

• Based on ‘stories’ of significant change

•‘Monitoring without indicators’

• Developed by Davies 1996 - Bangladesh

• Now used in numerous development programs and in thepublic sector

• NOT a stand-alone approach to M&E

6


7/76

Why use MSC?

• Monitor and evaluate complex project

• Identify changes

• Improve staff’s skills in observation and analysis

• Identify strategic direction

• Understand project better and improve ownership

• Provide learning process

•Improve accountability

7


8/76

Purpose of MSC in M&E

• Facilitate improvement by focusing direction of work (what toachieve and how)

• Contributes to summative evaluation: – Information about unexpected outcomes

– Performance information on very best success stories

– Can inform criteria used to judge projects

8


9/76

Impact MSC

• Staff gained more fully shared vision

• Process boosted their morale – ownerships of project

•Beneficiaries, staff, collaborators sitting together andinterpreting qualitative data & casting evaluative

judgements

• Project committees became better at conceptualising

impact

9


10/76

Why stories?

• People tell stories naturally

• Stories can deal with complexity and context

• People remember stories• Stories can carry hard messages /un-discussabled

• But stories not known for accuracy/truth

10


11/76

What is Significant?

• Relative

• Important to story tellers

• A central part of MSC to surface individual values

•NOT necessarily about HUGE or BIGGEST changes

• Can be positive or negative

• Difficult to get

How to get? (probing, local language, list of changes)

11


12/76

Steps of MSC

1. How to start and raise interest?

2. Establishing ‘domains of change’

3. Defining the reporting period

4. Collecting and writing stories of change

5. Selecting stories of change

6. Feeding back the results of the selection process

7. Verification of stories

8. Quantification

9. Secondary analysis and meta-monitoring

10. Revising the system

12


13/76

Basics of what MSC involves

1. Determine sorts of change to monitor (domain of

changes)

2. Collect of significant change stories

3. Systematic selection of the most significant changes by

beneficiaries, stakeholders and/or staff

4. ‘Search’ for project impact (regular and in-depth

discussion) – part of content analysis

5. Communication results of selection process (feedback)

6. Monitor and review the process

13


14/76

Establish Domains of Change

• ‘Newspaper’ look

• Categories of SC stories:

a. Changes in quality of liveb. Changes in participation in development activities

c. Changes in sustainability of organizations

d. Any other changes

14


15/76

Hand-out 1: SSQ project

15


16/76

16


17/76

Planning the use of domains

1. Will they be used at all?

– or just a general “most significant change”

2. Who will select the domains to be used?

3. What will the domains be?

– Must use simple and clear words, that fit within theMSC question

4. Will you ask for negative changes?

– or “changes” – regardless whether positive or negative?5. Must the MS changes be caused by the developmentinitiative?

– or significant changes regardless of cause?

17


18/76

Assignment 2 –

Developing Domains of Change

In groups of 5

Work with the case of the SSQ program

2.1. List as many as possible domains of change

2.2. Choose domain(s) of change to be used in the story

collection process

2.3. Why do you select these domains of change?2.4. If you will implement MSC, whom will you involve in

selecting domains of change?

Write your answers down and discuss in the plenary18


19/76

Collecting SC stories

1. Whose stories will be collected?

2. Who will collect these stories?

–The same people who are in the MSC stories?

– Or others outside the story? Who and why?

3. How will they be recorded?

– Note book, audio tape, video, or…?

– How much time will this take?

– And how and where will they be stored? Why?

19


20/76

Stories Collection: the who and how?

Who are the story tellers?- Program participants/beneficiaries- Project staff- Project partners

- Stakeholders

How to collect stories?

- Individual interview- Focus group discussion (FGD)- Staff project writes stories they know about (activity journal)- Individual writes their own story (beneficiaries journal)

- Video (recorded story) 20


21/76

The core of MSC

• A question: – “In your opinion what was the most significant change that

took place in ….over the last … months?”

• [describe the change and explain why you think it issignificant]

• Re-iteration of the same kind of question – “Which of these SC stories do you think is the most

significant of all?” • [describe the change and explain why you think it is

significant]

21


22/76

Whose stories to collect?

• Depends on the Domains of Change• Stories from those directly involved can be the most

powerful, but they can be more difficult to collect

• May be best to start with fieldworkers who are in contactwith them• Also depends on whose perspective you want to make more

visible and open to discussion: staff or clients?

22

Funder meeting


23/76

Funder meeting

State meetings

Region 1 Region 2 Region 3 Region 4

Story tellers

feedbackflow of

stories

23


24/76

The voice of the story-teller

• Important to get the story…….

• You may need to re-phrase the question

(“examples of....” “a time when…..”) who?

where?

when?

what happened?

why significant?

24


25/76

What to document in a story? (1)

• Information on who is the story teller, who collect the storyand when Description of the story – changes that happened

• Significance (to the story teller) of the events described in thestory

• The process how changes happened

• Use a story collection form

25


26/76


1. Beginning: Things were bad2. Middle: Then the project came

3. Ending: Then things got better

To understand the significance of the change from ‘bad’ to ‘better’, the

two situations need to be described.

For instance, if somebody tells you that they used to have to walk to

school, but now it’s much easier because they can take a bus, what

does it actually mean?

•How long did they have to walk?•What were the implications of this for their life?

•How is the situation different now that they take the bus?•What difference has this made beyond the time saved?•What are they able to do with the extra time?•This extra detail helps the reader to understand the ‘significance’ of the change

26


27/76


Beginning Middle EndWhat was the situation

before the change

happened?

(background/

context e.g. before

they participated in xxx

project or received

Support)

What happened?

(the nature and type of

support provided)

+

The situation after .

What difference did it

make?

PLUS:

WHY WAS

THISSIGNIFICANT?

(captures the

perspective of

the storyteller

and why theyvalue the

change)

27

The story contains three central parts and ends by explaining

why the story teller believes it to be significant


28/76

Probing for a story

• Initial question is basis:

“Looking back over the 3 months, what do you think was the most

significant change in the attitudes of health staff to young people?”

• In reality: ask additional probing questions to ‘capture’ thefull story

28


29/76

Ethics

• Explain how the stories will be used, at the time of collection• Consider the time cost to unpaid participants• Consider the interests of 3rd parties mentioned in stories

• Take special care if stories are to go in media• Allow anonymity but maintain verifiability!• Use consent tick boxes on MSC forms, verbal consent with

witnesses

29


30/76

Example of Story of Change

Hand-out 2

30


31/76

Who writes SC stories

• ‘Beneficiaries’ write their own stories directly

• Hired people to interview beneficiaries

• Fieldworkers write down unsolicited stories they have

heard from beneficiaries

– Or they set up interviews/focus groups to elicit stories

• Through group discussion

– Of beneficiaries - self-organised or facilitated

– Of field workers

31


32/76

How to write MSC stories (1)

• At least:

• Who documented the story (name, position location,

date)

• Description of what happened (who, what, where,

when)

• story form, describing the sequence of events

• Explanation of why is it significant

• bring out the meaning of the story

• Use quotes to give the voice of the story teller

• Give a title to the story32


33/76

How to write MSC stories (2)

• Optional:

– Headline

–Recommendations (from storyteller)

– Length: Long enough to be verifiable, short enough for a

group to read many stories

• Beware:

– Over-specifying what is needed.• Fragmenting the story, losing the plot!

33


34/76

Assignment 3 - Making a story collection

guideline Work in pair

3.1. Develop MSC core questions for the story collectionguideline using the domains of change as agreed upon in the

plenary session

3.2. Prepare a story collection guideline based on the

questions made in 3.1.

Please use the backside of this paper or a separate paper

Discuss the guidelines in class to come up with a final story

collection guideline 34


35/76

Assignment 4 - Significant Change story

collection and story writing Trio with 2 or 3 training participants and 1 resource person from the

SSQ project. Roles:

1 training participant: interviewer

1 training participant: note taker

1 training participant: observer

Resource person from SSQ: interviewee

4.1. Conduct an interview with one resource person as assigned, using thestory collection form just developed

4.2. Write a story of change using a laptop in couples after the interview

4.3. Verify the content of the story with the resource person from SSQ

after the story has been written

Reflect on the interviewing process in the plenary35


36/76

Reflection on interview process

What was challenging What was rewarding

Interviewer

Interviewee/story teller

36


37/76

Reflection on story writing process

What was

challenging

What was rewarding

Preparation

During the writing Process

Finalizing the story

37


38/76

Assignment 5 -

Tips and Tricks for collecting and writingstories

Groups of 2/3 (same group during story collection)

5.1. Reflect on the process of collecting a story from the SSQ

resource person using the MSC story collection guideline?

5.2. Reflect on the process of writing the story of the SSQ

resource person using the MSC story collection guideline?

Write your answers separately on coloured paper and report

back in the plenary

38


39/76

Story Selection

• Why selection

• Selection process

•Reason for choice

• Selection committee roles and structure

• Mini workshop

39


40/76

Why selection?

• The heart of MSC

• Encourage discussion

•Part of analysis

• Reduces volume of stories in systematic way

40


41/76

How to do story selection?

• Focus group discussion (facilitated discussion)• Questions: which one is the most significant change and why?

• Transparent – always document reasons and process in eachlevel of selection

• Various way to arrive to a final decision:

(a)Negotiation

(b)Score/ranking

(c)Using criteria of selection

(d)Voting

41


42/76

Stories Selection process

1. Use a diagram to show flow of stories and feedback

– Who will be involved?

– Who will send selected stories to who?

–How many levels of story selection?

– How many stories go to and from each level?

– Who participates in each level?

– And explain these choices

2. How many person-days will the process take?

– Each selection round, on average?

– From start to finish - reporting, selection, feedback cycle?

42


43/76

SC Stories Selection Process

43


44/76

Using a layers of elected representatives, or

layers of existing authorities?

44


45/76

Or parallel processes of selection?

45


46/76

Selection committee roles

Two roles:

1. Reporting

2. Monitoring/reflection

46


47/76

Selection committee and structure

Issues:

• Composition of selection committee (representativeness,gender, etc)

• Consider practicalities of involving partners/beneficiaries

• Staff have difficulties to be neutral from stories they collected

• Structure and timing of the collection/selection

Discussion:How to deal the above issues?

47


48/76

Group Exercise: Mini Workshop

Class into 3 groups

Setting: Story Selection Committee of program staff

Divide roles for: one facilitator to lead the selection process, one note taker,

one time keeper and discussion participants

• Read all stories: make sure all participants know the content of thestories

• Clarify unclear information in the stories together

• Select the story reflecting the Most Significant Change by discussion

using the story selection form provided

• NOTE: Make sure notes of the process of selection are made

Reflect on the selection process and outcome in the plenary

48


49/76

Providing Feedback

• Why important

• Who provide feedback to and what sort of feedback

•How to provide feedback

49


50/76

Why Feedback?

• Often overlooked but essential

• To ensure findings from MSC is shared as part of M&E

•Facilitates learning and for improvement

• Improving accountability

• Opportunity to give acknowledgement

50


51/76

What sort of feedback?

• Results of selection

• Comments on stories

•Recommendations for action

• How we have used the stories

51


52/76

To whom feedback is provided?

• Community (beneficiaries/target groups)

• Story tellers

•Story collectors and selection committee

• Partners

• Stakeholders

52


53/76

How to provide feedback?

• Verbal or written

• Various ways:

a. via meetings

b. Emailc. Newsletters/printed materials

d. Reports

e. Exhibitions (photos, movies/film, etc)

f. Arts (drama, mimes, etc)

Just be creative!

53


54/76

Stories Verification

• Which stories will be verified? Any?• By who?• And why?

– Check and correct facts?

– Check participants’ interpretations of the event?

– Develop into a detailed case study for organisationallearning?

Recommended only for selected stories

54


55/76

Assignment 6 - Using the results of MSC

Groups per table

Discuss on the following questions:

6.1. How will the MSC stories be used in yourorganization/program:

Internally?

By beneficiaries/partners?

By donors?

By others?

6.2. Which of these uses is most important for yourorganisation/program? Why?

Report on your discussion and answers in the plenary

55


56/76

Analysing SC Stories

• How and where will records be kept of all SCs that arecollected?

• What questions are important to ask about the whole set ofstories?

• How SC stories should be analysed and who will do the

analysis?

• Who will use the results of the analysis?

56


57/76

• Primary analysis:

– What participants do when they select MSc stories. This is

the core of MSC

• Secondary analysis

– Further analysis of MSC stories by other means

– May or may not involve original participants

MSC Analysis: What is the difference

57


58/76

Primary Analysis

• What are the type of changes?

• Select the MOST significant change story

• Always document reasons for choices in selection process

• Analysis positive and negative changes (if any)

• In what level changes happen? (individual, community,organization, etc)

58


59/76

• Meta-monitoring:

– Tracking who provides the stories, and who does not

– Whose stories get selected, or not

•Content analysis

– Looking into the content of the stories

• Selected and/or unselected stories

• Basic Quantification, what can be quantified?

• Against logic model (or Theory of Change)• Is the changes ‘one-off’ or often happen?

• How the changes happen and how we observe them?

E.g. the use of Progress Marker (see hand outs)

Two types of secondary analysis

59


60/76

Groups per table

• The group sorts stories into 2 piles based on the mostimportant difference and document the reasons why they do so

• Each of these piles are sorted into another 2 piles and again thereasons for the sorting process is documented.

• Continue this process of sorting to end up with story(ies) whichcannot be sorted anymore. Document the reason why sorting is

not possible anymore (Question: Is it necessary?)

Make notes of all reasons for sorting and of the process

Assignment 7 – Hierarchical Card Sorting

60


61/76

A definition of content analysis…

“Content analysis has been defined as a systematic,

replicable technique for compressing many words of text

into fewer content categories based on explicit rules ofcoding”

http://pareonline.net/getvn.asp?v=7&n=17

61

http://pareonline.net/getvn.asp?v=7&n=17http://pareonline.net/getvn.asp?v=7&n=17


62/76

• Meta Monitoring:

– Find out how to improve coverage of stories.

• Diversity of sources matters

• Content Analysis:

– Track specific issues of concern without changing the

ongoing MSC process

– Identify wider trends that may not emerge through MSC

selection process

Why do it?

62


63/76

Resources

www.mostsignificiantchange.com

63


64/76

• Record

– Who collects each story, from whom

– Who was involved in each selection process

• Even their “votes” for specific stories?

– Types of MSC stories of interest e.g neg

Meta-Monitoring: key tasks

64


65/76

Introducing MSC in an Organization

• Key resources to learn from

– Build on prior experience

• Finding champions and drivers

• Pre-testing

• Training

• Considering duration and sustainability

65


66/76

Champions and Drivers

• Who will ‘authorise’ the use of MSC

– and who will enthuse others?

• Who will push and pull to make it work?

– answer questions about the method

– help design the proposed use of MSC

– help plan or run training – monitor and guide the MSC process

66


67/76

Pre-testing MSC

• Advisable in large organisations or projects or ‘new’ alliances

• Enables early identification and resolution of problems

• Creates enthusiasm/support

• Helps customise MSC process

67


68/76

Duration and Sustainability

• MSC does not have to be used for ever!

– E.g.

• When used as part of an evaluation or planning

process• When a project or action research process ends

• If you do want to sustain its use then…

– Refresher training

– System for on-going adaptation in order to meet currentneeds

– Dedicated resources and people

68


69/76

Think about your context

• Who might be a good champion in your organisation, or evenoutside it?

• Who could drive the use of MSC

• What sort of training would be needed?• What sort of a pre-test would be useful?• How long will you need to use MSC?

69


70/76

Assignment 8 - Using MSC in your project

Groups of 5 participants

You are interested in using MSC in your programme/organisation

8.1. What issues need to be considered for implementation of MSC in your

programme/organization?

8.2. How you will implement MSC in your programme/organisation? List

all steps

8.3. Discuss the use of domains

Make notes of your discussions. Share observations and queries in plenary

Make notes of all reasons for sorting and of the process

70


71/76

MSC steps (full version)

1. How to start and raise interest?

2. Establishing ‘domains of change’

3. Defining the reporting period

4. Collecting and writing stories of change5. Selecting stories of change

6. Feeding back the results of the selection process

7. Verification of stories

8. Quantification

9. Secondary analysis and meta-monitoring

10. Revising the system

71


72/76

What will be

measured

M&E

Outputs - Quantitative and qualitative data

- Monthly reporting- Output indicators

- Information traceability

Outcomes - Quantitative and qualitative data

- Design monitoring technique and tools

- Semi-annual or annual reporting

- Sampling method

- Outcome indicators

- Information traceability

Impact Evaluation; Triangulation among M&E data (output,

outcome and MSC data) to identify evidence forchanges happened and possible impacts on the

long run

Mostsignificant

changeQualitative analysis

Learning


73/76

Assignment 9 - MSC in the M&E of yourorganisation

Groups of 5

9.1. Discuss how timing and integration of MSC in your

organisation’s/program’s M&E framework might work

9.2. Where do you expect there will be most need for adjustment of yourM&E framework? Why?

9.3. Where have you already made changes to the design? Why?

9.4. Draft your plan of action on integrating MSC in your

organisation’s/program’s M&E framework

Use a flipchart to draw and/or explain the integration

Make notes of your discussions.

Share observations and queries in plenary and present your plan of action

73


74/76

More info for Assignment 9

Timing and Integration

1. How often will SC stories be collected?

– Weekly, monthly, quarterly, yearly…

– Pros and cons for most choices

2. How will MSC related to existing systems?

– How will it build on existing P,M&E system?

– What extra requirements will there be?

74


75/76

Tuning the system

• Possible features to modify

– Domains

– Frequency of collection

– Structure of selection processes

– Who participates

– What happens to the SC stories

–Other?

75


76/76

Key resources to learn from

• MSC Guide (2005) athttp://www.mande.co.uk/docs/MSCGuide.htm

• http://www.mande.co.uk/docs/MSCGuide.pdf

• MSC Mailing list is at http://groups.yahoo.com/group/mostsignificantchanges

Including Files section

Google “Most Significant Change”
http://www.mande.co.uk/docs/MSCGuide.htmhttp://www.mande.co.uk/docs/MSCGuide.htmhttp://www.mande.co.uk/docs/MSCGuide.pdfhttp://groups.yahoo.com/group/mostsignificantchangeshttp://groups.yahoo.com/group/mostsignificantchangeshttp://www.mande.co.uk/docs/MSCGuide.pdfhttp://www.mande.co.uk/docs/MSCGuide.htm

most significant change (msc) handouts

Documents