Systematic analysis and synthesis in qualitative evaluationCase study evaluation of the

Oxfam Novib programme in Burundi (2005-2009)Ferko Bodnar

CDI Conference on Impact EvaluationWageningen, 25-26 March 2013

‹#›

Presentation outline

1. Purpose of the evaluation2. Oxfam Novib programme under evaluation3. Design (1): evaluation method4. Analysis and synthesis of different opinions5. Expectations underpinning design6. Expectations underpinning communication7. Factors affecting use of Evaluation results

2

Purpose of the evaluation

• Accountability: Dutch Ministry, other donors, Dutch public • Learning:

– input for strategic decisions in current and upcoming Oxfam-Novib (ON) programme in Burundi;

– input for ON thematic policies and ON strategic programme 2011-2015,

– input for Oxfam International; – support organisational learning by local partner organisations and

other stakeholders.

‹3›

Oxfam Novib programme under evaluation

Oxfam Novib (ON) worked on 5 themes:1) Sustainable livelihood: food security, income, employment,

markets, 2) Social services: health, education, 3) Security: emergency aid, conflict prevention, 4) Social and political participation,5) Gender and diversity.

2005-2009: 31 projects, 12 partner NGOs, 5.7m Euros.Post-conflict context. Weak social cohesion.Programme outcomes and impact, not outputs individual projects

4

5

6

7

8

9

Design (1): evaluation method overview

• Reconstruct intervention logic with ON staff Netherlands• Identify judgement criteria with partner organisations Burundi• Interviews with resource persons : partner organisations, other

organisations and government• Focus group discussions with beneficiaries and non-targeted households• Survey in targeted and non-targeted communities (300 households) • Restitution / synthesis meeting

• Analysis and synthesis table for different opinions

10

Design (2): Reconstruction intervention logic per theme

Impact beneficiaries

Outcome beneficiaries (behaviour)

Outcome other organisations

Outcome partner organisations (PO)

(Outputs PO)

(Activities PO)

‹11›

Analysis and synthesis of different opinions (1)

• Sources:– S: Survey – B: Discussion beneficiaries– N: Discussion non-beneficiaries, as reference– P: partner organisations, implementing ON programme– O: Other organisations (NGO)– G: Government

• Interpretation opinion:+ Confirms expected effect of ON programme□ Does not confirm expected effect, or change not attributed to ON- Confirms a change contrary to the expected effect

12

Analysis and synthesis of different opinions (2)

13

Evaluation question 1.

Judgement criterion 1.1 S B N P O G

Opinion 1 + +

Opinion 2 -

Judgement criterion 1.2 S B N P O G

Opinion 3 -

Opinion 4 + -

Draft conclusion, answering evaluation question (indicating the source : S, B, N, P, O or G)

14

1. Impact on food security?

1.1. +20% HH eating 2 meals / day S B N P O G

Number of meals has not changed □ □ (□)

Better-off, active HH increased, but poor HH reduced number of meals per day

-

Careful, HH tend to underestimate, hoping for WFP assistance

□

HH eat better now than 5 years ago, due to: + + (+)

Irrigation swamp rice fields +

Peace and collaboration +

1.2. Number of months per year with sufficient food increased

S B N P O G

Number of months with sufficient food decreased, especially among vulnerable groups

- (-)

Number of months with sufficient food increased, due to disease resistant cassava cuttings

+

Irrigation in swamp rice has increased production of actively involved participants (B), but this has not improved nutrition for the majority of surveyed households in targeted areas (S). The positive effect of irrigation is annulled by the general trend of declining food production (S, B). [incomplete]

Example:

15

Expectations underpinning design1. Individual project objectives fit in programme objectives

– Outcome and impact: +– Project outputs: -

2. Inclusion independent sources strengthens contribution to ON

– Valuable info, context, unintended effects, contribution: +– Relatively few targeted beneficiaries: -

3. Systematic presentation different opinions reduces evaluator bias

– Balancing findings, reduce bias: +– (Time consuming: -)

16

Expectations underpinning communication

1. Transparent presentation and synthesis makes conclusions more acceptable

– PO who felt their results were underappreciated could add clarifications and opinions without dominating conclusion.

– PO appreciated the overview and focus on higher level outcome and impact. Recommend this for planning

17

Factors affecting use of Evaluation results

Design ++:• Concern attribution / contribution: including independent

sources Stronger recommendations. • Concern acceptable conclusions: transparent analysis More

consensus• ON concern IOB evaluation assessment: rigorous evaluation

questions, criteria, triangulation More reliable conclusions

ON adaptation (see ON mgt response)Communication +/-:• Restitution workshop: consensus acceptable conclusions• ON management response adopted recommendations• Follow up in planning ON programme 2011-2015: limited?

18