SJI Impact Assessment 2014

Based on Impact Assessment for Development Agencies – Learning to Value Change (Chris Roche, Oxfam, digital file 2010)

Impact Assessment

Sustained changes in people’s lives brought about by a particular

intervention

Attribution“One of the critical questions that impact

assessment traditionally sets out to answer is what caused any identified change. For most projects this means trying to determine if the

changes that have occurred would have happened anyway, and the degree to which an observed change can be attributed to a given project or

programme. This is notoriously difficult.”

PurposeTo demonstrate past impact

and to improve future practice

Objectives and Indicators• Impact is the indicator to overall objective• Outcome is the indicator to specific objectives • In SJI’s case indicators are for the most part not defined (some

organic agriculture and GNH indicators, CSOFF)• -> Impact assessment exercise will help defining those indicators

(participatory) for future planning/programming/project design

Objectives and IndicatorsFor example:

Specific objective: - Minimization of waste -> To what extent has it been achieved?

Outcome indicator(s):- What are the outcomes that contributed to this change? - What outcomes can be attributed to the project?- What outcomes can be attributed to external factors?- Any negative/unintended (postive or negative) impacts of this change?

Impact Assessment Team

• SJI team! • Internal training/capacity development exercise

(no external evaluators/facilitators) • External support from Dr. Gonsalves and Dr.

Marchetti in design of impact assessment • Other capacity development required?

Units/Levels of Assessment• Individual• Household• Group/CBO• Community/village• Local CSO• Institution

ParticipantsMen, women, youth, rich, poor, beneficiaries, non-

beneficiaries, project staff, donors, local government…

Research Tools• Surveys• Interviews• Workshops/discussions• Direct observation• Participatory research• Case studies/stories

Research Tools

-> Importance of using several research tools and participants for cross-checking (triangulation)!

Research Method

• Quantitative method

Specific objective:- Minimization of waste -> To what extent has it been achieved?

Ranking on scale between 1-10 before and after programme/project implementation (with or without project)

Will at the same time generate baseline info

Research Method• Qualitative method

Outcome indicator(s):- What are the outcomes that contributed to this change? - What outcomes can be attributed to the project?- What outcomes can be attributed to external factors?- Any negative/unintended (postive or negative) impacts of this change?

Semi-structured questions

Research Method• Qualitative method

Other questions:- Suggestions for improvement of the programme/project (relevance, efficiency, effectiveness, impact, sustainability)- Potential for replication, upscaling to other parts of the district/beyond- Potential for policy uptake/impact (integration into local government policy)

Semi-structured questions

Research Method

Relevance: Real problems, needs, priorities of target group and beneficiaries, contextual appropriateness (policies)

Efficiency: Benefits compared to resources spent

Effectiveness:Results contribution to achievement of specific objectivesImpact: Projects contribution to overall objective Sustainability: Benefits of project persisting after donor support, institutional

and financial sustainability

Research Tools

“Methods should evolve and be adapted based on a mutual analysis of their strengths and weaknesses; and one should monitor the extent to which the information gathered actually has an impact - for instance, whether it leads to change in the policies or practice of the project or organisation being assessed and whether it is meaningful to all parties.”

Communication Tools

• Community feedback• Presentations and workshops with staff• Multistakeholder workshops• Report• Case studies/stories• Radio• Video• Etc.

Task of the Day

For each SJI programme/project:

Determine units/levels of assessment Determine participants Determine research tools Determine timelines

SJI Internal Impact Assessment

• Multiple impact assessments: i) SJI internal, ii) SJI partners/beneficiaries, iii) external evaluator, iv) donors

• Sometime next week • Purpose: To conduct internal impact assessment and

test out research tools