overview of impact evaluation - home | better evaluation · steps in conducting an impact...
TRANSCRIPT
![Page 1: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/1.jpg)
Overview of Impact Evaluation
Patricia Rogers
RMIT University/ BetterEvaluation
![Page 2: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/2.jpg)
![Page 3: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/3.jpg)
UNICEF Impact Evaluation Reports
![Page 4: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/4.jpg)
![Page 5: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/5.jpg)
1. I’ve conducted an impact evaluation
2. I’ve managed an impact evaluation
3. Both – I’ve managed AND conducted an impact evaluation
4. Neither – I have not managed or conducted an impact evaluation
What experience have you previously had with impact evaluation?
![Page 6: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/6.jpg)
UNICEF Impact Evaluation Briefs
![Page 7: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/7.jpg)
IMPACTS
Not only:
• direct, intended
impacts...
But also:
• Unintended – positive and
negative
• Direct and indirect – splash
and ripple
![Page 8: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/8.jpg)
Steps in conducting an impact
evaluation
1. Deciding to conduct an impact evaluation
2. Establishing governance and management
arrangements
3. Preparing to conduct the evaluation
4. Developing a Terms of Reference
5. Engaging the evaluation team
6. Overseeing the evaluation
7. Following up the evaluation
![Page 9: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/9.jpg)
1. Deciding to conduct an
impact evaluation
![Page 10: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/10.jpg)
Is there clear intended use?
PURPOSES OF AN EVALUATION
![Page 11: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/11.jpg)
Are there adequate resources?
PURPOSES OF AN EVALUATION
![Page 12: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/12.jpg)
Is the impact evaluation linked to
national and UNICEF priorities?
PURPOSES OF AN EVALUATION
![Page 13: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/13.jpg)
2. Establishing governance and
management arrangements
![Page 14: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/14.jpg)
3.Preparing to conduct the
evaluation
• Review the theory of change
![Page 15: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/15.jpg)
3.Preparing to conduct the
evaluation • Use the theory of change to inform the impact
evaluation
– identify relevant variables that should be included in data
collection
– identify intermediate outcomes
– identify aspects of implementation that should be
examined to see if the failure to achieve intended impacts is
due to a failure to implement the intervention successfully
– identify potentially relevant contextual factors that should
be included in data collection
– guide data analysis
– provide a framework for reporting findings.
![Page 16: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/16.jpg)
3.Preparing to conduct the
evaluation
• Identify and mobilize resources
– Funding for an external evaluation team
– Time of key contributors, including
governance group, review bodies, key
informant, site visit facilitators
– Access to data systems
![Page 17: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/17.jpg)
3.Preparing to conduct the
evaluation
• Decide the process for developing the
evaluation methodology
![Page 18: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/18.jpg)
4. Developing a Terms of
Reference
• Clarify purpose and objectives
• Specify Key Evaluation Questions
• Set out a design or some guidance on a
design
• Specify evaluation team profile
![Page 19: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/19.jpg)
4. Developing a Terms of
Reference
• Key Evaluation Questions linked to OECD-DAC
evaluation criteria
• Relevance
• Effectiveness
• Efficiency
• Impact
• Sustainability
![Page 20: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/20.jpg)
Impact evaluations need to:
• Identify changes – for individuals, households,
families, communities, organisations
• Understand what has caused these
• And evaluate whether these are positive or
negative – taking into account equity issues.
(not enough to just look at the average effect)
4. Developing a Terms of Reference
![Page 21: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/21.jpg)
4. Developing a Terms of
Reference
Different types of questions lead to
different design options
![Page 22: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/22.jpg)
Options for answering
descriptive questions
– Collecting data from individuals or groups
– Observation
– Physical measurement
– Existing documents and data
![Page 23: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/23.jpg)
Options for answering
causal questions
![Page 24: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/24.jpg)
Options for answering
evaluative questions
![Page 25: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/25.jpg)
5. Engaging the evaluation team
Good practices in recruiting and selecting a
team:
Knowledge of:
local
culture
local
language
data
collection
experience
evaluation
expertise
project
management
![Page 26: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/26.jpg)
6. Overseeing the evaluation
Evaluation work plan
Reporting findings
Management response
![Page 27: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/27.jpg)
6. Overseeing the evaluation
Address:
• Respect for dignity and
diversity
• Rights
• Confidentiality
• Avoidance of harm
![Page 28: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/28.jpg)
7. Following up the evaluation
• Disseminating findings
• Tracking follow-up
• Documenting lessons learned about
evaluation
![Page 29: Overview of Impact Evaluation - Home | Better Evaluation · Steps in conducting an impact evaluation 1. Deciding to conduct an impact evaluation 2. Establishing governance and management](https://reader036.vdocuments.mx/reader036/viewer/2022071003/5fc03a2c3f14b970000a7c77/html5/thumbnails/29.jpg)
UNICEF impact evaluation briefs
and animated videos
Available at www.UNICEF-IRC.org