evaluation terms of reference
TRANSCRIPT
Stong End Programme Evaluation TOR- September 2021 1
Evaluation Terms of Reference
Stoung Area Programme TOR Writing Date: August 31, 2021
Programme Number: 02606
Programme Phase: Final Implementation Phase
World Vision Cambodia
Stong End Programme Evaluation TOR- September 2021 2
Contents I. Acknowledgements ............................................................................................................. 3 II. Affirmation .......................................................................................................................... 3
III. III. Glossary ..................................................................................................................... 4
IV. VI. Introduction ............................................................................................................... 5
1. Evaluation Summary ............................................................................................................... 5
2. Description of Programme or Project Being Evaluated ..................................................... 6
3. Evaluation Target Audiences (i.e., For whom is the evaluation intended?) ....................... 7
4. Evaluation Type ................................................................................................................... 7
5. Evaluation Purpose and Objectives..................................................................................... 8
5.1 Evaluation Purpose: .......................................................................................................... 8
5.2 Evaluation Objectives: ...................................................................................................... 8
6. Evaluation Methodology ...................................................................................................... 9
7. 7. Limitations ..................................................................................................................... 10
8. 8. Authority and Responsibility ......................................................................................... 11
8.1 Team Members and Roles .............................................................................................. 11
8.2 Partners .......................................................................................................................... 12
9. Team Advisors .................................................................................................................. 12
10. Logistics ......................................................................................................................... 12
11. Products ........................................................................................................................ 13
12. Budget ............................................................................................................................ 13
13. Documents .................................................................................................................... 13
14. Lessons Learned ............................................................................................................ 13
15. Appendices .................................................................................................................... 13
Stong End Programme Evaluation TOR- September 2021 3
I. Acknowledgements Firstly, I would like to express deeply thanks to District Governor, Operational District of Health
Chief, District Office of Education (DoE) Chief, Health Center Chief, Commune Leaders, Commune
Council, Commune Committee for Women and Children (CCWC), Village Leader (VL), Youth Groups
(YG), Village Health Support Groups (VHSGs), Agriculture Cooperative (AC), Community Lead Child
Monitoring (CLCM), Sanitation Action Group (SAG), Water Management Committees (WMC) who
are actively participating in programme activities and also provided the consultation and advice to
develop the Terms of Reference (ToR) for Evaluation.
Secondly, I really appreciated for Stoung Area Program (AP) team such as Community Development
Facilitators (CDF) and Community Engagement and Sponsorship Plan (CESP) project coordinator who
provided necessary data for this evaluation TOR. Moreover, special thanks to M&E team and especially
Senior M&E Officer, Mr. Keo Sereivuth, who provided technical support for the whole process of
writing of the ToR.
Thirdly, I would also like to express my special thanks to management team, including Mr. Leng Vireak,
Operation Director (OD), Mr. Ny Sopharan, Associate Director (AD) and Mr. An Lyhorn, Regional
Manager (RM), of WVI-C, who always provided support for the development and review of the ToR.
Finally, we would like to give deeply thanks to Support Office, WV New Zealand to review and approve
the fund for the programme evaluation.
II. Affirmation Except as acknowledged by the references in this paper to other authors and publications, the
evaluation ToR described here consists of our own work, undertaken to advance learning, as part of
the requirements of World Vision’s Design, Monitoring and Evaluation learning system.
Primary quantitative and secondary data are collected throughout the evaluation process which
remains the property of the communities and families are described in this document. Information and
data must be used only with their consent.
Stong End Programme Evaluation TOR- September 2021 4
III. Glossary
AC Agriculture Cooperative
AD Associate Director
AP Area Program
CATCH Caring All Together for Child’s Health
CCWC Commune Council for Women and Children
CDF Community Development Facilitator
CESP Community Engagement and Sponsorship Plan
CLCM Community Led Child Monitoring
CP&A Child Protection and Participation
CSMP Child Sponsorship Management Project
CY Calendar Year
DME Design, Monitoring and Evaluation
DMQ Decision Making Question
DoE District office of Education
FGD Focus Group Discussion
FY Fiscal year
HC Health Centre
HHS Household Survey
KII Key Informant Interview
LEAP Learning through Evaluation with Accountability and Planning
M&E Monitoring and Evaluation
MY WASH Mobilizing Youth for Water, Sanitation and Hygiene
NGO Non-Government Organization
OD Operational District ( Government Health Structure)
OD Operation Director ( World Vision Cambodia)
RC Register Child
RM Regional Manager
SAG Sanitation Action Group
SO Support Office
STO Sectoral Technical Officer
TCEP Together for Child’s Education and Protection
ToR Terms of Reference
TP Technical Project
VHSG Village Health Support Group
VL Village Leader
WASH Water and Sanitation Hygiene
WMC Water Management Committees
WVC World Vision Cambodia
Stong End Programme Evaluation TOR- September 2021 5
IV. Introduction
Stoung district is located in the West of Kampong Thom province, where is approximately 52 Km from
Kompong Thom town with good condition of National Road number 6. It is bounded by Preah Vihear
province in the North, Siem Reap province in the West, Tonle Sap Lake in the South and Prasath
Balang and Kompong Svay districts in the East.
Stoung district is comprised of 13 communes with 133 villages with a total population of 110,596
people from 21,025 families. The total land area is 221,242 hectares in which farming land equal to
31,333 hectares. Moreover, the Buddhism is the official national religion and 99% of the people in the
district are Buddhist and most people are subsistence farmers. Stoung Area Programme implements
its activities in 3 communes including Banteay Stoung, Chamnar Leu and Trea communes with 27 target
villages are the primary focus area.
Stoung AP has started since October 2005 with AP assessment and design under support of WV New
Zealand. The first phase of programme covered 10 villages of the three target communes. With this,
AP team went through a long process of 1 year to conduct an assessment and design sponsorship
management and community mobilization projects for implementing in FY 2008. Later on, AP designed
02 sectoral projects, food security and education, in FY 2009 to implement in FY 2010, and finally the
program designed the WASH project in FY 2010 for the implementation in FY 2011. After starting
with 10 villages in October 2008, the program extended more target areas to 05 villages in FY 2009
and 05 more villages in FY 2010. The Stoung AP has been implemented its sector project in FY 2008
after 2 years-program assessment and design with the new approach, focused on community
mobilization and empowerment. This approach, the AP has engaged community people with
development activities in their community through the appropriated facilitation skills of the AP staffs.
In the five-year redesign of the phase 2 from 01 October 2013 to 30 Sep 2017, AP implemented
the Education project, Health & Nutrition project, food security project and Mobilize community for
Child Protection project and the Implementation Social Accountability Framework (ISAF).
Since 2018, AP has adapted LEAP 3 and conducted Decision Making Question (DMQ) with community
people, children and local authorities and key partners including DOE, OD, HC and school directors
to collect the information and issues related to child well-being. Based on the result of DMQ, Strong
Area Programme has selected Community Engagement and Sponsorship Plan (CESP) Project, and three
other Technical Projects (TPs) such as TP-Education and life skill, TP-Child Protection and TP-Integrate
Nutrition. During the new strategy revision in FY20-FY22, AP selected 2 core project models for
implementing such Child Participation & Advocacy and Education for Life and Community Engagement
Sponsorship Plan (CESP) for implementing on ward.
1. Evaluation Summary
Programme/Project Stoung Area Programme 02606
1. Community Engagement and Sponsorship Plan (CESP)
2. Child Protection and Participation (CP&A)
3. Education for Life
4. Nutrition (Heath/Nutrition and Wash)
Programme Phase Transition phase from 01 October 2017 to 30 September
2022 of Final Implementation Phase Evaluation Type End programme evaluation
Evaluation Purpose To assess the impact and progress of the programme goal and
outcome and level of sustainability, community ownership and
empowerment, as the result of this program implementation.
Primary Methodologies Quantitative:
1. Household survey (HHS)
2. Health and Nutrition Survey
Stong End Programme Evaluation TOR- September 2021 6
Qualitative:
1. Focus Group Discussion (FGD)
2. Key Informant Interview (KII)
3. Observation and field visit
Evaluation Start and
End Dates December 2021
Anticipated Evaluation
Report Release Date
February 2022
2. Description of Programme or Project Being Evaluated
The Stoung AP has started its operation in Stoung district in 2005 with AP assessment and design
under support from WV New Zealand. During the first 5-year of implementation, the programme
selected to implement five projects including Food Security Project (FSP), Water Sanitation and
Hygiene Project (WASH), Education, and Child Sponsorship Management and Community Mobilize
project (2008-2012). In 2013, the redesign of phase II for programme implementation continued to
work with key partners and primary focus area. With this, The Stoung AP selected five projects
implemented such Education, Health & Nutrition, Food Security project, Child Sponsorship
Management and Mobilize Community for Child Protection. Otherwise, AP also implemented the
Implementation Social Accountability Framework (ISAF) project from 2015 to 2022.
In this transition phase, Stoung AP has still selected 02 technical projects (TPs) such Education for Life,
Child Protection & Participation and one Community Engagement and Sponsorship Plan (CESP) Project
for implementing, ISAF, Covid-19 onward. The projects below will be evaluated with specific outcome
and models as following: Community Engagement and Sponsorship Plan: To enhance the potential for World Vision's
technical programmes to produce sustainable improvements in child well-being.
1. Communities demonstrate increasing ownership over planning, implementing and monitoring
their development and child well-being activities.
2. Effective child participation and protection strengthened.
3. Improved policy implementation (Enabling model I-SAF)
4. Objective 1: Scale up preventive measures to limit the spread of disease (COVID 19)
5. Objective 2: Strengthen health systems and workers
6. Objective 3: Support for children impacted by COVID-19 through education, child protection,
food security and livelihoods
7. Objective 4: Collaborate and advocate to ensure children are protected
Child Protection and Participation: Children and adolescents are protected from all forms of
violence and participating as agents of positive change with following outcome
1. Children and adolescents, especially the most vulnerable girls and boys, are meaningfully
participating and are engaging with local decision makers or local government to end violence
against children in their communities and countries.
2. The community, especially parents and caregivers, demonstrates attitudes and behaviours that
provide a caring and protective environment for all children and adolescent especially the Most
Vulnerable Children (MVC).
3. CP formal and informal mechanisms are collaborating effectively to protect girls and boys,
especially most vulnerable children, from abuse, exploitation, neglect and other forms of violence
4. Governments take all appropriate measures to ensure the protection of girls and boys. (NO
level)
Education for Life: Children are equipped with literacy for life with the four following outcomes
1. Increased support by families & communities for children's learning.
2. Strengthened classroom & school-based instruction in literacy,
3. Increased proportion of schools achieve "Quality Learning Environment"
4. Government systems supporting children’s education strengthened.
Stong End Programme Evaluation TOR- September 2021 7
Nutrition project has been implemented in the AP target during the previous phase and has
dropped out in FY20. Health& Nutrition will also include in the end programme evaluation. The goal
objective of Health & Nutrition is to increase in children who are healthy and well-nourished with
the following outcomes
1. Increase in children who are healthy and well nourished
2. Children are free from infection and disease
3. Caregivers exhibit good infant and young child feeding practices
4. Improved legal and policy frameworks
Programme
Direct
Participants
Type CESP Education CP&A Nutrition Total
Men 7000 3377 1991 7000 19,368
Women 8585 4237 2542 8585 23,949
Boy 4217 662 1266 4217 10,362
Girl 4206 676 1334 4206 10,422
Funding
Source WV New Zealand, Sponsorship Funding
3. Evaluation Target Audiences
The main purpose of the evaluation is to assess the impact and progress of the programme goal and
outcome and level of sustainability, community ownership and empowerment, as the result of this
program implementation toward the transformation lives of the communities and children resulting
from the programme implementation.
Additionally, this evaluation result will be clearly informed other sector programme staff and Support
Office (SO) colleagues to know how they can better provide support to AP staff and to help improve
the other future AP implementation to maximize impact in the lives of community members and
vulnerable children. To achieve the goal, the evaluation target audiences are mentioned below:
1. Local Partners/stakeholders– The local cooperation partners level including Youth Groups
(YGs), Community Small Groups (CSG), Commune councils, district government offices:
Development sector, Child Protection sector and Health sector will learn from their involvement
in community development activities and review and share achievements, challenges, gaps, and
opportunities in relation to their capacity for leadership and management, and level of
responsibility. They will play major roles in planning their community future plan.
2. World Vision in Cambodia – WV Cambodia will learn from its current development strategy,
and its impact on the well-being of community members, especially children both micro and macro
levels. The evaluation will help WV to see how and to what extent its strategies have contributed
to the changes of the lives of community members and understand the mechanism and the aspect
of sustainability.
3. World Vision New Zealand – The Support Office of WV New Zealand will be provided with
new evidence illustrating AP progress, learning and AP contribution toward improvement of child
well-being.
4. Evaluation Type
According to LEAP (Learning through Evaluation with Accountability and Planning) guidelines the
evaluation will be an end programme evaluation. The evaluation should be conducted no later than five
years after the start of first programme phase. The external consultant team will lead the whole process
of end programme evaluation from planning to write the evaluation report.
Stong End Programme Evaluation TOR- September 2021 8
5. Evaluation Purpose and Objectives
5.1 Evaluation Purpose:
To assess the impact and progress against the program goal and outcomes especially measure the level
of sustainability, community ownership and empowerment, as the result of this program
implementation.
5.2 Evaluation Objectives:
Effectiveness: To assess the progress and effectiveness of AP interventions to improve well-being
the community members
To what extent has the program achieved its goal and program objective.
What have been the factors that affect the program/project implementation?
Impact: To assess the impact of the Programme implementation contribute to the well-being of
children
What have been the changes (attitude, knowledge, and practice) of community members result
from AP intervention?
What is the long-term social change at the community level the project has contributed to?
Sustainability: To determine the sustainability of potential programme impact and results by assessing
the level of community ownership and strength of partnership with community partners by the AP.
What are the hindering factors to the function and ownership of communities, stakeholders, and
partners?
What enabling factors exist to support the function and ownership of communities?
Which organizations, groups, individual are critical in fostering sustainability and what should
their role be?
Stong End Programme Evaluation TOR- September 2021 9
6. Evaluation Methodology
Initial Planning
Document Review After reviewing the documents listed in section 13, information gaps will be determined, and the scope
of other data collection methods refined accordingly.
Quantitative Data Collection and Analysis
Household Survey DME team with cooperation with every Sectoral Technical Officer (STO) has developed generic survey
questionnaires. The generic questionnaires will be used for this quantitative data collection. The
evaluation lead will check the questionnaires against the matrix to ensure that it covers all in indicators
stated in Matrix and add more question as appropriate to the generic questionnaires. The indicators
from relevant secondary strategies will, as far as possible, be included within the survey questionnaire.
The questionnaire will be finalized by the evaluation team in consultation with the AP team.
The HHS data collection team will be formed by the AP team with assistance from the DME Officer.
Key community members and local stakeholders will be involved with household data collection
including HHS usage training.
The calculation of household survey sample size will be based on 95% confidence level, a design effect
of 1.0 and margin of error of 0.05. The sampling frame will be households with children 0-18 years and
the mothers with children under 24 months old in the AP target communities. The sample size
calculated using the actual household statistic of each subgroup.
There are 3797 households with children 0-18 years old in the AP target communities, and as the
result of the sample size calculation, 384 households to randomly select for the survey. The head of
household or spouse will be targeted for interview and if the head or spouse are not available for
interview, any members who are over 18 years old or are less than 60 years old can also be targeted
for interview. The care survey questionnaires will assess education, child protection, and economic of
the target community members
There are 1563 mothers with children under 24 months in the AP target communities, and as the
result of the sample calculation, 339 mothers with children under 24 months to randomly select for
the interview. Only the mothers will be targeted for the interview, if the mothers are not available for
the interview, the mothers with children under 24 months in the nearest household will be selected
as the replacement. The Health and Nutrition survey will assess health care of the mothers and
children.
Qualitative Data
Focus Group Discussions Focus Group Discussions (FGDs) will be held with program beneficiaries and program partners to
obtain qualitative responses and data of relevance with respect to the scope and requirements of the
Evaluation Matrix (in the Appendix X).
As standard practice for AP evaluations at least 15 Focus Group Discussions will be conducted with
programme stakeholders, including (but not limited to government authorities, Community Small
Groups, school teachers, school students, community members including Parents of RC, Registered
Children, and mothers with children under 24 months. The FGD participants will be organized
homogenously to help ensure participation and minimize the scope for domination or elite capture
during FGD. The numbers of participants for each FGD will be between 8 to 12 participants.
Stong End Programme Evaluation TOR- September 2021 10
The consultant team will discuss with the AP team to determine groups of stakeholders or partners,
then the respondents for the FGD will be randomly selected from groups identified.
Key Informant Interviews Between 10-12 key Informant Interviews (KIIs) will be held with programme stakeholders, such as
commune and district governors, relevant department representatives, peer/partner agency
representatives and some AP staff to obtain qualitative data following the information needs outlined
within the Evaluation Matrix in the Appendix A.
Field visit and observations It is expected that there will be field visits by the evaluation team to more directly observe actual
programme practice or observable results from the AP’s activities, which should also be guided by the
Evaluation Matrix. A Field visit and observation guide will be developed by the evaluation lead and
oriented to evaluation team prior to the visit. The visit will assess on the functionality and utility of
hardware/facilities that was provided by ADP such as well, latrines etc.
Qualitative Data analysis
The consultant team will use his/her own approaches in doing qualitative data analysis and the
preliminary result will be discussed with the AP team and M&E.
Evaluation findings dissemination
It is required that once the analysis has been completed; it will be summarized into easy way to
understand modules such as factsheets and share to AP team to validate the findings and explore more
understanding to the AP implementation and recommendation.
7. Limitations
Unpredictable and unstable situation of Covid-19 pandemic in the communities may prolong, so
AP will keep contact with local authorities to follow up the situation and get permission to
conduct data collection with individual or with a small group of participants. Moreover, AP will
prepare the preventing materials for using during data collection such as face mask, alcohol and
keep distancing. If the pandemic of Covid19 remain the issues that can lead to mobility restrict
and face to face interview is prohibited, the phone interview will be considered for the
substitution of face-to-face interview and the qualitative will also consider conducting through
phone interview with group of FGD members virtually from their home.
In every 5 years, Cambodia country always conduct commune election one year before national
election, so in 2022 all the communes in Cambodia usually conduct election campaign awareness
and voting event. To avoid from this duplicate event, AP will keep consult with local authorities
and plan ahead to conduct evaluation at the beginning the calendar year which will be in
December 2021.
Some of the randomly selected households might migrate out to seek jobs outside their
communities, so it might be time consuming for the surveyors that affects to the completion of
the survey as deadline. However, the evaluation team shall work with AP team, local authorities,
and community partners to ensure that those households shall be replaced before the fieldwork
for evaluation commenced.
Stong End Programme Evaluation TOR- September 2021 11
8. 8. Authority and Responsibility
8.1 Team Members and Roles
This evaluation will be led externally by consultant team with geographically guidance by AP team with
the participation of M&E officer to help ensure the quality of the data collected. M&E officer will ensure
the quality of the evaluation for the whole process of evaluation from writing TOR, planning and data
collection and the evaluation report.
The detail roles and key partners are in the table below:
Evaluation
Phase
Responsible
Person Primary Task
Planning
AP team
Together with DME officer, develop ToR
Refine ToR, budget and schedule with evaluation team
leader.
Obtain the list of target respondents for random selection
to collect the data. (List of households with children 0-18
years and list of mothers with children under 24 months)
Gather all relevant document for the consultant review
after the consultant selected
Senior M&E
Officer
Review and finalize the TOR and send to Regional Manager
to review and send to Support for further comments and
endorsement of the TOR.
Participation in the recruitment of qualified consultant to
lead the whole evaluation process
Procurement and
People & Culture
Prepare the announcement to recruit consultant
Coordinate in selecting the shortlist of the applicants
Coordinate with applicants and the panel to interview
selected consultant for the interview
Prepare the contract for the consultant
Data
collection
AP team
Inform the authority on the schedule of the evaluation and
get their approval to do data collection in the communities.
Prepare data collection schedule with consultant team
leader
Consultant team
leader
Provide training on qualitative data collection tools and
skills to data collection team
Closely monitor the data collection team to ensure quality
of data
Conduct the analysis and interpretation of technical data.
Provide training on quantitative data collection tools to all
enumerator to understand data collection process and the
tools
Closely monitor the data collection to ensure quality data
Develop Database for data encoding using any software
he/she familiars with. Data cleaning and analysis
Regional Manager
(RM)
Troubleshoot any management issues arose during the
course of the evaluation.
Support the analysis and interpretation of technical data
Stong End Programme Evaluation TOR- September 2021 12
Evaluation
Phase
Responsible
Person Primary Task
Reporting
and follow-up
Consultant
Formulation of recommendation for the redesign with
WVC staff through preliminary findings presentation to AP
and Management team
Prepare the evaluation report
Circulate the draft for comments and inputs and feedback
Send final Report to AP, RM and OD for review and
comments
RM, M&E and AD
Provide input to the findings
Develop practical recommendations
Review and approve final evaluation report
AP team Provide input to the findings for validation
Develop practical recommendations
8.2 Partners
Partner Role SO Input into Evaluation TOR
Review Evaluation Report and provide feedback
AP stakeholders Will fully participate in the evaluation process as the informants
9. Team Advisors
Here is the proposed Team advisor to ensure the quality of evaluation process and report.
No. Position Roles
1 Associate Director
of WVC
The Associate Director will provide overall support and oversight
for coordination of the evaluation, especially for trouble shooting any
structural or systematic issues.
2 Senior Manager –
ME quality assurance
The Senior Manager DME will provide overall guidance to the
evaluation and ensure that it follows LEAP guidelines and meets
donor expectations and approves the evaluation design (namely the
evaluation matrix and sampling framework). He will approve
evaluation ToR and matrix as well as any significant changes.
3 Regional Manager The RM will provide overall support with regard to evaluation data
collection processes, logistic and other needs identified by the
consultant Team.
10. Logistics
The logistics for the evaluation will be strongly supported by Admin and Finance Assistant (AFA),
driver, project coordinators, and so everything is managed such as meeting rooms, Computers, LCD,
handouts, refreshment, etc.
No Responsible
staff Materials needed Description of Responsibilities
1
Community
Development
Facilitator and
Project
Coordinator
List of project partners, HH
statistic, and village maps, list of
CSGs, list of wells/pond, list of
local NGOs, record of capacity
self-assessment,
All relevant documents must be in
hand before starting process.
Involve in the evaluation process
and data collection.
Stong End Programme Evaluation TOR- September 2021 13
11. Products
The key findings will be organized into a summary sheet by project and share this to the ADP and
management team to validate and or further review findings and to explore more recommendation.
The final evaluation report will be circulated to line management of WVC as well as to SO for
feedback.
The report from all ADP team, line management and support office will be incorporated into the
final report.
12. Budget
The funding for the programme evaluation has already allocated in CY22 budget. The total budget for
whole process of evaluation is XXXXX US Dollars. Please refer to the embedded file in the appendix
for more detail budget breakdown for this evaluation.
13. Documents
The major documents that should be reviewed in the document review phase of the evaluation as
below:
1. AP design documents including sector Design documents
2. AP progress report, Semi and Annual report.
3. Sponsorship Monitoring Report
4. Sector Evaluation report
5. Evaluation report
14. Lessons Learned
The evaluation planning and process will be recorded and documented properly including training, filed
work and other key activities into minutes. The evaluation team will meet regularly to do a reflection
after completing each key activity through using development tool. The lesson learned will be
incorporated into the evaluation report and will be reflected by the M&E team to improve evaluation
practice for the next evaluation.
15. Appendices