Decentralized evaluation for evidence-based decision makingWFP Office of Evaluation

Decentralized Evaluation Quality Assurance System (DEQAS)

Quality Checklist for Decentralized Evaluation Terms of ReferenceVersion June 2018

[Title of the decentralized evaluation]

Overall

General Comments/Status

Title: Clear without being too long, reflecting the subject of the evaluation

Length: does not exceed 15 pages (excluding annexes) Accessibility: o TOR written in a clear and understandable languageo TOR adequately emphasises the strategic and/or operational

focus of the decentralized evaluation.o TOR provides a good substantive overview of the subject of

the evaluation. o TOR provides sufficient information to stakeholders on how

the evaluation will unfoldo TOR provides sufficient information to the evaluation team

on what is expected from them.

Editing

o Template has been followed and all its elements are included in the TOR (titles)

o Acronyms are spelt out the first time they are used. o Paragraphs and pages are numbered electronically. o Cross-references are used.o Tables and diagrams are used as relevant and are

numbered.o When data or quotes are used, TOR provides sources of the

data/quotes (either directly below the table/graph or in footnotes)

1. Introduction

Overall: The introduction should provide key information about the purpose of the TOR and the facts relevant to the subject of the decentralized evaluation.

Expected Content Assessment criteria Comments/Status

Purpose of TOR Factual information about

the type of the evaluation (i.e. activity/operation/thematic area/country portfolio/transfer modality/pilot project/other being evaluated) and title

Brief description of the subject

The period covered by and scope of the evaluation

Expected content is provided. Clearly indicates Name of commissioning office Type of evaluation and title Subject of evaluation Period covered by the

evaluation Scope of the evaluation

T O R Q C – V e r s i o n J u n e 2 0 1 8 P a g e 2 | 15

2. Reasons for the Evaluation

Overall: Concise explanation of why the evaluation is being conducted, and for who.

Expected Content Assessment criteria Comments/Status

2.1 Rationale

Specify why the evaluation is to be undertaken

Specify why it is needed at this time

Specify how the evaluation will be used by the WFP commissioning office and other stakeholders

Valid rationale Clarity on how the timing of

the evaluation meets the stated needs in decision-making processes

2.2 Objectives

Specify the objectives of the evaluation (e.g. accountability and learning)

Nuance the standard text in an additional paragraph to specify whether more weight is placed on accountability or on learning and why

Standard text on accountability and learning used

Additional objectives added where appropriate and/or the accountability and learning objectives contextualised

Text has been nuanced as required if more emphasis on one objective

2.3 Stakeholders and Users

T O R Q C – V e r s i o n J u n e 2 0 1 8 P a g e 3 | 15

Specify the key internal and external stakeholders and what their stakes/interests are in the evaluation

Specify intended primary and secondary users of the evaluation, both internal to WFP and external where applicable

Indicate how beneficiaries’ perspectives (men, women, boys and girls) will be included in the evaluation process, including in relation to accountability to affected populations, and how this will be addressed

All relevant stakeholders, including beneficiaries, are identified/listed

Demonstrates understanding of stakeholder interests and concerns

Identification of users is linked to the reasons for and objectives of the evaluation

Beneficiaries are identified as stakeholders

Clear analysis of how WFP commitments to accountability to affected populations will be upheld

Stakeholder analysis is gender-responsive

3. Context and Subject of the Evaluation

Overall: Comprehensive description of the evaluation context and information on the nature of the evaluation subject

Expected Content Assessment criteria Comments/Status

3.1 Context

T O R Q C – V e r s i o n J u n e 2 0 1 8 P a g e 4 | 15

3. Context and Subject of the Evaluation

Inclusion of information about the context within which the subject of the evaluation has been/is being implemented, including: Poverty, food security Government policies,

priorities and institutional capacity

Humanitarian issues, including social protection programme/s, migration patterns and host community/social tensions

Key data and trends related to SDG 2 / SDG 17 in the context (region, country sub-national/local level)

Gender equality and empowerment of women (GEEW) dimensions of the context

International assistance in the area

Key external events Other WFP work in the area

and Work of other key actors

Contextual information is focussed and concise.

Information is relevant and important to understanding the context for the subject of the evaluation

The section focuses on trend data that is relevant, is coherent with scope and important to the subject of the evaluation.

3.2 Subject of the Evaluation

Type of activity/operation/ thematic area/transfer modality/pilot project/other subject of evaluation

Geographic scope of the

Information is relevant and important to understanding the subject of the evaluation:

What it is? When it was designed?

T O R Q C – V e r s i o n J u n e 2 0 1 8 P a g e 5 | 15

3. Context and Subject of the Evaluation

evaluation subject Relevant dates: Approval

date; start date; expected end date of the evaluation subject

Main partners (Government; NGOs; Bilateral; Multilateral)

Planned outputs at design 1. Beneficiary numbers

(planned and revised) disaggregated by sex and age/activity

2. Amount of transfers (food, cash, vouchers), outputs of technical assistance (TA)/capacity development activities)

3. Other outputs If existing, logic model or

logical framework should be mentioned. If not available it should also be mentioned

Resources (approved and funded, and % of total funded requirements) and key donors. If subject funded from pooled funds, show resource allocated

Key objectives, intended outcomes and activities

Other (from WFP and/or other actors) relevant

What are the key inputs ($ value, technical assistance (TA)/capacity development activities)

What are the key objectives outcomes and planned activities?

What are the planned outputs (beneficiaries, C&V $)

What is the total level of funding to the evaluation subject over the period under evaluation?

Availability of this information imperative for all DE TOR

Who is involved in the implementation?

Highlights relevant issues from past evaluations and reviews that are relevant to the evaluation

Gender equality and empowerment of women (GEEW) dimensions explained and whether programme design was informed by a gender analysis

Differences between original design and final design are explained if appropriate

T O R Q C – V e r s i o n J u n e 2 0 1 8 P a g e 6 | 15

3. Context and Subject of the Evaluation

preceding/ concurrent activities/ interventions

Any changes in planned implementation in terms of coverage, budget, planned beneficiaries and explanations

Relevant conclusions/recommendations from past evaluations and reviews that are relevant to this evaluation

Gender equality and empowerment of women (GEEW) dimensions as relevant

Reference whether the programme design was informed or not by a gender analysis

Maps/graphs for illustration

4. Evaluation ApproachOverall: Clarity on the scope of the evaluation, evaluation criteria and key questions, availability and quality of data available to support it, including any significant gaps; as well as the overall approach and aspects of methodology which the evaluation team will incorporate in their response to the TOR and in their proposed methodology.

Expected Content Assessment Criteria Comments/Status4.1 Scope

The defined scope should narrow down and specify

Clarity on what will be covered and what will not in

T O R Q C – V e r s i o n J u n e 2 0 1 8 P a g e 7 | 15

what will be included and not included in the evaluation, in relation to the activities of the subject of the evaluation

Specify the focus of the evaluation including: time frame, limitations, geographic areas, operations and types of activities and specific target groups (including women and girls)

Integrate gender in the evaluation scope of analysis in a way that will ensure that gender equality and empowerment of women related data will be collected

relation to the contents/coverage of the subject of evaluation

Clarity on duration/period that will be covered by the evaluation

Clear justification for the scope and focus, including selection of activities and areas

Clear justification for the target groups covered by the evaluation

Clear integration of gender within the scope

4.2. Evaluation Criteria and Questions Evaluation criteria selected

from the list and justified as appropriate and consistent with evaluation purpose and objectives

Specification of evaluation questions which address the selected criteria

GEEW issues mainstreamed throughout evaluation questions

Clear explanation of selected evaluation criteria and justification in case some criteria are not included.

Evaluation questions are clear and relevant to the subject of the evaluation

Evaluation questions sufficiently address the selected evaluation criteria

GEEW is effectively mainstreamed throughout questions

4.3 Data Availability

Identify the main sources of Clear indication of the data

T O R Q C – V e r s i o n J u n e 2 0 1 8 P a g e 8 | 15

information/data available to the evaluation team

List any gaps in the data and proposed ways for the evaluation team to deal with them

Describe data availability and quality of gender-disaggregated data, including existence of (or gaps in) data related to gender-specific outcomes.

available and their sources Limitations explained and

included in the TOR Suggested mitigation

measures Availability and quality of

gender disaggregated data and existence/ gaps of gender specific outcomes

4.4 Methodology Standard text provided in

the template. Present the overall

methodology for the evaluation outlining the criteria, impartiality requirement, and the methods that will be employed (quantitative, qualitative or mixed)

Identify key risks and appropriate mitigation/ management measures for the methodology approach proposed for further refinement during inception as appropriate.

Specify how gender issues will be addressed by gender-responsive evaluation methodology, tools and data analysis

Standard text used. Clear delineation of the

overall methodology for the evaluation outlining the data collection methods and overall approach to ensure impartiality

Key risks and appropriate mitigation/management measures for proposed approach, satisfactorily identified;

Sufficient detail to understand how GEEW issues will be addressed by the methodology

T O R Q C – V e r s i o n J u n e 2 0 1 8 P a g e 9 | 15

techniques.

4.5 Quality Assurance Standard text provided in

the template of TOR. Identify additional measures

to assure the quality of the process and product, and thus increase the credibility and impartiality of the evaluation.

Standard text used. Quality standards are

included in the TOR. Quality assurance is built

into the whole process by briefly showing how different products will be quality assured

Requirements are spelled out for the evaluation team to ensure validity and accuracy of data `

5 Phases and Deliverables

Overall: Clear statement of the overall timing and key deliverables for the evaluation.

Expected Content Assessment Criteria Comments/Status

Clarify the timing of the five evaluation phases and the deliverables including milestones

Specify if other products, in addition to the evaluation report, will be required (e.g. Power Point; Free-standing Summary, Video clip or if suggestions for innovative products are requested from the evaluation team).

Suggested table in annex 2 of the ToR Template is used

Reasonable amount of time for each of the phases/steps is provided for

Clearly identified deliverables with stated responsible persons

T O R Q C – V e r s i o n J u n e 2 0 1 8 P a g e 10 | 15

6 Organization of the Evaluation & Ethics

Overall: Clear statement on the overall organisation of the evaluation including its management arrangements, the evaluation team and required composition/competencies as well as anticipated lines of communication, timing of key deliverables and key considerations, including security, to be considered when planning to undertake the evaluation. It is important to include any potential ethical issues that have already been identified and propose mitigation measures.

Expected Content Assessment Criteria Comments/Status

6.1 Evaluation Conduct

Indicate how the evaluation will be conducted and led, including key aspects of communication

List any potential ethical issues that have already been identified and propose mitigation measures

Specify ethical considerations for the conduct of the evaluation

Include the evaluation schedule

Specific tasks and outputs are identified for team members in the evaluation schedule.

Specifies team structure and relationship with WFP Evaluation Manager

Consideration of ethical issues that may relate to the subject of the evaluation

Evaluation schedule is logical and provides adequate time for all phases of the evaluation (including consultations on draft documents), and annex 2 in the TOR is used

6.2 Team composition and competencies

Indicate how the evaluation team will be composed (including gender, nationality,

The balance of competencies of the evaluation team are

T O R Q C – V e r s i o n J u n e 2 0 1 8 P a g e 11 | 15

and if appropriate cultural background)

Key competencies and knowledge of evaluation team, including the appropriateness of the skills combination for the evaluation; language skills; and leadership skills for the team leader

Specify gender expertise on the team and appropriate skills to assess gender dimensions

consistent with the subject of the evaluation and include, for each member, expertise in one or more topics of the evaluation

Inclusion of gender parameters for evaluation team

Address fluency in the language/s spoken

Gender expertise is included Leadership, analytical and

communication competencies are spelt out for the team leader

6.3 Security Considerations

Specify any security considerations that may be relevant

Security considerations are relevant to the context for the evaluation

Security considerations are relevant to the nature of the contracting arrangements with WFP

T O R Q C – V e r s i o n J u n e 2 0 1 8 P a g e 12 | 15

7 Roles and Responsibilities of Stakeholders

Overall: Clear statement of expectations in terms of involvement of internal and external stakeholders and their respective roles and responsibilities

Expected Content Assessment Criteria Comments/Status

Describe roles of key stakeholders in the evaluation process (WFP Country/Area/Sub Office; WFP Regional Bureau; WFP Headquarters; National government stakeholders (relevant Ministries, departments, units); Partner UN agencies; Implementing partners); Office of Evaluation, Internal Evaluation Committee, Evaluation Reference Group.

Describe responsibilities of key stakeholders clearly.

All relevant stakeholders are identified, demonstrating impartiality.

Clear understanding of stakeholder interests and concerns.

Indirect stakeholders who may have a role are identified.

Clarity on measures to ensure impartiality and credibility of the evaluation, as they related to each stakeholder

Internal Evaluation Committee and the Evaluation Reference Group have been established and their roles and responsibilities have been set out

8. Communication and BudgetT O R Q C – V e r s i o n J u n e 2 0 1 8 P a g e 13 | 15

Overall: Clear statement of communication arrangements throughout the process, including how the report will be disseminated and the roles and responsibilities for communication and sharing the report.

Expected Content Assessment Criteria Comments/Status

8.1 Communication

Specify intentions in relation to the Communication and Learning Plan, and how this will be developed and include a GEEW responsive dissemination strategy, indicating how findings will be disseminated

Determine communication-related roles and responsibilities

Specify the need for translation and the language of each report, if not English

Communication-related roles and responsibilities have been set out

Communication and learning plan is well explained and taken into account in timeline and budget

Requirements for translation clearly set out

8.2 Budget

Overall: Any budgetary guidelines or limitations are clearly set out.

Expected Content Assessment Criteria Comments/Status

Standard text provided in the template of the TOR.

Clarify which contracting option will be used Clarify whether the budget will include the cost of workshops or special communication

Standard text used Clarity on the contracting

option to be used (procurement through open tender, LTA/SLA or HR action)

T O R Q C – V e r s i o n J u n e 2 0 1 8 P a g e 14 | 15

efforts if needed. Clarity on the cost elements that will/should be included in the evaluation budget such as of workshops or special communication efforts included, as appropriate.

Annexes Comment Comments/Status

Map(s) Evaluation Timeline/schedule Evaluation committee

membership Evaluation reference group

membership Acronyms Technical annexes

Assessment criteria: Complete and necessary

T O R Q C – V e r s i o n J u n e 2 0 1 8 P a g e 15 | 15