documents.wfp.orgdocuments.wfp.org/.../documents/reports/wfp280129.docx · web viewevaluation...

Decentralized evaluation for evidence-based decision makingWFP Office of Evaluation

Decentralized Evaluation Quality Assurance System (DEQAS)

Quality Checklist for Decentralized Evaluation Inception ReportVersion November 2015

[title of the decentralized evaluation]Overall

General Comments/Status

Length: report does not exceed 20 pages (excluding annexes)Accessibility: Report is written in a clear and accessible manner Report clarifies and builds on the Terms of Reference,

extending its evidence base and analysis Report provides a clear operational plan for how the

team will carry out the decentralized evaluation Report reflects a common understanding between the

evaluation team and Evaluation Manager on expectations and standards

Report demonstrates ownership of the process by the evaluation team.

Editing

Template has been followed and all elements included Table of contents is included and lists tables, graphs,

figures and annexesI R Q C V e r s i o n N o v e m b e r 2 0 1 5 P a g e 1 | 13

Tables and diagrams are used as relevant List of acronyms is included

Cover Page

Template for cover page has been followed Title of the decentralized evaluation identical to that in

the TOR Date and status of the report (draft/final) indicated on

the cover page

1. IntroductionOverall: Key information on the evaluation and the subject are included in 1-2 pages.

Expected Content Assessment criteria Comments/Status

Main objectives of the evaluation are clearly stated (accountability and learning)

Purpose of the Inception Report clearly stated

Expected users of the Inception Report clearly stated

Introduction clearly sets the scene for the evaluation, including its subject, timing and objectives.

Purpose of Inception Report clearly stated, and expected users specified

2. ContextOverall: Succinct overview of the surrounding context, as pertaining to the subject of the evaluation, setting the scene for the evaluation in 3-4 pages.


I R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 2 | 13

Overview of the geographical context directly relevant to the evaluation including:

Poverty, food security Government policies and

priorities, including policy gaps

Humanitarian issues Gender dimensions of

the context in relation to food security, nutrition situation, architecture in the country and indicators

Key external events Features of international

assistance in the area Other WFP work in the

area Work of other key actors

Contextual information is focussed and concise.

Information is relevant and important to understanding the context for the subject of the evaluation

Information is explicitly geared to the evaluation subject, rather than being generically presented

3.Subject of the Evaluation

Overall: Comprehensive description of the evaluation subject in 1-2 pages


Key features of the Information is relevant


evaluation subject should be included:

Type of intervention (operation, activity, thematic area, transfer modality, pilot project) Geographic scope of the evaluation subject Relevant dates: Approval date; start date; end date Planned outputs at design: Beneficiary numbers (planned and revised) disaggregated by gender/activity Amount of transfers (food, cash, vouchers) Any other outputs Planned outcomes at design Key activities Main partners (Government; NGOs; Bilateral; Multilateral) Resources (% funded of total requirements) and key donors. If subject funded from pooled funds, show resource allocated Assessment of the Logical Framework or similar tool from evaluation perspective or its reconstruction Other relevant

and important to understanding the subject of the evaluation:o What it iso When it was designed o What are the key

inputs ($ value) o What are the planned

outputs? (beneficiaries, MT, Cash &Voucher $)

o What is the target/scope?

o What are the planned outcomes?

o Who is involved in its implementation?

Soundness of logical framework assessed. If reconstructed, it has been discussed and agreed about with the evaluation commissioner

Highlights relevant issues from past evaluations and reviews that are relevant to the evaluation

Gender dimensions explained

Differences between original design and implementation are explained if appropriate


preceding/ concurrent activities/ interventions/operations Any amendments to initial design Gender equality and women’s empowerment dimensions relevant to subject of the evaluation and context Assessment of whether quality gender analyses were undertaken and whether this analysis was properly integrated in programme design. Include reference to: Past evaluations/reviews related to the subject Maps/graphs for illustration

4. Stakeholder AnalysisOverall: Comprehensive mapping of stakeholders, including their interests and involvement in and needs from the evaluation in 1-2 pages.


Building on the related TOR section, the stakeholder analysis should identify:o Who are the different

groups involved in the evaluation subject

All relevant stakeholders have been identified

Relevant analysis of who should be involved in the evaluation is


(including beneficiaries)o Why they have a stake

in the subject of the evaluation and the evaluation itself

o How they will be involved in the evaluation process

included along with the respective interests of stakeholder groups

Relevant analytical tools applied

Considerations regarding beneficiaries’ perspectives are included

Beneficiary analysis is disaggregated by gender

Stakeholders’ analysis is coherent with the proposed methodology and evaluation matrix

5. Evaluation approach and methodology

Overall: Methodology and specific methods present a comprehensive and systematic approach, which is sufficient to generate trust in the credibility of the evaluation as well as its independence and impartiality, in 3-4 pages.


5.1 Proposed approach and methodology Explanation and

justification of the evaluation criteria selected, and how they will be applied

Inclusion of evaluation questions to be addressed

Proposed methodological approach is coherent, logical and in line with the TOR

Methodological approach is comprehensive and presents a systematic approach that will


Full description of the methodological approach, including a mixed-methods approach

Clear description of the evaluation matrix, and how it will be used

Clear statement of how gender will be addressed in the methodology

generate trust in the credibility of the evaluation

Mixed-method approach is specified

Evaluation matrix is present; contains the required elements (see ‘expected content’, below) and meets required quality standards;

Proposals for the integration of gender into the methodology are sufficient to ensure a credible and comprehensive approach

5.2 Site Mapping

Presentation of the geographic coverage of the evaluation

Explanation of the sampling for the selection of areas to be visited (selection criteria explicit)

Assurance on the impartiality of the selection process

Relevant site mapping tool has been used to present the analysis if appropriate

The site mapping is linked to the analysis of the operation

Gender considerations explained

The analysis informs the selection of areas to be visited during the mission, according to a sound rationale and impartial approach

5.3 Data Collection Methods and Tools


Description of and justification for, specific methods to be applied

Chosen methods explicitly linked to the Evaluation Matrix and informed by stakeholder analysis

Specific consideration of how the methods proposed will address gender issues

Approaches to addressing any data gaps identified at Inception stage

Description and supply (in Annexes) of data collection tools

Description of the sampling strategy

Explanation of analytical methods to be applied, including how data will be triangulated for drawing conclusions

How data will be cleaned, where relevant

The data collection methods to be applied in the evaluation are justified and described in full, with strategies described for addressing data gaps

Consideration of how the data collection methods will address gender considerations

Consideration of how data collection activities will be undertaken in a gender-sensitive manner

Transparent presentation of data collection tools

Sampling methods are robust and impartial

Proposals for analysis, including triangulation, likely to generate credible conclusions

5.4 Limitations and risks Limitations or gaps in

evidence are presented

Indication of how the

Risks are correctly identified, and mitigation strategies are realistic

Limitations anticipated in


evaluation team will mitigate limitations

Risks are identified and mitigation strategies proposed

the evaluation due to e.g. availability of data, timing of field visits or security considerations are clearly stated, alongside how these will be mitigated.

5.5 Ensuring Quality

Description of quality assurance mechanisms to ensure the impartiality, independence, credibility and utility of the evaluation

Mechanisms for ensuring the utility (e.g. communication and learning plan in place) credibility (robust methodology, clear mechanisms for minimising bias i.e. impartiality) and independence (use of external evaluation team) are clear and explicit

6. Organization of the evaluationOverall: Comprehensive operational plan gives confidence that the evaluation can be implemented as planned, in up to 6 pages.


6.1 Team composition and workplan

Description of the expertise of each team member in line with ToR

Team expertise matches all competencies required in ToR (including gender)


requirements (including gender expertise) and their respective role and responsibilities Inclusion of a workplan for each team member in line with deliverables if appropriate Intended mechanisms for ensuring teamwork and co-ordination

There is clear complementarity among team members’ skill sets

Gender expertise is included

Tasks to be undertaken by each team member are clear and in line with consultants’ profiles

There is a clear plan to ensure co-ordination and teamwork among team members

6.2 Timeline

Clear presentation of timeline, revised if applicable, with associated deliverables

The (revised) timeline respects the time required for each evaluation phase

The (revised) timeline has been agreed with the Evaluation Manager

Associated deliverables, with specific dates, are included

6.3 Data collection mission schedule

Evaluation mission schedule (by days/team member/locations/stakeholders etc.)

Evaluation mission schedule provides a practical tool to facilitate WFP planning

List of stakeholders is consistent with stakeholder analysis

Detailed plan is presented in the annex

6.4 Information/Support Required


Description of support (logistical/ operational) required during the evaluation

List of support is clear and has been agreed with WFP Evaluation Manager

Annexes Comment Comments/Status

Annexes support and expand on text in the main report. Including: Map of intervention/project area Evaluation Matrix Data Collection Tools Evaluation Mission Schedule Documents Gathered

Relevant and up-to-date map is included Annexes listed and numbered Annexes referenced, where appropriate in the main report Data collection tools included Not all working documents to be included

Quality assessment – Evaluation Matrix

Evaluation Matrix – expected content

Assessment Criteria Comment/status

The Evaluation Matrix should provide an overview of how each of the key evaluation questions, as identified in the Terms of Reference, will be addressed. It should include: Breakdown of the

main questions into

The matrix summarises the evaluation methodology and addresses each of the evaluation questions in the TORs

The number of sub-questions is adequate to keep the evaluation team focused on answering all the main questions and attain


Sub-questions A set of indicators to

measure performance, explicitly referring to the logic model used

Possible benchmarks (including good practice standards, performance assessment of comparator agencies, etc.)

Links to the relevant parts of the methodology that will contribute to answering the sub questions

Explanation of how the findings will be triangulated

Sources of information (specifying whether secondary data will be used and where primary data is needed)

depth of analysis (i.e. not too many and not too few)

The sub-questions are developed to guide the evaluation team, but are not as detailed as a survey instrument or interview guide.

For each evaluation question, sub-questions, performance indicators (building on those in the Logic model or Logframe), possible benchmarks and sources of information are specified

The matrix clearly demonstrates that triangulation will take place

The evaluation matrix is informed by the stakeholder analysis

The matrix refers to the relevant evaluation criteria (including relevance, coherence (internal and external), coverage, efficiency, effectiveness, impact, sustainability, and connectedness) and standards (e.g. SPHERE standards)
