somalia program support servics (spss)

Somalia Program Support Services (SPSS)

AWARD NO: IDIQ AID-623-I-14-00009

FINAL REPORT

SUBMITTED: 05/28/2021

First Submission: May 28, 2021

Resubmitted: June 21, 2021 (508 Compliant)

This report was produced for review by the United States Agency for International Development (USAID) and prepared by International Business & Technical Consultants, Inc. (IBTCI), under AID-280-TO-17-00001.

SOMALIA PROGRAM SUPPORT SERVICES (SPSS) FINAL CONTACT COMPLETION REPORT

Award No: IDIQ AID-623-I-14-00009

First Submission: May 28, 2021

Resubmitted: June 21, 2021 (508 Compliant)

Prepared for Joshua Vetter, COR United States Agency for International Development/Somalia C/O American Embassy United Nations Avenue, Gigiri P.O. Box 629, Village Market 00621 Nairobi, Kenya

International Business & Technical Consultants, Inc. (IBTCI)

In the US: 8618 Westwood Center Drive Suite 400 Vienna, VA 22182 Tel: +1.703.749.0100 In Kenya: 8th Floor, Block C Westcom Point Building Mahiga Mairu Road, off Waiyaki Way Nairobi Tel: +254.20.3740370

DISCLAIMER

The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

i

TABLE OF CONTENTS

TABLE OF CONTENTS ....................................................................................................................... I

LIST OF FIGURES .............................................................................................................................. II

LIST OF TABLES ................................................................................................................................ II

ACRONYMS ....................................................................................................................................... III

INTRODUCTION .................................................................................................................................. 1 A. OVERVIEW OF SPSS .............................................................................................................. 1 B. SOMALIA COUNTRY CONTEXT .......................................................................................... 1 C. OVERVIEW OF SERVICES .................................................................................................... 3

KEY ACHIEVEMENTS AND CHALLENGES BY COMPONENT................................................ 3 A. CORE/SUPPORT SERVICES ................................................................................................... 3

KEY ACHIEVEMENTS ................................................................................................................... 3 CHALLENGES ............................................................................................................................... 4

B. PEFORMANCE AND DATA MANAGEMENT ..................................................................... 5 KEY ACHIEVEMENTS ................................................................................................................... 5 CHALLENGES ............................................................................................................................... 6

C. DATA QUALITY ASSESSMENTS (DQA) ............................................................................. 7 KEY ACHIEVEMENTS ................................................................................................................... 7 CHALLENGES ............................................................................................................................... 9

D. EVALUATIONS AND ASSESSMENTS ............................................................................... 10 KEY ACHIEVEMENTS ................................................................................................................. 10 CHALLENGES ............................................................................................................................. 11

E. MONITORING, VERIFICATION, AND REPORTING ........................................................ 15 KEY ACHIEVEMENTS ................................................................................................................. 15 CHALLENGES AND LESSONS LEARNED ................................................................................. 17

LESSONS LEARNED ......................................................................................................................... 24

RECOMMENDATIONS FOR FUTURE WORK ............................................................................ 25

ANNEX 1: SPSS TASK ORDERS 2014–2021 .................................................................................. 28

ANNEX 2: DETAILS OF CUMULATIVE SPSS ACHIEVEMENTS ........................................... 29

ANNEX 3: LIST OF EVENTS & MEETINGS ................................................................................ 62

ANNEX 4: SPSS INDICATOR PERFORMANCE TRACKING TABLE..................................... 64

ANNEX 5: PHOTOS FROM SITE VISIT MONITORING ........................................................... 67

ANNEX 6: SUMMARY OF USAID PROJECTS PERFORMANCE DATA ................................ 68

ANNEX 7: SPSS FACT SHEET ........................................................................................................ 88

ANNEX 8: MV&R GEO-TAGGED SITES ...................................................................................... 89

ANNEX 9: EVALUATION DATASETS .......................................................................................... 90

ANNEX 10: POWERPOINT OVERVIEW OF LIFETIME OF SPSS .......................................... 91

ANNEX 11: LIST OF DELIVERABLES ........................................................................................ 105

ii

LIST OF FIGURES

FIGURE 1: TOTAL SPSS SITE VISITS BY STATES ...................................................................................... 17 FIGURE 2: LINKAGES BETWEEN DATA MANAGEMENT PROCESSES AND DATA QUALITY

STANDARDS ............................................................................................................................................. 39 FIGURE 3: TOTAL SPSS SITE VISITS BY STAT ........................................................................................... 52 FIGURE 4: MODULES COVERED DURING THE INDUCTION WORKSHOPS ......................................... 55 FIGURE 5: BLOOM’S TAXONOMY USED TO CLASSIFY PARTICIPANTS’ SELF-STATED

EXPECTATIONS ........................................................................................................................................ 57 FIGURE 6: SPSS FLAGGING SYSTEM ........................................................................................................... 61

LIST OF TABLES

TABLE 1: DQAS INDICATORS NUMBER, ACTIVITIES, AND AREAS VISITED ...................................... 7 TABLE 2: LIST OF EVALUATIONS AND ASSESSMENTS DONE OVER THE SPSS LOP ....................... 10 TABLE 3. LIST OF AMELPS REVIEWED ...................................................................................................... 33 TABLE 4: DQAS INDICATORS NUMBER, ACTIVITIES, AND AREAS VISITED .................................... 36 TABLE 5: DATA QUALITY RISK SCORES ................................................................................................... 42 TABLE 6: DATA QUALITY STANDARD RISK SCORES INTERPRETATION .......................................... 42 TABLE 7. DQA DELIVERABLES BY YEAR .................................................................................................. 43 TABLE 8: LIST OF EVALUATIONS AND ASSESSMENTS DONE OVER THE SPSS LOP ....................... 44 TABLE 9: SPSS FLAGGING STATUS BY IP .................................................................................................. 52 TABLE 10: SPSS VERIFICATIONS BY SUB-ACTIVITIES OVER TIME .................................................... 53 TABLE 11: SPSS VERIFICATIONS BY USAID SOMALIA ACTIVITIES OVER TIME ............................. 54

iii

ACRONYMS

ABE Alternative Basic Education ADS Automated Directives System AFRICOM AMELP

Africa Command Activity Monitoring, Evaluation, and Learning Plan

AMISOM AOR

African Union Mission in Somalia Agreement Officer’s Representative

AVF African Voice Foundation B&M Branding and Marking BUILD Bringing Unity, Integrity, and Legitimacy to Democracy Activity CDCS CO

Country Development Cooperation Strategy Contracting Officer

COP Chief of Party COR Contracting Officer’s Representative COVID-19 CPARS

Coronavirus Disease 2019 Contractor Performance Assessment Reporting System

DARS Data and Research Solutions DCOP Deputy Chief of Party DEC Development Experience Clearinghouse DEL# DQA

Deliverable Number Data Quality Assessment

EAJ EFS

Expanding Access to Justice Emerging Federal States

ETD Environmental Threshold Determination FGD Focus Group Discussion FGS FM

Federal Government of Somalia Field Monitor

FMS FS

Federal Member States Field Supervisor

FY Fiscal Year GEEL Growth, Enterprise, Employment, and Livelihood GoK HO

Government of Kenya Home Office

HoP BFC IBTCI

House of Parliament Business and Finance Committee International Business and Technical Consultants, Inc.

IDIQ Indefinite Delivery Indefinite Quantity IED IEE

Improvised Explosive Devices Initial Environmental Examination

IOM International Organization for Migration IP Implementing Partner IPC Integrated Phase Classification IPTT Indicator Performance Tracking Table IR IRG IS

Intermediate Result International Resource Group Islamic State

iv

ISIS IYCF KII LOP M&E

Islamic State of Iraq and Syria Infant and Young Child Feeding Key Informant Interview Life of Project Monitoring and Evaluation

MV&R Monitoring, Verification, and Reporting NGO Non-governmental Organization OTI Office of Transition Initiatives PEG Partnership for Economic Growth PIKA Pool of In-Kind Activities PMP Performance Management Plan PO Program Office PPR Performance Plan and Report Q2 Quarter 2 Q3 Quarter 3 Q4 Quarter 4 R1 Round 1 R2 Round 2 R3 Round 3 RF Results Framework SAP SNA SO SORADI

Systems Assessment Protocol Somalia National Army Sub-Objective Social Research and Development Institute

SOW Scope of Work SPSS Somalia Program Support Services SRS Strategic Review Session SSG Strengthening Somali Governance SYLI Somali Youth Learners Initiative TIS+ Transition Initiatives for Stabilization Plus Activity TO Task Order TOC Theory of Change TPM Third Party Monitor UNICEF United Nations Children’s Emergency Fund USAID United States Agency for International Development USG United States Government UVIED Under Vehicle Improvised Explosive Device VBIED Vehicle-Borne Improvised Explosive Devices WHO World Health Organization YR 6 Year 6 YR 7 Year 7

1

INTRODUCTION

A. OVERVIEW OF SPSS

In 2014, the United States Agency for International Development (USAID) awarded the Somalia Program Support Services (SPSS) contract [Indefinite Delivery/Indefinite Quantity (IDIQ) AID-623-I-2014-00009] to International Business & Technical Consultants, Inc. (IBTCI). The primary goal of SPSS was to provide monitoring, verification, and evaluation support services to the USAID Mission for Somalia. SPSS was originally contracted to be implemented from August 2014 to July 2019, after which USAID Somalia approved a five-month no-cost extension from August 2019 to December 2019.

In December 2019, IBTCI received an 18-month cost extension of the IDIQ AID-623-1-2014-00009. A new Task Order (TO) #72062320R00004 incorporating Monitoring, Verification, and Reporting (MV&R VII) and Data Management and Support Services was subsequently awarded. It was implemented from January 1, 2020, to June 30, 2021. Under this TO, IBTCI provided support services, strategy support, performance management, activity level MV&R, data quality assessments, and data management services. These services are described in subsequent sections of this report.

This final project report covers the entire Life of Project (LOP) of SPSS from August 2014 to June 2021, providing an overview of the activities conducted, key achievements, challenges and lessons learned of the contract, as well as recommendations for the implementation of future monitoring and evaluation (M&E) support activities in Somalia and other Missions with similar contexts.

The SPSS close-out in the field is scheduled for June 30, 2021, when all SPSS expatriate staff will leave the project, all SPSS expendable (EXP) and non-expendable property (NXP) will have been disposed of, and the SPSS offices will close. An exit plan documenting how the close-out will be conducted was approved by the Contracting Officer (CO) on May 26, 2021.

Over the LOP, IBTCI worked with various sub-contractors, most notably Management Systems International (MSI) for the communications function, and the Development Gateway (DG), for adapting and delivering the clearinghouse (knowledge management platform) based on prior work conducted under the predecessor project. In 2014-2015, IBTCI worked with mFieldwork and Ecotech on software issues regarding the input of encrypted data at the field level into customized survey forms on hand-held devices for M&V. In 2020, IBTCI engaged Africa Voices Foundation (AVF) to implement TO#72062320F00007 – Citizen Consultation and Gender Assessment 2020.

B. SOMALIA COUNTRY CONTEXT

The SPSS project was implemented under very difficult circumstances in Somalia described as follows:

Geopolitical space: Somalia has had a history of poor governance, power struggles, and weak institutions, which has led to a state structure with limited legitimacy. The inability

2

of the government to sustain state-led security, control territories, manage revenues, provide services, or organize effective institutions has consistently undermined the various repetitions of national governance, resulting in a patchwork of local governance arrangements and private provision of public services. Clan identity and fragmented politics is an enduring feature of the country and the manipulation and politicization of clan identities sparks violent conflict throughout the country. Attempts to ensure inclusivity of clans in key state-building and programming initiatives have also come under criticism for reinforcing, rather than overcoming, clan identities with different leaders surrounding themselves with clan loyalists. For example, the Federal Government of Somalia (FGS) and Federal Member States (FMS) were to conduct both parliamentary and presidential elections based on an agreement reached on September 17, 2020, but due to an impasse between the incumbent president and his opponents, the process is yet to take place.

Terrorism: Al-Shabaab has been the most visible terrorist group in Somalia because it controls large parts of Southern and Central Somalia, with frequent attacks in the capital of Mogadishu. Al-Shabaab militants regularly carry out bombings, ambushes on government facilities, and civilian movements, abductions, kidnappings, and illegal arrests, including targeted killings, contributing to the loss of life and livelihoods. The threat of terror attacks has been increasing daily even with the presence of international peace forces from Ethiopia, Kenya, the African Union Mission in Somalia (AMISOM), Turkey, the U.S. Africa command (AFRICOM), Britain, and other actors within Somalia. However, SPSS managed to achieve its assigned tasks without suffering any attacks.

Infrastructure: Somalia has underdeveloped infrastructure in terms of roads, communication, and power. Most of the roads are not paved, with one main tarmacked highway linking the north to the southern parts of the country through the capital Mogadishu. This makes travels out of the capital difficult, especially during rainy seasons when regions outside the capital become inaccessible due to flooding. Roads are dusty with poor visibility during dry seasons. Extremist groups erect illegal roadblocks, and stage abductions and forced detentions. Clan militias often charge illegal road use levies. Travel outside the capital Mogadishu heavily relies on either scheduled flights or security escorts. Somalia has about ten paved civilian airfields and fewer than 20 additional gravel airfields. Power supply is heavily reliant on diesel- or petrol-powered generators. Some rural areas have no access to power. The public telecommunications system was destroyed or dismantled by civil conflict. While communication capacities have improved with local cellular telephone systems, many rural areas remain out of reach as insurgent groups frequently destroy communication masts installed within their control zones. However, international connections are available outside Mogadishu by satellite. Working within these conditions, SPSS was able to use air transport, and secure roads to guarantee staff safety. It relied on a satellite network where there was no access to internet or local cellular networks.

Clan-based Conflict: Somalia is comprised of three heterogeneous areas: the Federal Government-controlled area (South/Central); the semi-autonomous regions (FMS – Federal Member States) that includes Puntland; and the independently governed region of Somaliland. Civil wars in the 1980s and 1990s led to a weakening of the government and the military, and allowed for the rise of independent military groups that have vied for control of the country since. Autonomous armed groups, including clan militias, private domestic

3

security forces, armed criminal groups, and government security forces compete to control territories and interests, and frequently clash with one another. Even with repeated conflicts throughout the contract implementation period, however, the SPSS project goals were fulfilled through regular security training and advice and use of local monitoring partner firms with controlled field travel.

The COVID-19 Pandemic: The first known case of COVID-19 in Somalia was recorded in mid-March 2020, and the disease has since spread across all regions of the country. The pandemic led to the closure of borders and suspension of international travel, curfews across regions, and the imposition of other guidelines to prevent the spread, although these restrictions were later lifted in August 2020. A swelling second wave was reported beginning in March 2021, leading the government to reintroduce guidelines including closure of learning institutions and suspension of political gatherings. The pandemic has therefore had a direct effect on project operations, and the SPSS team had to adopt virtual and remote working modalities in conducting Activity Monitoring, Verification & Reporting, Data Quality Assessments, and supervision of data collection to minimize the chances of infection. Despite the challenging operational environment, the IBTCI-SPSS team successfully accomplished its project deliverables without any major incident reported by staff, local monitoring partners, or implementing partners (IPs).

C. OVERVIEW OF SERVICES

Over the life of project (LOP) of SPSS, IBTCI offered a suite of technical, operational, and logistical services to USAID Somalia, which included Assessments and Evaluations; Communications and Outreach; Data Management; Data Quality Assessments (DQAs); Monitoring, Verification, and Reporting (MV&R); Performance Management; and Strategy Support. These services were defined, organized, and implemented under various Task Orders (TOs) over the LOP. Details of these TOs can be found in Annex 1 of this report. A cumulative overview of key achievements per component and the applicable methodologies used can be found in Annex 2.

KEY ACHIEVEMENTS AND CHALLENGES BY COMPONENT

A. CORE/SUPPORT SERVICES

Key Achievements

Communications TO: IBTCI provided a Communications Advisor to the Mission who collected information from IPs to produce fact sheets, success stories, snapshots, and other documentation at the activity, Intermediate Result (IR), and Development Objective (DO) levels. SPSS produced activity fact sheets for all USAID/ Somalia activities including the Joint Health and Nutrition Program implemented by UNICEF; the Partnership for Economic Growth (PEG) activity implemented by DAI; the Strengthening Somali Governance (SSG) activity implemented by Chemonics; the Somali Youth Learners Initiative (SYLI) implemented by Mercy Corps; and the Transition Initiatives for Stabilization (TIS) program implemented by both DAI and the International Organization for Migration (IOM). IBTCI developed a USAID/Somalia Portfolio Overview factsheet, produced success story documents, wrote speeches and talking points for the Mission Director, accessed the regular

4

messaging toolkits that were distributed by USAID’s Legislative and Public Affairs Bureau and helped set up regular social media feeds on Twitter and Facebook.

Strategy Support TO: SPSS was asked to provide support for the Somalia Stabilization Strategy through a stand-alone task order including background research and consultations and leading a Results Framework workshop. Three expert seminars with Somalia think tanks and subject matter experts in 2015 helped SPSS draft strategy documents for USAID. USAID accepted and approved the final deliverable of the task order on July 28, 2015.

USAID Clearinghouse Support: SPSS enhanced the pre-existing web-based online knowledge management system accessible to United States Government (USG) agencies and the Somali government. The clearinghouse helped USAID/Somalia offices and staff to monitor, verify, and evaluate the performance, results (outputs and outcomes), impact, and environmental compliance of USAID programs and activities. SPSS trained Mission staff in its use, and enhanced it with GIS mapping software. The Clearinghouse allowed partners to input data (disaggregated by activity and location) and USAID to review patterns of progress and deviations in the context of its Results Framework. The Clearinghouse used open-source software and included a data entry module to facilitate IPs in their ongoing updates. In 2015, SPSS coordinated with USAID Mission ICT staff for the data download and handover of software and dependencies and testing the deployment in the local environment. USAID found the Clearinghouse, ultimately, to be an unnecessary expense and the TO ended mid-way through SPSS. Should a clearinghouse-like mechanism be used in the future, it should be more user-friendly for USAID and IPs.

Event Organizing: IBTCI was tasked under its Core Services TO with providing logistical and administrative support, and facilitation services when needed. SPSS arranged and secured meeting locations and facilitated some of the meetings. Throughout the SPSS project, IBTCI hosted over 60 meetings and events. Annex 3 provides a listing of all meetings and events hosted by IBTCI over the SPSS LOP.

Challenges

Communications Task Order: A major challenge throughout this TO was the lack of a clear and concise organizational structure. It was not always clear who was tasked with management and oversight of the Communications Advisor because the role was to be embedded at USAID. While the SPSS Contracting Officer’s Representative (COR) was the point of contact with USAID, at times technical direction and approvals came or were needed from Mission Communications personnel.

At the start of the TO there was a delay in getting the Communication Advisor housed in the Mission. This created confusion among all parties about to whom the Communication Advisor was to report and from whom to receive guidance. IBTCI mitigated this by working with the USAID Somalia Mission to establish a clearer management structure in which the COR provided technical oversight and general guidance for the TO within the Mission. IBTCI also worked with USAID to ensure the Communications Advisor attended biweekly meetings with the COR, and ensured both parties had common understanding of issues,

5

monitored progress, and solved problems. The TO began in September 2014 and ended in September 2015 when USAID hired their own communications specialist.

Strategy Support Task Order: The first Strategy Support Specialist hired by IBTCI led several interviews and conducted a document review in support of USAID’s evolving new strategic plan for Somalia 2016-2019. She left her assignment because of concerns over the worsening conflict in Mogadishu and her reluctance to travel there. SPSS sought a modification of the TO and USAID authorized the replacement of the Strategy Support Specialist. The new specialist successfully completed all the deliverables within the assignment.

B. PEFORMANCE AND DATA MANAGEMENT

Key Achievements

SPSS provided the following knowledge management services to the Mission:

1. Support for the Mission’s Theory of Change (TOC) and Results Framework (RF), and new strategy review. In FY 2016, SPSS began participating in various meetings on RF review under specific programs. Those included a series of sessions with TIS+ AECOM to discuss their RF and their draft Performance Indicator Reference Sheets (PIRS). The sessions also reviewed indicators along the hierarchy of objectives, and made amendments where necessary. SPSS continued TOC and RF review support throughout its implementation. In FY 2019, SPSS helped USAID Somalia to finalize the Expanding Access to Justice (EAJ) TOC and RF in two sessions held at the EAJ Head Office in Nairobi. Over its implementation, SPSS provided technical support to the Mission on strategy review of ongoing activities in Somalia. This support included analysis of MV&R reports and performance reports including Indicator Performance Tracking Tables (IPTTs) to generate key findings and conclusions about performance and context that USAID used to organize strategy review sessions with IPs.

2. SPSS provided support in baseline data collection by compiling information from IP reports and from other secondary sources of information in Somalia.

3. SPSS designed the Mission’s Performance Management Plan (PMP) for the 2014–2018 Somalia Strategy. SPSS also designed the Mission’s PMP for the revised Strategy 2016–2019. This included IBTCI-led workshops with Transition Objective Teams that resulted in initial drafts of the PIRS for select performance indicators. These workshops also covered the evaluation questions that the Transition Objective teams were interested in pursuing. Subsequently, SPSS reviewed the Mission PMP annually. In reviewing the Mission’s PMPs, SPSS aligned its findings with the Automated Directives System (ADS) to integrate new approaches to monitoring, evaluation, and learning introduced by USAID. In Q3 FY 2015, SPSS conducted a review of USAID’s Policy, Planning, and Learning’s Office of Learning, Evaluation, and Research (PPL/LER) Toolkit. In FY 2017 SPSS led sessions on Collaborating, Learning, and Adapting (CLA) for each of the three Transition Objective Teams.

6

4. SPSS reviewed the AMELPs of USAID IPs, starting with the Growth, Enterprise, Employment, and Livelihood (GEEL) and Alternative Basic Education (ABE) AMELPs in Q3 of FY 2016. In FY 2017, SPSS developed a detailed AMELP template for the Mission. The template was revised in FY 2018 and has served as the basis for subsequent AMELP reviews.

5. SPSS provided performance data management via the USAID Clearinghouse, the IPTT, the Performance Plan and Report (PPR) annual data analysis, and the Development Information Solution (DIS) rollout. This included managing information via the Clearinghouse and extending it via our subcontractors Development Gateway (DG) and Synergy, developers of the Clearinghouse.

6. The Indicator Performance Tracking Table (IPTT) tool was developed by SPSS to monitor the performance of activities through MV&R teams collecting indicator data from IPs during sites visits. The goal of the IPTT was to validate, collate, analyse, and present results of IP performance indicator data. It identified variances between indicator targets and actual observed achievements.

Challenges

Few challenges were encountered in the performance and data management support services processes. Because SPSS was involved from the beginning of activities in the processes of designing projects’ RFs and TOCs, selection of indicators, revision of AMELPs, and other performance management activities, the support was successful.

Regarding data management processes, one significant challenge was the use by IPs of different approaches to reporting their performance data. While the Mission, through SPSS, designed a template for AMELPs, there was no harmonized format designed for IP performance reporting documents. Based on their internal practices, IPs submitted various types of performance reports to USAID that SPSS used to compile IPTTs and PPRs. As a result, data management of IP activities had information gaps in many of the IPs’ reports. To mitigate this, SPSS recommended the adoption of a harmonized template for performance reporting, which was not implemented. However, SPSS adopted a standardized “Queries on performance data,” which was accepted by USAID’s Program Office and used to minimize data errors, missing data, non-relevant narratives for deviations, and delays in reporting IPTTs and PPRs.

In working with IPs, SPSS noted a few challenges:

Challenges in selecting relevant indicators: In some cases, SPSS noted a number of indicators dropped by IPs over the implementation of their activities. Discontinued indicators were sometimes due to program shifts and changes in the RF. In some cases, they were due to poor approaches to indicator selection. SPSS mitigated this through support to Agreement Officer’s Representatives/ Contracting Officer’s Representatives (AORs/CORs) and IPs during the design of the projects, and during reviews and finalization of the RF, TOCs, and indicator selection.

7

Challenges in setting good targets: SPSS noted challenges in IPs setting realistic targets in their performance plans. In most cases, this was due to IPs not taking carefully into account the local context of their projects. This led to extensive use of irrelevant deviation narratives in the performance reports. To mitigate this, SPSS recommended better approaches to assess and monitor local context and use of context indicators that capture the context under which the activities are implemented in Somalia. SPSS also recommended that the risks and hypotheses in the RFs be clearly and directly linked to the level of results they influence and then be monitored carefully for better planning and program adaptation.

Challenges among IPs reporting relevant narratives in cases of under- or overachievements: For the Mission to better monitor activity performance, IPs must provide relevant information when they under- or overachieve against their targets. SPSS often found hidden or underreported contextual factors in IP responses to its queries. To mitigate against IPs underreporting of contextual factors that hampered achievement of targets, SPSS recommended tracking of contextual factors in IP RFs.

C. DATA QUALITY ASSESSMENTS (DQA)

Key Achievements

Data Quality Assessment (DQA) support services is one of the IBTCI-SPSS’s flagship activities.

In total, SPSS prepared and submitted, each on time, 39 deliverables for the six DQAs implemented over the LOP. The deliverables were organized by USAID Somalia in the following seven categories: (1) Training/Sensitization materials, tools, and protocols to be used during field work; (2) Training/Sensitization Workshop Report; (3) Indicator Assessment Summary Matrix (input for PPR); (4) Data Validation Workshop Materials, (5) Data Validation Workshop Report, (6) Draft DQA Report; and (7) Final DQA Report.

Because security challenges and travel restrictions prevented USAID Somalia staff from directly conducting annual DQAs, SPSS was assigned to lead the DQA work for USAID in coordination with the Program Office (PO) and A/CORs to assess selected indicators managed and reported by the IPs working in Somalia. Table 1 describes the DQAs by year.

Table 1: DQAs indicators number, activities, and areas visited

DQA Year TO Number

Number of Indicators Assessed

Activities (IPs) Areas Visited

2014 AID-623-TO-14-00014 6 TIS (DAI, IOM), PEG (DAI) Nairobi, Mogadishu, Hargeisa

8

DQA Year TO Number

Number of Indicators Assessed


2015 AID-623-TO-15-00008 18 TIS (DAI, IOM),

SYLI (Mercy Corps)

Nairobi, Mogadishu, Hargeisa, Garowe

2016 AID-623-TO-16-00005 100

TIS+ (AECOM),

SYLI (Mercy Corps), SSG (Chemonics), ABE (UNICEF), GEEL (IRG), TIS (DAI)


2017 TO# 72062318F00002 27

TIS+ (AECOM),

BUILD (CREATIVE), SSG (Chemonics), ABE (UNICEF), GEEL (IRG)

Nairobi, Mogadishu, Hargeisa

2018 TO# 72062318F00005 16

TIS+ (AECOM),

BUILD (CREATIVE), SIFPO 2 (PSI), ABE (UNICEF), GEEL (RTI)


2020 TO # 72062320F00004 (DQA included in MV&R VII TO)

46

TIS+ (AECOM),

BUILD (CREATIVE), EAJ (PACT), Damal (CHEMONICS), ABE (UNICEF), GEEL (RTI)

Fully implemented remotely due to COVID-19 pandemic

Total 213 11 4

In total, SPSS assessed 213 indicators from 2014 to 2020 across 11 USAID Somalia activities. These DQAs comprised six separate Tasks Orders, out of which five were specific DQA TOs that mobilized short term technical assistance (STTA). The last DQA in 2020 was embedded into the MV&R TO and implemented internally with SPSS staff led by the Deputy Chief of Party (DCOP)/ Technical. SPSS assessed the 213 indicators monitored by USAID programs in varying sectors, which included the following:

1. Transition Initiatives for Stabilization (TIS) – Development Alternatives Inc. (DAI) International Organization for Migration (IOM) in DQA 2014, 2015, and 2016

2. Partnership for Economic Growth (PEG) – Development Alternatives Inc. (DAI) in DQA 2014

9

3. Somali Youth Learners Initiative (SYLI) – Mercy Corps in DQA 2015 and 2016

4. Strengthening Somali Governance (SSG) – Chemonics in DQA 2015, 2016, and 2017

5. Transition Initiatives for Stabilization-Plus (TIS+) – AECOM, DT-Global in DQA 2016, 2017, 2018, and 2020

6. Alternative Basic Education for Pastoralists (ABE) – UNICEF in DQA 2016, 2017, 2018, and 2020

7. Growth, Enterprise, Employment, and Livelihoods (GEEL) – IRG, RTI in DQA 2016, 2017, 2018, and 2020

8. Support for International Family Planning and Health Organizations 2: Sustainable Networks (SIFPO2) – Population Service International (PSI) in DQA 2018

9. Bringing Unity, Integrity, and Legitimacy to Democracy (BUILD) – Creative Associate in DQA 2017, 2018, and 2020

10. Expanding Access to Justice Program (EAJ) – PACT in DQA 2020

11. Strengthening Public Accountability in Parliament (Damal, meaning “Gathering Place” in Somali) – Chemonics in DQA 2020

Challenges

In the 2016 DQA the Mission requested SPSS to assess 100 indicators selected from TIS+ (AECOM), SYLI (Mercy Corps), SSG (Chemonics), ABE (UNICEF), GEEL (IRG), and TIS (DAI). The large number of indicators, and the poor M&E approaches of IPs at that time, led to significant challenges in the implementation of the DQA. SPSS changed the DQA Team Leader at USAID’s request and reinforced the team with a strong M&E Specialist. To mitigate occurrence of such situations, in the following DQAs USAID selected a relevant number of indicators to make the process more useful with a focus on PPR indicators and non-PPR indicators relevant to programming and decision-making or learning processes. The 2017 DQA focused on 27 indicators while the 2018 and 2020 DQAs focused on 16 and 46 indicators, respectively.

Lack of good record keeping and poor organization of data sources represents an ongoing challenge when working with data of IPs with inadequate protocols and poor habits of generating or organizing their data. Hence, early DQAs faced various difficulties in the field due to challenges in accessing data sources, mainly in 2014, 2015, and 2016. Progressively, IPs adopted better approaches to meta-data and data sources management, which improved greatly the verifications by the DQA teams in 2018 and 2020. This shift was evident in the easy accessibility to data sources and documentation during the 2020 DQA when Google Folders were, at the request of USAID, used by every IP to store data sources. SPSS recommended using Google Folders to store data sources, facilitating data verification.

10

Lack of understanding of USAID data management and M&E standards by local partners in Somalia affected the quality of information IPs reported to USAID. In some instances, IPs contested the DQA results because they were not fully aware of the ways their local partners at the primary level collect and report data. In some instances, when data issues were identified at the primary level, the IPs at the central level realized those difficulties of local partners with data management and reporting only during the DQA process. SPSS proposed various approaches for IPs at central levels to build the capacity of local implementers at the primary level, and to reinforce and clarify their roles and responsibilities in the AMELP sections. SPSS also organized DQA sensitization at the primary level to make sure local partners were aware of USAID standards for data quality.

D. EVALUATIONS AND ASSESSMENTS

Key Achievements

Over the SPSS LOP, USAID Somalia tasked IBTCI to design and manage performance evaluations of individual activities and/or projects implemented by USAID/ Somalia IPs. They included baseline, midline, and end-line evaluations. IBTCI was asked to evaluate impact against IRs and/or overall stabilization/development objectives. These were accomplished through attitudinal surveys, focus group discussions (FGDs), key informant interviews (KIIs), and other participatory methods applied to beneficiaries, communities, and local organizations.

The following table provides a summary of those evaluations and assessments conducted by SPSS.

Table 2: List of evaluations and assessments done over the SPSS LOP

# Task Order Number Dates of Implementation Description

1 AID-623-TO-15-00001 05/04/2015 to 11/30/2015 Somali Youth Learners Initiative (SYLI) Baseline Evaluation

2 AID-623-TO-15-00006 07/16/2015 to 07/31/2016 Partnership for Economic Growth (PEG) Evaluation

3 AID-623-TO-15-00014 09/30/2015 to 07/30/2016 Transition Initiatives for Stabilization (TIS) Evaluation

4 AID-623-TO-15-00015 09/30/2015 to 01/31/2017 Baseline System Development for Strengthening Somali Governance (SSG)

5 AID-623-TO-16-00015 05/30/2016 to 07/25/2019 TIS+ Baseline and End-Line Assessment

11


6 TO#72062318F00003 02/01/2018 to 03/30/2018. Gender Analysis 2017

7 TO#72062320F00007 08/10/2020 to 06/30/2021 Citizen Consultation and Gender Assessment 2020

The SPSS Final Performance Evaluation of the TIS Program – This evaluation focused on a five-year initiative implemented from 2011 to 2016 in Somaliland, Puntland, and South Central Somalia by two USAID IPs [Development Alternatives Incorporated (DAI) and the International Organization for Migration (IOM)]. SPSS evaluators reviewed documents, conducted 87 FGDs, 221 KIIs, and a large survey, with a sample size of 1,719, in each implementation region of Somalia. The evaluation found that TIS had contributed to increasing confidence in the ability of local government to improve lives; the level of professionalism of local government; community satisfaction with basic services; collaboration between civil society, private sector, and local government; and stability in TIS communities. Through the process of gender-inclusive, community-based consultation, TIS had contributed to increasing the role of women in community-level decision-making.

The SPSS evaluation of the follow-on activity, TIS Plus, implemented by AECOM, found increased confidence in government in implementation areas related to improved service delivery, increased satisfaction of service delivery by the government, improvements in citizen participation in shared activities, and lessened conflict over resources.

The SPSS Baseline Evaluation of SYLI established baseline measures for a number of indicators from which to assess changes in outcomes achieved by the activity at end-line. The data collection process in 15 secondary school communities took place in Somalia from June 4 to 20, 2015, and was rolled out in six regions in 11 districts. The evaluation concluded that the most significant factor positively influencing learning outcomes was the quality of teachers. The most significant community-level factor positively influencing learning outcomes was parental support and encouragement for their secondary school-age offspring.

IBTCI piloted a novel Citizen Consultation sub-activity via a subcontract arrangement with the Africa Voices Foundation from September 2020 to June 30, 2021. It consisted of a six-part radio series with integrated social media (Facebook) amplification in Banadir and the South West State. It addressed social and civic engagement of youth in the decision-making in their communities, among other topics. The measurable outputs can inform USAID about improving the design and implementation of its strategy and activities.

Challenges

SYLI Task Order

12

The timing of the baseline should have straddled the end of the academic year to collect data from schools in a similar manner to baseline and others while school was in session.

The qualitative data collection process in Somalia could have been strengthened by ensuring that appropriate and sufficient training took place for the FGD & KII data collection process, including a practicum element to provide facilitators and enumerators with the training and opportunity for trainers to observe.

PEG Evaluation Task Order

Given the logistical, operational, and security challenges experienced for the evaluation of the PEG activity, it became evident that it is better to plan for a longer time for data collection than the ten days originally allocated for PEG evidence generation.

In planning for field data collection using the mobile phones, it is important to provide for a reasonable time to test the mobile data collection tools before actual deployment in the field. A number of issues that arose during the PEG evaluation could have been resolved if there had been more time provided during the pretesting of the mobile data collection tools.

SPSS should have vetted all the field enumerators hired by subcontractors to collect data. This vetting would have ensured that only quality enumerators were selected for the tasks. Where possible, enumerators should be deployed only to their home bases/regions where they have familiarity with the socioeconomic, political, cultural, and security contexts.

It is important for the data collection subcontractors engaged by SPSS to conduct thorough feasibility checks to ensure that all logistical, operational, and security arrangements are fully in place before actual data collections begins.

Translation and transcription of audios should be done by an independent entity to ensure quality. Translations and transcriptions done by an independent entity result in better quality and reduce enumerator bias. Once the audio recordings are obtained from the data collection subcontractor, the translations should be done by an independent professional firm to avoid bias.

Mitigation measures for data quality control and assurance should take place from the start of activities, when data starts streaming into the mobile data collection back-end. SPSS should not wait until the complete dataset is obtained before fully enforcing the mitigation measures.

After PEG, SPSS reverted to a more rigorous screening and assessment of its Team Leaders and other key personnel. When necessary, SPSS worked with known specialists to avoid poor performance.

Following USAID rejection of the initial draft PEG quantitative data analysis, SPSS technical personnel led the process of survey design, data collection, and data analysis and

13

were closely involved for subsequent evaluation activities. Corrective actions included the following:

• Designing and implementing more rigorous data quality control measures, such as inclusion of data validation syntax in the questionnaires, before data are collected in the field

• Conducting pilot/test surveys to test the data collection tools, as well as the robustness of the entire data collection and storage system and process, before actual rollout of the entire exercise

• Conducting mini-surveys prior to launching the data collection process to enable identification of issues with the datasets early enough to address them before completion of field data collection

During the presentation of preliminary findings and conclusions about the PEG evaluation to USAID staff and to other stakeholders, the COR made a recommendation that the key findings of the evaluation should be packaged for engaging the private sector. Although the structure of the task order did not allow for this, SPSS felt that future task orders could be structured to allow follow-on actions even after submission of the final report to the Development Experiences Clearinghouse (DEC).

SSG Evaluation Task Order

SPSS replaced both the first Team Leader and the first Evaluation Specialist because of performance issues and undertook a more rigorous assessment of Team Leaders and other key personnel. When possible, SPSS strove to work with known quantities. SPSS senior management provided coverage in instances of unexpected and sudden departures of key personnel.

TIS Evaluation Task Order

Like SSG, the TIS evaluation was faced with the need to replace key personnel for personal or performance reasons. SPSS worked more closely with the COR to identify team composition competencies early as shortened recruitment times work against recruiting the most competent consultants, who are often committed well in advance.

Based on lessons learned from previous evaluations, SPSS decided not to outsource qualitative data collection to data collection firms but rather managed it internally. This entailed working with existing networks and local data collection partners in Somalia to help with recruitment of high-quality FGD and KII facilitators. Translations and transcriptions of audio from the field were then managed directly by SPSS in collaboration with professional firms, and with a team of SPSS staff back-checking them for quality. Engagement of Somali speakers better ensured quality control checks immediately after data had been collected. Through listening to audio loaded onto a safe and secure platform against approved protocol and survey instruments, the evaluation team could provide real-time feedback to the qualitative facilitators on performance within 24 hours of the qualitative data being collected.

14

Any audio that did not show adherence to agreed protocols or survey guides was rejected and documented as a performance issue for the respective qualitative facilitator.

The successful implementation of the TIS evaluation received positive feedback from USAID. As a result, SPSS decided to replicate the successful elements of this task order in subsequent task orders.

TIS+ Assessment

This assessment provided the following lessons:

• Qualitative data collection in Somalia requires carefully composed teams and staff to gain ready access to relevant stakeholders, as well as ensure a local understanding of clan dynamics.

• Reaching the perspective of women and marginalized groups is difficult. Female field operatives are essential, as are local staff knowledgeable about the clan dynamics and impacts on cluster (geographic area) conditions.

• Ensuring that data analysis is well articulated within cluster and across cluster groups to show variations and distinctions will contribute to the design and implementation of future stabilization programs in Somalia.

• Remote data collection teams require contingency planning for data protection and extraction in areas influenced or controlled by Al-Shabaab.

• Tool consistency is essential but adaptations to the tools will be needed at end-line to capture the effects of community contracting, bridging plan development, consensus process, etc.

• Balancing the validity of data and outcome analysis with a reasonable causal analysis (e.g., applying contribution analysis) and balancing this with quantitative data evidence is needed for a study of this nature.

It is critical that USAID adhere to the TO deliverables schedule to avoid placing too heavy a demand on evaluators with ad hoc requests not in the original TO. The TO team should know in advance all anticipated additional demands on the deliverables schedule so that it can plan accordingly. This will improve quality of the products that are submitted to USAID.

The dissemination of findings from the TIS+ assessment, first to USAID Somalia and then to Nairobi stakeholders working on stabilization projects in Somalia, helped generate interest in the TIS+ evaluation. Interest was evident by the additional request that USAID Somalia made to IBTCI to disseminate the TIS+ evaluation findings to the stakeholders in Mogadishu on December 2, 2019. This was four months after the end of the TIS+ evaluation task order.

15

Citizen Consultation and Gender Assessment

While USAID strongly recommended that IBTCI work with African Voices Foundation (AVF) on this sub-activity, more due diligence about AVF would have been helpful because this was the first time IBTCI had worked with them. This would have avoided the problems of contractual compliance with geographic “code 937” encountered at the beginning of the activity (AVF was a British organization and required a waiver for subcontracting). Also, there should have been more time allocated at the planning phase of this TO to avoid the rush that was faced during the implementation of series 1 radio shows.

E. MONITORING, VERIFICATION, AND REPORTING

Key Achievements

ADS 201 stipulates that USAID oversee responsibility for the tracking of Implementing Partners’ (IP) performance and ensuring that they are accomplishing the tasks they set out to achieve. Delays in completing tasks or producing outputs provide an early warning that results may not be achieved as planned. USAID/Somalia is also responsible for monitoring partner compliance with Code of Federal Regulations, environmental determinations, and Branding and Marking Plans. Given that travel to Somalia for USG staff is restricted, USAID/Somalia utilized the services of IBTCI to conduct monitoring and verification visits to all the USG-supported sites, i.e., to be USAID’s eyes and ears on the ground.

SPSS received seven MV&R Task Orders, and conducted a total of 2,696 verifications across all regions of Somalia for the following USAID Somalia activities:

1. Somali Youth Learners Initiative (SYLI) – Mercy Corps 2. Partnership for Economic Growth – DAI 3. Transition Initiatives for Stabilization (TIS) – International Organization for

Migration (IOM) 4. Transition Initiatives for Stabilization (TIS) –DAI) 5. Somalia Towards Reaching Resilience Project (STORRE) – CARE 6. Program to Enhance Resilience in Somalia (PROGRESS) Catholic Relief Services

(CRS) 7. Resilience and Economic Activity in Luuq (REAL) – World Vision International

(WVI) 8. Strengthening Somali Governance, led by Chemonics (SSG-Chemonics) 9. Alternative Basic Education for Pastoralists (ABE) – UNICEF 10. Transition Initiatives for Stabilization Plus (TIS+) – DT Global 11. Growth, Enterprise, Employment, and Livelihood (GEEL) – International Resource

Group (IRG) 12. Office of Transition Initiatives (OTI) – International Organization for Migration

(IOM) 13. Bringing Unity, Integrity, and Legitimacy to Democracy (BUILD) – Creative

Associates 14. Expanding Access to Justice (EAJ) – Freedom House/Pact

16

15. Strengthening Public Accountability in Parliament (Damal) – Chemonics 16. Bar ama Baro – “Teach or Learn” – Creative Associates International

Figure 1 is a map of SPSS site visits by state.

17

Figure 1: Total SPSS Site Visits by States

Challenges and Lessons Learned

SPSS experienced challenges throughout all the years of implementing MV&R task orders. Addressing these challenges was an iterative process that often resulted in adaptive management based on lessons learned. Over the LOP, SPSS learned, for instance, that normal confirmation of each IP’s implementation timelines for each selected site is imperative. Especially in the earlier years, a significantly high number of sites were classified as unverifiable for several reasons including the selected sub-activities being out of the planned verification timelines. Initially, pre-deployment discussions between SPSS and IPs that sought to reduce occurrences of non-verifiable sub-activities linked to implementation scheduling or delays were not very successful. As MV&R continued, SPSS helped institute a process for formal, timely response from the IPs regarding the verifiability of sites where visits were intended to avoid incurring costs on verifications that end up not being conducted.

Authentication of the beneficiary lists: Earlier, SPSS found that some verifications could not be conducted because the list of names shared for sampling represented planned

18

beneficiaries as opposed to actual beneficiaries. SPSS made clear that IPs needed to provide the list of beneficiaries confirmed as having received training or participated in events.

Verification support documentation: SPSS verification is evidence-based, and supporting documentation constitutes an integral part of the verification process. SPSS adopted three different models of support document submission by IPs based on lessons learned.

• Under the first model, IPs share supporting documents with their field offices. SPSS field monitors (FMs) are provided with the same documents for review before the verification, take photos of them, and attach them with the draft verification reports.

• Under the second model, IPs submit all supporting documents to the SPSS management office once the approved list of sites has been reviewed by the IP and SPSS and the corresponding grant numbers are confirmed. This model, however, experienced challenges for several reasons. In some cases, SPSS FMs were not briefed on time by SPSS management on the availability of the documents, leading to multiple requests to the IP. In other cases, the documents submitted by the IP did not match the documentation required for verifications. This happened particularly for verifications of infrastructure and construction. There was often significant time required by SPSS management to align the grant numbers in the support documents with the SPSS verification numbers. This affected the time allocated to final report writing.

• Under the third model, the IP was responsible for attaching the relevant supporting documentation to each verification report once the draft was shared for review. This model worked well. SPSS prefers that FMs have access to the lists before data collection starts because verification of the names of selected participants is an important element of the overall verification. Regarding the certificates of substantial completion, SPSS observed that in some cases, engineers highlight aspects of the construction that need to be remedied. It is important for the FMs to have prior information on such issues to incorporate them during the verifications.

Adaptation of the environmental compliance tool: SPSS reviewed the environmental compliance tool to address inadequacies noted during the verification of compliance problems related to the local use of pesticides. The final tool was shared and approved for use by USAID. However, this tool was ultimately not used because it was complex and sensitive, requiring that local partners be formally inducted before applying it. Therefore, SPSS postponed use of this tool.

Adaptation of the outcome/benefits mapping tool: For the outcome/benefits mapping tools protocol, SPSS engaged in a consultative process to adapt the tool to address the specific needs of the planned verifications. A draft tool was prepared and approved by USAID for adoption on a pilot basis. Findings from 13 verifications conducted using the outcomes tool highlighted significant performance-related issues, which underscored the importance of the case-by-case approach to tool adaptation. SPSS also found that it needed to strengthen the evidence base of the outcome verifications, either by conducting more

19

qualitative interviews per verification or opting for closed-ended questions backed with documentation of performance improvements.

Unpredictable security and political context/situation in Somalia: The unpredictable security situation in parts of South-Central Somalia and Mogadishu, coupled with transport challenges, affected the pace of verification. Given this, SPSS undertakes comprehensive reviews of the security situation before deployment of monitors. The security of FMs, respondents, and IP personnel targeted for interviews is paramount, and verifications were conducted only when security permitted.

Communication with the IPs: IPs sometimes complained about, what they saw as, the excess amount of communication required with SPSS field teams as well as with the SPSS management team in Nairobi. SPSS recognized the concerns expressed formally and informally by IPs about the need to minimize questions and multiple requests. In addition to measures already undertaken such as restricting communication between IPs and SPSS at the field level to the field supervisors, SPSS considered allocating more time to the pre-deployment consultations at the Nairobi level.

Flagging Issues:

• Tool A 3-1&2 (Post-Construction tools focusing on usability and sustainability of constructed/rehabilitated facilities): Areas of concern related to the roles of community members in participating and contributing towards the sustainability and maintenance of facilities especially government facilities. Questions that triggered a yellow flag for these verifications revolved around the availability of sustainability plans and resources, availability of maintenance plans and resources, and training about maintenance. Feedback from IPs indicated that these elements were not factored into the design of government-owned facilities such as offices, hospitals, and sports stadia. This experience calls for a review of both the tool and the flagging algorithm.

• Tool B1, B2, and F1, F3 (Events): Absence of USAID branding and marking would sometimes trigger a yellow flag. Some partners had an exemption on this requirement, but the system was not designed to isolate such elements. Therefore, verifications were flagged yellow even when the indicator questions did not apply to certain activities. This calls for a redesign of scoring to assign a value for “Not Applicable” that does not disadvantage the IP by leading to low overall scores.

• Tool F3 (Trainee interview tool): Acquisition and subsequent application of skills was a major issue for verification. When respondents indicated that they did not acquire skills, even though they may have gained some knowledge from the training, the interview was stopped. This generated an automatic yellow flag because most of the subsequent indicator questions were pegged to skill acquisition. Feedback from IPs, and further discussions between USAID and the SPSS team, focused on a set of questions on this tool that need to be reviewed or eliminated. Both the tool and the flagging algorithm for this tool were reviewed.

20

Challenges encountered and addressed during each MV&R year: The sections below show how SPSS addressed challenges during each MV&R year, ensuring improvement in subsequent years.

MV&R Year 2: Pre-deployment planning and consultations between the SPSS team and USAID implementing partners improved communication and sharing of information. Continuous updates and regular communication between field-based IP staff and SPSS Field Supervisors on selected sites contributed to better and timelier responses on the sites that would be potentially unverifiable.

Tool J-1: Community-Based Saving and Lending Schemes: Respondents had some difficulty in understanding the questions related to the existence of an agenda for the groups’ meetings and documented minutes/notes. The group methodology relies on oral memory to ensure the inclusion of less literate members. Some questions revealed that agendas in these groups are usually communicated orally. Respondents usually answered “yes” to questions related to written notes of the meetings, while in essence there was no documentation.

In rounds 3 and 4, SPSS responded to specific requests from USAID to conduct site visits outside of the data collection “window.” These site visits took place in Puntland and Emerging Federal States (EFS) and were conducted by SPSS Field Supervisors. This flexible approach to ensure SPSS’s responsiveness to USAID’s informational requirements was replicated in subsequent rounds

Mobile phones experienced occasional “freezing” screens, at which time the data collection tools failed to load (only a white screen was visible). This occurred despite rigorous testing before teams deployed. Consequently, SPSS FMs were briefed to carry additional phones and to always carry pen and paper instruments as a backup.

MV&R Year 3: SPSS noted that the pre-deployment meeting with the IPs was crucial and needed to be considered separately from the data collection period. The IPs needed to give adequate time for pre-deployment meetings, to complete records and provide supporting evidence, including details of locations and districts where they had implemented sub-activities.

SPSS found a need to contact MV&R respondents, including local beneficiaries and IP managers, as early as possible during the data collection window to determine whether there would be a need for replacements.

MV&R Year 4: Overcoming challenges in pre-deployment activities: SPSS found that the addition of open-ended questions resulted in more in-depth answers from the verifications, especially the benefits mapping tools. It also noted that early updates and thorough review of the data collection tools on Askia (the mobile data collection platform) ensured adequate time for testing, review, and piloting before the round. This reduced scripting issues and minimized errors entered into the Askia tool.

The development of new tools tailored to unique and IP-specific scenarios in the field helped gather appropriate information and extend the scope of activities verified. Examples

21

included the Child to Child Club (CtC) formation tool and the Small-Scale Infrastructure Tool.

SPSS instituted IP pre-deployment meetings as a quality control measure to ensure that each protocol was followed, and the right sites were visited. The inclusion of SPSS Nairobi staff aided in quick decision-making and clarification of sites as intended. Prior confirmation and sharing of site locations for intended visits enhanced planning and budgeting for data collection. It also minimized instances of sites being unverifiable due to access/security problems. SPSS also found that the availability of comprehensive information sheets and supporting documentation about sites selected better prepared FMs on what to expect at the site. SPSS’ pre-deployment experience also found that IPs should share supporting documents before the data collection starts. Also, verifications should always consider time lapses between implementation and verification so that recall challenges are avoided. As a result it was agreed that sub-activities would not be verified more than six months after implementation.

Overcoming challenges in deployment activities: SPSS determined that IPs needed to always introduce the FMs to the right subcontractors, awardees, or key points of contact, to avoid delays in data collection. In order to better access hard-to-reach respondents in government, SPSS team sought to better leverage the relationships with established networks of people with local and state governments.

SPSS found that site visit planning (including logistical plans) by the local partners with the SPSS management team is necessary to guarantee smooth data collection within the required time frame. Travel challenges and logistics could often be mitigated through field trips included as part of the induction workshops. Debrief sessions with local partners and Field Supervisors (FSs) helped clarify their terms of reference and expectations for common understanding and enabled SPSS to gather feedback and plan for actions and resolutions to be taken in the following rounds. SPSS took steps to ensure gender balance within field teams, sometimes requesting that female field monitors interview female respondents.

SPSS found that the best way to ensure quality control was to carry out random spot checks and re-interviews. As a result, SPSS was better able to ensure that FMs adhered to the protocol for data collection. The presence of field supervisors during data collection in the field and spot checks fostered quick decision-making on field queries and enhanced FMs’ confidence in data collection.

Overcoming challenges in post-deployment activities: To address tracking problems during the post-deployment phase, the MV&R team developed an Excel verification and reporting tracker for all outputs and reports for each stage of verification and monitoring from the field to report generation.

MV&R Year 5: Overcoming challenges in pre-deployment activities: SPSS found the best way to address problems regarding third party monitors (TPM) understanding of monitoring technology was to engage partner monitoring firms early to review tools on the Askia platform. This ensure adequate time for their testing, review, learning, and piloting before the actual round of fieldwork started, reducing scripting problems.

22

Debrief sessions with Field Supervisors (FSs) helped clarify terms of reference and expectations among TPMs for common understanding and enabled SPSS to gather feedback and plan for actions and resolutions to be taken in following rounds. IP pre-deployment meetings continued to provide valuable opportunities to understand verification context and sub-activity details and to gather supporting documentation, which further enhanced information sheets in readiness for site verification.1

Overcoming challenges in deployment activities: SPSS concluded that starting with the easiest verifications rather than the more challenging sites helped avoid delays during data collection. In order to ensure all photos taken during verifications were appropriate and compliant, SPSS took steps to ensure consent forms were always used. To address issues of lost information that affected some interviews in earlier rounds, SPSS emphasized use of audio recorders during interviews2. Consent forms signed by the beneficiaries whose interviews were recorded were kept by SPSS TPM firms. However, there were no contractual requirements to document any lessons on storage/security of these forms.

MV&R 6 and 7: Many challenges with the MV&R process had been addressed by the time of the final two years of SPSS. Nonetheless, lessons continued to be learned throughout the later rounds of MV&R.

Pre-Deployment Lessons:

• The participation of partner firms in the IP pre-deployment meetings, together with the SPSS MV&R Nairobi team, is important in providing clear insights into all verification sites for the local monitoring firms.

• IPs need to notify respondents in advance, especially high-level respondents such as government officials, company heads, and external trainers/facilitators, to inform them about the interviews. This makes it easier for the monitoring firms to schedule appointments with them.

• The IP should share all the supporting documents before data collection to enable the partner organizations to plan. This reduces unnecessary delays.

1 IP pre-deployment meetings are meetings held between the IPs, SPSS staff from Nairobi, SPSS Field Supervisors, and the Third-Party Monitoring (TPM) firms. The meetings are led by SPSS Nairobi and are held immediately after the TPM Induction Workshops and just before data collection starts.

2 A typical face-to-face interview always started with consent for recording the interview being sought from the respondents. One Field Monitor would enter the responses on the Askia platform while the other would take notes.

23

• Provision of contact details for the key IP focal points and the event facilitators/ organizers is crucial in ensuring smooth communication between the local monitoring subcontractors and the target respondents.

• When securing interview schedules, teams should consider the available number of field monitors, logistics, timing, and interview venue for the respondents.

Deployment Lessons:

• Spot checks and re-interview exercises conducted by SPSS enable the team to get a clear picture of the quality of work implemented by the monitoring partners and quickly identify whether reliability shortfalls affected the quality of the data collected.

• It takes more time, patience, and good communication skills to get Mogadishu-based respondents and senior government officials for interviews than respondents in other regions. The possibility of interviewing a government official respondent at the scheduled appointment was low compared to other respondents, and one might be required to make several appointments before the interview happens.

• Prior notification of respondents and scheduling of interviews at the respondent’s convenient time was key. This enables respondents to plan and prepare for the interview. Some respondents requested to be called in the evening.

• It is important to give respondents whose phones were off or who did not pick up messages more time before requesting to replace them with backup respondents, because they may be busy or their phone batteries may be dead.

• Partner organizations should ensure that their FMs have sufficient phone airtime to complete the calls to avoid getting disconnected, especially when dealing with high-profile respondents who are difficult to access.

• FMs should review the daily log every day to update the status of the interviews and plan for the next day. This enables each team to track progress in contacting individual respondents and, when necessary, plan for replacements.

• In Puntland, one partner firm mentioned that the number used to make calls was an important element in the interview. Some respondents did not respond to unfamiliar phone numbers. Therefore, some TPMs provided FMs with unique and old numbers to use in conducting the interviews.

• Several monitoring partners found that physical/face-to-face interviews are more effective and fruitful than telephone interviews, because the interviewer can observe the facial expressions of the respondent and make eye contact.

24

• The FMs should be ready to conduct interviews at all times. Sometimes, when calling to schedule an interview, the respondent might mention that they are ready and available for the interview at that moment.

• Incorporating a COVID-19 preparedness plan into the MV&R activities was useful to ensure the FMs’ health and safety during field data collection. No cases of COVID infection were reported by our TPMs during MV&R 7.

• SPSS found that long-term relationships with TPM firms grew stronger through collaboration on MV&R TOs over the years. Personal relationships built over time between TPMs and SPSS personnel allowed higher levels of trust and ease of communication.

LESSONS LEARNED

Lessons to Ensure Qualified Teams

A regular SPSS Tool Induction Workshop for the local monitoring firms before each round of data collection enhanced the understanding of field monitors and team leaders about the MV&R tools and protocols and other research-related topics such as ethics that strengthened the quality of data collection.

• Our monitoring partner firms’ retention of trained field monitors (by paying monthly salaries) facilitated ongoing learning and improvement in data collection exercises.

• Incorporation of the TPM lessons learned presentations during the Induction Workshop facilitated cross-learning and among partners.

Efficient Planning Mechanisms

• Joint review and individual discussions between SPSS and partner staff on TPM work plans and logistical arrangements for data collection facilitated efficient planning and administration of data collection.

• The use of the TPM checklist manual for pre-deployment and logistic arrangements ensured adherence to deployment plans.

• Ongoing security assessments helped provide continuous updates to SPSS and partner firms on the overall security situation of the selected sites and potential implications of planned verification activities.

• Advance sharing of site lists with monitoring firms smoothed the field monitoring planning processes.

25

Efficient Implementation Mechanisms

• Clarity of roles and responsibilities, clear communication channels, and coordination among SPSS Nairobi, Field Supervisors, partner firms, and Field Monitors during MV&R pre-deployment, deployment, and post-deployment phases facilitated the complementarity and collegial execution of MV&R tasks.

• Rotation of partner firms across IPs helped the monitors better understand IP activities and engage with different SPSS tools.

• Dividing the MV&R year into manageable cycles/rounds enhanced implementation of MV&R tasks and smooth delivery of deliverables.

• Data quality control systems: Participation of SPSS in TPM-level Tool Induction Workshops as well as spot checks and re-interviews conducted by SPSS Nairobi and Field Supervisors, as a mechanism of supervision in the field, promoted fulfillment of SPSS data collection protocols and helped to immediately solve/correct field issues.

• During field data collection, the role of the Field Supervisor was indispensable to the overall success of each round of verifications.

• The use of mobile platforms reduced FMs errors and improved the accuracy and integrity of data collected.

• Availability of Somali-translated tools and protocols for ease of reference by the FMs before, during, and after data collection helped greatly.

• SPSS systematically handled issues of conflict of interest by not allocating any one monitoring partner to related partner IPs.

Intended and Unintended Benefits of SPSS Engagement

Former team leaders and field monitors have been promoted to higher positions due to SPSS capacity-building initiatives, resulting in strengthened knowledge and skills in M&V. Local field workers gained skills that could be used elsewhere. The interactions have also built social capital among local monitoring partners.

SPSS has given Somali communities the opportunity to voice their concerns and share feedback on how best their needs can be met. This was because FMs interacted with community members and relayed their thoughts to USAID through SPSS.

RECOMMENDATIONS FOR FUTURE WORK

Recommendation #1: PERFORMANCE MANAGEMENT

26

• Design a harmonized template for performance data reporting to be used by all IPs to facilitate IPTTs and PPR preparation and help IPs adopt better approaches to performance reporting that better align with USAID standards.

• Continue and reinforce the Program Office and local subcontractor participation in the process of projects’ RF and TOC review, as well as indicator selection in the inception phases prior to designing the AMELP where possible.

• Integrate context assessment and analysis in IP baseline studies to ensure better understanding of underlying contextual factors prior to starting implementation. The TPM contractor can then provide relevant approaches to harmonize the IPs’ assessment and analysis of local context during baseline studies.

• Reinforce and integrate context monitoring and reporting approaches in IP AMELPs. Train IPs in context monitoring and good practices in reporting deviations.

• Ensure that all IP progress and performance reports are submitted to USAID A/CORs and shared timely with the local subcontractor to reinforce timely completion and use of IPTTs and PPR.

Recommendation #2: DATA QUALITY ASSESSMENTS (DQA)

• Maintain the approach of A/CORs in selecting DQA indicators based on PO priority criteria: (1) PPR indicators that require DQA, and (2) usefulness for programming, decision-making, and/or demonstrated evidence for learning purpose.

• Maintain the approach of using Google Folders to organize and store data sources and M&E documentation of IPs during and after the COVID-19 pandemic. IPs should combine paper-based and online data storage systems for easy access during verifications.

• Build the capacity of IPs (mainly the local partners) at the primary level in USAID M&E protocols and data quality standards.

• Continue and reinforce the organization of remote sessions during DQAs to allow PO and A/CORs direct supervision and leadership over the process, and IPs better participation.

• Reinforce the use and update of the DQA recommendations follow-up tool by the IPs to ensure effective monitoring of the implementation of corrective actions following DQAs.

Recommendation #3: EVALUATIONS AND ASSESSMENTS

27

When TOs are issued for evaluations or assessments, it is best to allow ample time for the contractor to do pre-planning, hiring, and deployment of key personnel. This helps in ensuring quality of the evaluations/assessments.

As happened at the later stages of SPSS, USAID/Somalia should thoroughly vet Team Leaders and other key personnel for the TOs. If possible, recruitment should focus on consultants who are known quantities to both USAID and the local data collection contractors.3 Other than being subject matter experts in their own areas of specialization, USAID should ensure that the local subcontractor hires Team Leaders who have demonstrated experience in conducting evaluations of similar size and magnitude.

When subcontractors are needed, a competitive contracting process by the prime should be put in place and provisions made to ensure that the subcontractors also understand the standards and requirements of USAID.

3 IBTCI can provide contact information to USAID on request

28

ANNEX 1: SPSS TASK ORDERS 2014–2021

Task Order 1: Core Services

Task Order 2: MV&R Y1

Task Order 3: Communications

Task Order 4: Clearinghouse

Task Order 5: Strategy

Task Order 6: DQA

Task Order 7: SYLI Baseline

Task Order 8: PEG Evaluation


Task Order 10: DQA

Task Order 11: TIS End-line

Task Order 12: SSG

Task Order 13: Calendar

Task Order 14: DQA




Task Order 18: DQA

Task Order 19: Gender Assessment

Task Order 20: DQA




Task Order 24: Gender Assessment and Citizen Consultation

29

ANNEX 2: DETAILS OF CUMULATIVE SPSS ACHIEVEMENTS

Communications Task Order

On September 26, 2014, IBTCI received a Task Order (TO) to provide communication support services for USAID Somalia. This entailed collecting information from implementing Partners (IP) to produce fact sheets, success stories, snapshots, and other documentation at the activity, Intermediate Results (IR), and Development Objective (DO) levels. Through a subcontract arrangement with Management Services International (MSI), IBTCI provided a full-time Communications and Website Advisor embedded at the USAID Somalia Mission and supported the mission in carrying out its Development and Outreach Communications services. This TO was implemented for one year until September 30, 2015. During this period, the SPSS team produced activity fact sheets for all USAID/Somalia activities including the Joint Health and Nutrition Program that was implemented by UNICEF; the Partnership for Economic Growth (PEG) activity that was implemented by Development Alternatives Inc. (DAI); Somalia Program Support Services (SPSS) implemented by IBTCI; the Strengthening Somali Governance (SSG) activity implemented by Chemonics; the Somali Youth Learners Initiative (SYLI) implemented by Mercy Corps; and the Transition Initiatives for Stabilization (TIS) program implemented by both DAI and the International Organization for Migration (IOM). IBTCI also developed a USAID/Somalia Portfolio Overview fact sheet. These fact sheets were updated on a quarterly, rolling basis for a period of one year. IBTCI also produced success stories for USAID/ Somalia, wrote speeches and talking points for the Mission Director, accessed the regular messaging toolkits that were distributed by USAID/W’s Legislative and Public Affairs Bureau, worked closely with the regional USAID Development Outreach and Communications (DOC) to set up regular social media feeds (including Twitter and Facebook), collected photos from IPs and incorporated these photos into various channels (including USAID’s Somalia blog and Instagram), and produced videos and uploaded them to the video library created by the Somalia Monitoring and Evaluation Program (MEPS) project on YouTube.

Strategy Support Task Order

SPSS was asked to provide support for Somalia Stabilization Strategy through a stand-alone task order signed on September 22, 2014. This task order was implemented though July 28, 2015. USAID approved a Senior Strategy Specialist (SSS) who was contracted by IBTCI for this task order on October 15, 2014. She held initial workshops with USAID and then conducted Nairobi stakeholder consultations from December 8–19, 2014. She resigned in December over security concerns. Building on the work of the first consultant, a replacement consultant conducted Somalia stakeholder consultations in Garowe, Hargeisa, and Mogadishu, after which the SSS compiled the consultation summaries and drafted the USAID Somalia Results Framework (RF) diagram. On March 9–15, 2015, SPSS worked with USAID’s Bureau for Policy, Planning, and Learning (PPL) to conduct a final RF workshop and a Digital Video Conference (DVC) with USAID/ Washington. These were followed by three expert seminars with Somalia think tanks and subject matter experts held on March 6, 20, and 27, 2015. On April 8, a video teleconference was held with the USAID Deputy Assistant Administrator for Africa, Linda Etim, to present the draft RF. SPSS submitted an initial draft on April 20, 2015, a second draft of the strategy on April 25, 2015, and a third draft on July 14, 2015. USAID accepted and approved the revised third draft as the final deliverable of the task order on July 28, 2015.

30

USAID Clearinghouse Support

On September 30, 2014, USAID awarded IBTCI TO #AID-623-TO-14-00022 to maintain the Clearinghouse (CH) for Somalia that had been developed under the Monitoring and Evaluation Program Support (MEPS), the predecessor to SPSS. This was a secure web-based application accessible via a web browser (Internet Explorer 9.0 or higher) accessible to both United States Government agencies (including the Department of State, Department of Defense, and USAID Headquarters, as well as USAID implementers) and the Somali government. The CH enabled USAID/Somalia to monitor, verify, and evaluate the performance, results (outputs and outcomes), impact, and environmental compliance of USAID programs and activities. The CH provided performance monitoring and reporting on specific activities and programs. Data was entered primarily by USAID IPs and was accessible to IBTCI, USAID Contracting Officer’s Representatives (CORs), Agreement Officer’s Representatives (AORs), and technical teams.

Prior to October 2014, the application had been maintained and supported by its original developer, Synergy International Systems (a subcontractor to IBTCI). Subsequently during SPSS, IBTCI contracted Development Gateway (DG) to maintain and provide user support for the CH in cooperation with IBTCI through June 26, 2015. During this period, DG hosted and maintained the system, addressed technical defects, trained users, and provided on-the-ground support as needed. In this role, DG implemented bug fixes and provided continuous improvement of the software and other aspects of the CH.

SPSS achieved the following under the CH support:

• Training for USAID/Somalia mission staff • Ongoing support to users as issues were reported • GIS mapping software—developed database script to manually pull all activity data to

compare with reporting module tool • Pulled all historical USAID Somalia IP data from 2011 by indicator, IP, and period

o Identified indicators with formulas and explained them o Prepared handover notes and provided a handover presentation on June 12,

2015 • CH handover activities

o Walkthrough of submitted User Manuals, performance data, and any other CH documents on June 24, 2015 at USAID

o Deployment of CH in local environment and smoke testing on June 26, 2015, at USAID

o Trained USAID ICT Officer on activation and maintenance of User Manuals in local deployment

o Deactivated all CH active users

The CH Task Order officially ended on June 26, 2015.

31

Support for USAID Meetings and Events

IBTCI was tasked under its Core Services Task Order to provide logistical and administrative support and facilitation services when needed for USAID/Somalia technical team meetings with partners or other stakeholders for the purposes of planning, monitoring, evaluating, and/or learning. This included arranging and paying for the meeting location and sometimes assuming the role of a facilitator at the meetings, but this did not include arrangement for or the costs of travel, transportation, and/or lodging for meeting attendees. These were one-day meetings, for a maximum of 12 meetings per year at a location near the U.S. Embassy/ Nairobi complex with an average of 25 participants. Throughout the Life of Project (LOP) of SPSS, IBTCI hosted over 60 meetings and events. Annex 3 provides a listing of all meetings and events hosted by IBTCI over the SPSS LOP.

Performance Management and Data Management

Performance management support services started under Core Services TO, in Q3 FY 2014. The USAID/EA/Somalia Country Development Cooperation Strategy (CDCS) 2014–2018 was in the final steps of its development and SPSS was tasked to provide the support required to create project log-frames, results frameworks, and M&E Plans/AMELPs for USAID/EA/ Somalia; collect baseline data at the goal, DO, IR, and context levels; and train/advise all IPs to create aligned log-frames and M&E Plans for their activities.

Data management support services started with the CH TO in Q3 FY 2014. USAID/ Somalia tasked SPSS to maintain the Clearinghouse for USAID staff and implementing partner use. During the SPSS start-up period, the system focused on collection and management of performance data, including supporting documentation, but did not facilitate USAID management of its activities or information relating to them. SPSS implemented performance and data management support services throughout its implementation from 2014 to 2021.

Performance management

Performance management support services SPSS delivered to USAID/Somalia covered the following domains: (1) supporting Theory of Change (TOC) and RF review and strategy review; (2) collecting baseline data and context-level information at the goal, DO, and IR levels; (3) designing the Mission’s PMP and updating annually; (4) reviewing IP AMELPs; and (5) capacity-building of USAID and IP staff on performance management.

1. Supporting TOC and RF review and strategy review

In FY 2016, SPSS participated in various meeting and sessions on RF review under specific programs. Those included a series of sessions with Transition Initiatives for Stabilization Plus Activity (TIS+ AECOM) to discuss their RF and their draft Performance Indicator Reference Sheets (PIRS). The sessions also reviewed all the indicators along the hierarchy of objectives, and made amendments where necessary. SPSS also met with Growth, Enterprise, Employment, and Livelihood (GEEL) and Alternative Basic Education (ABE) in Q3 in different sessions to discuss their RF and draft PIRS, to review all the indicators along the hierarchy of objectives and make amendments where necessary, and to support GEEL and ABE to revise relevant targets as well as clarify data collection processes

32

for some indicators, e.g., number of new jobs created (GEEL), and number of school children enrolled, dropping out, retained, and re-enrolled after dropout (ABE).

SPSS continued TOC and RF review support throughout its implementation. In FY 2019, SPSS supported USAID/Somalia to finalize the Expanding Access to Justice (EAJ) TOC and RF in two sessions held at the EAJ Head Office in Nairobi. SPSS specifically helped EAJ to better define its results and related indicators, contextual factors that may influence the program mainly in the inclusivity strategy. SPSS supported in FY 2019 USAID and Chemonics staff to finalize the Damal RF. In FY 2020, SPSS Deputy Chief of Party/Technical participated in various sessions to support BUILD’s revision of its TO and RF, and to better link risks and hypotheses to the RF levels.

Over its implementation, SPSS provided technical supports to the Mission on strategy review of ongoing activities in Somalia. These supports covered synthetic analysis of MV&R report and performance reports including IPTTs to generate key findings and conclusions on performance and context that USAID used to organize strategy review sessions with IPs. Those synthetic analyses were critical in program adjustment because in some cases they also integrated summaries from evaluation findings. From FY 2017 to FY 2021, SPSS supported strategy review processes for TIS+, ABE, EAJ, and BUILD projects.

2. Collecting baseline data and context-level information at the goal, DO, and IR levels

SPSS provided support in baseline data collection by compiling information from IP reports and from various other secondary source of information in Somalia, mainly in FY 2015 and FY 2016. For FY 2017 upward, SPSS used only IP reports to generate baseline information with the exception of TIS+ goal-level and IR-level outcome indicators whose baselines were generated from the TIS+ evaluation SPSS carried over.

3. Designing the Mission’s PMP and updating annually

SPSS designed the Mission PMP for Strategy 2014–2018. SPSS also designed the Mission’s PMP for the revised Strategy 2016–2019. For example, in FY 2017, SPSS mobilized an STTA to develop and draft the Mission PMP. The Performance Management Plan (PMP) Specialist assisted in drafting the new PMP for USAID/Somalia. He conducted workshops with each of the Transition Objective Teams that resulted in a first draft of the PIRS for the selected performance indicators. These workshops also briefly covered the evaluation questions that the Transition Objective teams were interested in pursuing, either by way of independent or IP-led evaluations of various types.

Subsequently, SPSS reviewed the Mission PMP annually. The reviews started in Q3 each FY during the process of reviewing IP AMELPs and developing Work Plans for the next FY. SPSS collected relevant information from the IPs’ revised AMELPs to compile and analyze changes in TOCs and RFs at the project level, new indicators introduced, or indicators dropped or archived to update sections of the PMP. SPSS also collected information on new projects introduced during the FY to update the PMP with the relevant information. In reviewing the Mission’s PMPs, SPSS will align with the directives of the most up-to-date ADS 201 published that FY to integrate any new approaches to MEL introduced by USAID.

33

In Q3 FY 2015, SPSS conducted a review of the Policy, Planning, and Learning’s Office of Learning, Evaluation, and Research (PPL/LER) Toolkit following USAID’s request. In line with the PMP design and reviews process, in FY 2017 SPSS also facilitated sessions on Collaborating, Learning, and Adapting (CLA) for each of the three Transition Objective teams. SPSS facilitated the exchanges of the Transition Objective teams and staff from the Program Office (PO) through a defined CLA process to determine where they were situated that time with respect to this function and where they hoped they would be in next FY with structured action plan and milestones.

4. Reviewing IP AMELPs

The first full AMELPs SPSS reviewed were for GEEL and ABE in Q3 FY 2016.

SPSS AMELP reviews used USAID ADS 201 guidelines and the USAID detailed PIRS template. SPSS also used the Mission PMP to help IPs align their M&E framework to the Mission PMP and improve target setting and integrate better context monitoring indicators. In FY 2017, SPSS developed a detailed AMELP template for the Mission in relation with PO. The template was revised in FY 18 and approved by USAID. It served as the basis for subsequent reviews of the AMELPs.

The SPSS AMELP review process helped IPs adopt a more harmonized approach to developing their AMELPs and setting targets. It also helped IPs to use more commensurate techniques and tools for indicator selection, data collection and analysis, and design of a comprehensive PIRS section. Over its implementation, SPSS reviewed the following AMELPs as shown in the table below.

Table 3. List of AMELPs reviewed

FY AMELPs Reviewed Number FY 2016 GEEL, ABE 2 FY 2017 GEEL, ABE, BUILD, TIS+ 4 FY 2018 GEEL, ABE, BUILD, TIS+ 4 FY 2019 GEEL, ABE, BUILD, TIS+, EAJ 5 FY 2020 GEEL, ABE, BUILD, TIS+, EAJ, Damal 6 Total 21

SPSS used different formats to implement AMELP revision that depended on what A/CORs and the PO wanted the support to focus on, and capability of IPs in M&E. In most of the cases, SPSS embedded a capacity-building approach to ensure the IPs’ M&E staff and local stakeholders understand and apply USAID M&E rules and guidelines per the ADS 201 directives and other USAID materials and guidelines. In many cases, SPSS clarified the importance of the process of setting targets.

For example, when targets are set for PPR indicators, and USAID reporting system to Washington has already set those targets, there will be little chance for IPs to change those targets under those PPR indicators. Setting targets during AMELP revisions must carefully analyze what was set, especially under PPR indicators, and IPs must seek clearance from their A/CORs before any change is made in a PPR target.

34

In many cases, SPSS advised IPs to use indicator data collection approaches commensurate with the local context and the security challenges in Somalia. For example, using one-time annual survey indicator data might be risky in the Somalia context because of weak local capacity to prepare and carry out surveys, and security challenges. In most of the cases, those indicators where data was collected only once through annual surveys led to timeliness issues when they were assessed under a DQA.

In the AMELP review process, SPSS also prioritized the clarification by the IPs of roles and responsibilities of local stakeholders in the performance monitoring processes and to link that with the M&E capacity-building plan and technical resource mobilization. Building local capacity in performance monitoring of USAID programs is key in the Somalia context. That will foster appropriation, engagement, and commitment to achievement of results.

SPSS AMELP revision processes also focused on helping IPs to define and update good context indicators well aligned with their results frameworks. Understanding better and integrating the context analysis in monitoring of the assumptions and risks linked to the projects TOCs improved greatly the ways IPs reported on deviation narratives over the implementation of their activities. It also helped USAID better understand how achievements related to IP performance only and/or Somalia local context, which is beyond the IP’s control. For example, where BUILD underachieved on indicators that related to electoral process and elections were hampered by a hard-hitting local political and institutional context, good political and institutional context indicators were key in the performance reporting for deviation narratives. That helped BUILD to go beyond saying only “This was underachieved because elections did not take place” to better understand the underlying factors and integrate activities that will help unlock the electoral process and help USAID achieve its targets.

5. Capacity-building of USAID and IP staff on performance management In response to concerns about the quality of IP AMELPs in FY 2015 and FY 2016,

formal capacity-building of USAID and IP staff by SPSS was requested by USAID in FY 2017 and FY 2018. SPSS submitted capacity-building plans that were discussed with USAID and that proposed different formats for USAID staff and IP staff. However, these proposals were put on hold by A/CORs and SPSS did not organize formal M&E training for IPs until FY 2019, when following a DQA recommendation the AOR of ABE requested SPSS DCOP/Technical to organize an M&E training for UNICEF staff in Mogadishu.

M&E Training sessions were then organized for ten staff from UNICEF and its local partners HIDIG Relief and Development Organization (HIDIG), Bay Regional Education Committee (BREC), and Himilo Relief and Development Association (HIRDA).

SPSS performance management, DQAs, and data management activities adopted an informal participatory approach to capacity-building of USAID and IP staff. Experts were allowed to interact daily during various sessions of the activity’s M&E capacity with IPs, USAID staff, and local partners in Somalia. SPSS also provided capacity-building support to the Third-Party Monitoring (TPM) partners involved in the implementation of MV&R activities in Somalia.

Data management

35

Over its implementation, SPSS provided intensive and varied data management supports to USAID Somalia that essentially covered (1) performance data management via the USAID Clearinghouse, (2) Indicator Performance Tracking Table (IPTT), (3) PPR annual data analysis, and (4) Development Information Solution (DIS) rollout. SPSS is positioned as a unique M&E platform that provided USAID with all types of data management supports services.

1. Performance data management via the USAID Clearinghouse As described under the Core Services section above, SPSS supported the CH for

about one year. Following USAID’s request, SPSS also conducted an independent assessment on the practical utility of the CH to USAID and its implementing partners (IPs) as it was designed and managed. The assessment determined whether to retain/improve the CH or to move to another information management system. Under this assignment, SPSS worked with subcontractors Development Gateway (DG) and Synergy, developers of the Clearinghouse. The CH assessment by SPPS found that it fell short in helping the Mission manage the performance of its portfolio in Somalia. Constraints linked to the troubleshooting and complexity of the system, while it did not effectively respond to USAID needs for its growing portfolio in Somalia, and it was finally closed and handed over to USAID in June 2015.

2. Indicator Performance Tracking Table (IPTT) The SPSS IPTT tool was critical for data management and was used to support

USAID Somalia to monitor the performance of its activities. During FY 2014 and FY 2015, the IPTT [then called the ITT (Indicator Tracking Table)] was used by the MV&R team to collect indicator data from IPs during site visits that will serve to triangulate MV&R findings and conclusions. In late FY 2015, SPSS designed a comprehensive IPTT in relation with the Program Office that was used after the closure of the Clearinghouse.

The IPTT was then initially designed by SPSS and approved by USAID in September 2015. Its primary purpose was to validate, collate, analyze, and present results on IP performance indicator data. IPTT identified any variances between indicator targets and actual achievements. Data from IPs was validated through the IPTT. This involved identifying inconsistencies in indicator data and working with the IP to resolve the inconsistencies. For resolving inconsistencies, SPSS designed a tool called “Performance data queries form”

The IPTT used several sources for data including approved AMELPs, Progress Reports, DQA reports, and annual Mission PPR. For data collation and analysis, SPSS designed an Excel sheet and developed a summary performance analysis for each IP. A summary was presented for all indicators disaggregated by PPR indicators and non-PPR indicators. The results of the analysis were reflected through the flagging of indicators that are ± 10% either side of the IP’s annual target. In total SPSS completed and submitted 84 IPTTs for 11 activities over five FYs from 2016 to 2021. SPSS developed for each activity four IPTTs per year except for ABE, which reported to USAID on a half-year basis.

3. PPR annual data analysis

36

SPSS provided support to USAID in compiling and analyzing PPR indicators data annually generally following DQA data validation workshops from FY 2016 to FY 2021. USAID PO designed a matrix for each of the Mission’s PPR indicators which summarizes key information including last DQA data, annual target and actual, the next two years targets, deviation narratives if any. It also includes Mission Objective Linkages, baseline values and date, final year reporting narrative (if applicable), and current and future indicator performance analysis.

SPSS used approved IPTTs, IP Q4 progress reports, AMELPs and PIRS sections, and DQA reports to prepare the PPR matrices. The PPR matrices filled by SPSS served as input to the Mission annual PPR reports to Washington.

4. Development Information Solution (DIS) rollout

Starting in Q1 FY 2021, SPSS provided support to USAID Somalia Program Office in the process of the Development Information Solution (DIS) rollout. USAID Somalia selected 24 PPR indicators from its ongoing activities that have information on baseline and targets for the upcoming FYs through end of activity to migrate them progressively into the USAID DIS system.

SPSS participated in DIS training and working sessions with the DIS team and several Missions across Africa and prepared in relation with Program Office the needed information and data for the migration of a total of 24 PPR indicators of USAID Somalia into the DIS. SPSS also supported the PO to populate the data into the online standards formats of the DIS.

Data Quality Assessments (DQA)

Data Quality Assessment (DQA) support services is one of the IBTCI-SPSS flagship activities in this IDIQ. Because security challenges and travel restrictions prevent USAID Somalia staff from conducting regular DQAs, SPSS was assigned to lead the USAID DQAs work under the coordination of PO and A/CORs to assess selected indicators managed and reported by the IPs in Somalia.

Table 4: DQAs indicators number, activities, and areas visited

DQA Year TO Number

# of Indicators Assessed


2014 AID-623-TO-14-00014 6 TIS (DAI, IOM), PEG (DAI) Nairobi, Mogadishu, Hargeisa

2015 AID-623-TO-15-00008 18 TIS (DAI, IOM), SYLI (Mercy Corps)


2016 AID-623-TO-16-00005 100

TIS+ (AECOM), SYLI (Mercy Corps), SSG (Chemonics), ABE (UNICEF), GEEL (IRG), TIS (DAI)


2017 TO# 72062318F00002 27

TIS+ (AECOM), BUILD (CREATIVE), SSG (Chemonics), ABE (UNICEF), GEEL (IRG)

Nairobi, Mogadishu, Hargeisa

2018 TO# 72062318F00005 16 TIS+ (AECOM), Nairobi, Mogadishu, Hargeisa, Garowe

37

BUILD (CREATIVE), SIFPO 2 (PSI), ABE (UNICEF), GEEL (RTI)

2020 TO # 72062320F00004 (DQA included in MV&R VII TO)

46

TIS+ (AECOM), BUILD (CREATIVE), EAJ (PACT), Damal (CHEMONICS), ABE (UNICEF), GEEL (RTI)

Fully implemented remotely due to Covid-19 pandemic

Total 213 11 4

In total, SPSS assessed 213 indicators from 2014 to 2020 across 11 USAID/Somalia Activities through six separate Tasks Orders, out of which five are specific DQA TOs that mobilized STTAs. The last DQA in 2020 was imbedded into the MV&R TO and implemented internally with SPSS staff led by the DCOP/Technical. SPSS assessed the 213 indicators monitored by USAID programs in Somalia covering various sectors, including the following:

12. Transition Initiatives for Stabilization (TIS) – Development Alternatives Inc. (DAI) International Organization for Migration (IOM) in DQA 2014, 2015, and 2016

13. Partnership for Economic Growth (PEG) – Development Alternatives Inc. (DAI) in DQA 2014

14. Somali Youth Learners Initiative (SYLI) – Mercy Corps in DQA 2015 and 2016 15. Strengthening Somali Governance (SSG) – Chemonics in DQA 2015, 2016, and

2017 16. Transition Initiatives for Stabilization-Plus (TIS+) – AECOM, DT-Global in DQA

2016, 2017, 2018, and 2020 17. Alternative Basic Education for Pastoralists (ABE) – UNICEF in DQA 2016, 2017,

2018, and 2020 18. Growth, Enterprise, Employment, and Livelihoods (GEEL) – IRG, RTI in DQA

2016, 2017, 2018, and 2020 19. Support for International Family Planning and Health Organizations 2: Sustainable

Networks (SIFPO2) – Population Service International (PSI) in DQA 2018 20. Bringing Unity, Integrity and Legitimacy to Democracy (BUILD) – Creative

Associate in DQA 2017, 2018, and 2020 21. Expanding Access to Justice Program (EAJ) – PACT in DQA 2020 22. Strengthening Public Accountability in Parliament (Damal, meaning “Gathering

Place” in Somali) – Chemonics in DQA 2020

In relationship with USAID Somalia PO, SPSS designed and administered various tools to assess the selected performance indicators and the systems within which the Mission and IPs deployed approaches, plans, and techniques to collect data and report on their performance. For both system-level and indicator quality standard-level information, SPSS mobilized and operationalized USAID definitions and agreed protocols.

38

During the DQAs in 2014, 2015, and 2016, SPSS operationalized the USAID data quality criteria based on USAID priorities and the context under which IPs collect, analyze, collate, and report performance data. For example, DQA 2014, which was prepared quickly by IBTCI during the project start-up period, focused on data reported by IPs in Q3 FY 2014; for TIS and PEG, USAID found data issues that needed thorough reviews by the DQA team. The criteria were then operationalized to focus on data collection protocols used by IPs, the collation systems, and reporting formats. DQAs in 2015 and 2016 used the five standard criteria to assess 18 indicators’ performance data for TIS and SYLI in 2015, and 100 indicators for TIS, TIS+, ABE, SSG, SYLI, and GEEL in 2016.

In 2016 DQA data validation workshop, USAID pointed out the need to design a comprehensive Indicator Checklist Tool (ICLT) in order to harmonize approaches to assessing data quality under each standard criterion. SPSS then adopted a detailed ICLT in DQA 2017, built on lessons learned from DQA 2014 and 2015 and USAID PO guidelines, which was finalized and stabilized in DQA 2017. To better understand how data is collected and generated by IPs, during SPSS start-up phase and in the inception phase of the DQA 2014, the DQA teams identified various levels of data management and reporting that included USAID Somalia, IP Head Offices in Nairobi, and IP Field Offices in Somalia. SPSS implemented the DQA in various places to cover all the layers of data collection, including Nairobi in Kenya, and Mogadishu, Hargeisa, and Garowe in Somalia. Lessons learned from DQAs 2014 and 2015 led to a classification of the M&E systems of USAID and IPs into levels that improve understanding of roles and responsibilities of the various actors in performance data collection and reporting. These levels that SPSS adopted progressively and where specific and more tailored DQA tools were generated are as follows:

1. The Central Level, which comprises USAID/Somalia PO and A/CORs. The PO is the primary responsible for planning and managing the Mission-wide M&E System, defining protocols and techniques, operationalizing approaches and frameworks defined in the Mission Performance Management Plan (PMP). The PO manages SPSS, which provides technical support to the Mission in M&E and reporting first through the CH and then the Indicator Performance Tracking Table adopted in FY 2016. The A/CORs have the responsibility of supervising IP M&E and reporting systems with support of the USAID PO. They also have the responsibility of selecting Performance Plan and Reporting (PPR) indicators, and validating and approving the IP reports and performance data.

2. The Intermediary Level, which comprises IP management and technical staff, including the M&E Manager at the Head Office generally situated in Nairobi [until recently, when some IPs (Damal, GEEL, and BUILD) started having their Head Office in Mogadishu]. The intermediary level is responsible for the Activity Monitoring Evaluation and Leaning Plan (AMELP) design and implementation. Indicators are defined at that level and also data collection approaches tools before their application at subsequent levels.

The intermediary level is responsible for preparation, consolidation, and submission of progress reports and performance data matrices to USAID A/CORs. It is also responsible for capacity-building of various subcontractors/grantors and local stakeholders at the field level, where primary data is collected at sites and communities hosting activities implementation.

39

3. The Primary Level comprises various actors responsible for data collection and reporting at the field level. They include sub-IP staff, government representatives participating in data collection and reporting, local organizations, and institutional staff involved in data collection and reporting to the IP at the intermediary level.

In relation with PO and A/CORs, SPSS designed tailored tools from lessons learned in DQAs 2014 and 2016. SPSS developed specific ICLT tools for the Intermediary and Primary Levels.

The main tool that assesses M&E systems at various layers is the System Assessment Protocol (SAP) tool. The SAP tool assesses six dimensions of the projects’ M&E system: (1) M&E Structure, Functions, and Capabilities; (2) Indicators Definition and Reporting Guidelines; (3) Data Collection and Reporting; (4) Data Management Processes; (5) Feedback and Dissemination; and (6) Reporting Performance. The DQA team assessed the M&E Structure, Functions, and Capabilities at the USAID PO and at the IP intermediary and primary levels.

Figure 2: Linkages between Data Management Processes and Data Quality Standards

In line with the TOs’ scopes, SPSS implemented the DQA in various steps using

commensurate approaches along the way. These steps included the following: 1. DQA preparation

Prior to issuing the DQA TO, USAID PO and SPSS engage consultations with the A/CORs, organize discussions on the status of the PPR indicators, IP AMELP and PIRS, IPTTs, and performance reports analysis.

Through these preparatory activities, SPSS supported the Mission to identify indicators that needed quality assessment, generally by applying two criteria:

1. Missions must conduct a DQA for each performance indicator reported to external entities every three years (those included the PPR indicators reported to Washington, and other indicators the Mission will have to share with external stakeholders).

2. What are the indicators most relevant to USAID management and decision-making processes?

40

This approach was consistently used over the lifecycle of SPSS, with the exception of DQA 2014, which was launched during the SPSS start-up period and covered only six indicators whereas the Mission wanted to dig more and better understand data issues identified. At the end of the preparatory phases, the USAID PO finalized the indicators to include in the DQA TO.

During the 2020 DQA, which took place in the context of the COVID-19 pandemic, the preparatory phase included the request for a COVID-19 M&E Contingency Plan for each of the IPs to ensure that new M&E approaches adopted during the pandemic are factored, approved by USAID, and considered as an addendum to the IP’s approved AMELP prior to starting the DQA.

At the end of these steps, SPSS prepared a Letter of Information (LI) that USAID addressed to the respective IPs. The LI contained DQA milestones, dates of DQA sessions, and needs for documentation, as well as session fieldwork schedules. It was shared with IPs generally three weeks before fieldwork sessions started for DQA 2018 and 2020. Specifically, for DQA 2020, SPSS integrated into the LI a request for IPs to update data collection, analysis, and reporting protocols used at the field level to generate the data under each indicator during the COVID-19 context.

While the LI is drafted during the preparatory phases, it is only finalized and shared with IPs when the DQA TO is fully executed. Following the request of the USAID PO, the preparation of the DQA 2020 included a remote workshop with USAID staff including A/CORs.

2. Documentation review

SPSS experts and staff reviewed all relevant documents related to the 213 indicators assessed across 11 activities and six DQAs. The main objective of the desk review is to permit the DQA team to start collecting information on M&E systems, indicator data performance, data collection tools, and approaches defined in the AMELPs and Performance Indicators Reference Sheets (PIRS).

The documents also include various job descriptions of IP staff and subcontractors and local organizations staff involved in the M&E systems. The documentation review helped SPSS DQA teams to collect initial information and better prepare the fieldwork and data sources review. Over the six DQAs, SPSS stored in Google Drive folders and analyzed more than a thousand documents that relate to the indicators assessed for the 11 activities.

3. Sensitization Workshops

SPSS organized at minimum 33 DQA sensitization workshops over its implementation with more than 100 participants. The sensitization workshops were organized into three sessions: one session devoted to USAID staff, one session for IP staff at Head Office, and one session with IPs, subcontractors, and local organizations during the DQA fieldwork. The sessions with USAID staff focused on presentation of DQA standard criteria, indicators selected, approaches to assess data quality and M&E systems, DQA fieldwork plans, lessons learned from previous DQAs, recommendations for field visits and discussions on participation of USAID staff during the process, data validation workshop preparation, and any issues pointed by the PO and A/CORs.

41

The sessions for IP staff at Head Office covered the indicator lists, data quality criteria, SAP criteria, presentation of the DQA tools, and required additional documentation. The sessions also discussed IP data management protocols and storage of source documents for verification during the DQA fieldwork.

The sessions with IPs at the field level and local organizations focused on data collection processes, documentation needed, and lists of stakeholders for data verifications at the field level. The sessions also discussed their role in the M&E system and capacity-building support they received from the IP. All DQA sensitization sessions were organized face-to-face with the exception of sessions organized for DQA 2020 due to the COVID-19 pandemic. The remote sessions in DQA 2020 recorded increased participation and engagement from USAID and IPs including those at the primary level.

4. Fieldwork

SPSS mobilized more than 10 experts to implement the DQAs including the DQA Manager/Team Leader, SPSS COP, Senior M&E Advisor/DCOP Technical, a Security Specialist in DQA 2014 and 2015, and MV&R staff in DQA 2020. These experts traveled to Mogadishu, Hargeisa, and Garowe in Somalia to meet IPs and local stakeholders, and held meetings with IP Head Office staff in Nairobi, Kenya. The DQA fieldwork sessions focused mainly on administration of the DQA tools, namely, the ICLT and the SAP tool. During the fieldwork sessions, DQA teams organized various internal brainstorming discussions to summarize the data collected and assemble initial conclusions, or highlight areas that needed more attention during subsequent field sessions.

The field sessions involved review and analysis of data sources and a deep check of IP datasets, databases, and various information related to the M&E systems and indicators being assessed. The sessions also included discussions on emerging findings and potential actionable recommendations. For DQAs organized from 2014 to 2018, the team held discussions to update the PO and A/CORs on the process and prioritized USAID-focused areas in data management and M&E systems for the specific IPs and indicators selected. PO staff and A/CORs took part entirely in the DQA 2020 fieldwork sessions, which were organized remotely due to the COVID-19 pandemic.

5. Data analysis

Data analysis started from the DQA’s first step, about documentation review, and continued to the submission of the DQA final reports with a focus during fieldwork and after. Progressively, learning from previous DQAs and following new guidelines from USAID PO, SPSS designed more tailored and consistent approaches to data analysis and reporting processes. These approaches were presented to USAID and used during data analysis sessions. They also were shared during the data validation sessions to better explain how the findings and recommendations on data issues were generated from the documents and fieldwork interviews and discussion, source documents, and dataset reviews.

42

Table 5: Data Quality Risk Scores

To assess quality standard for the selected indicators, SPSS designed and applied the Data Quality Risk Score analysis technique, resulting in the color-coding shown above. Scores are generated with that technique using information collected during the DQA sessions and are summarized in the table below.

Table 6: Data Quality Standard Risk Scores interpretation

6. Data Validation

Over its lifecycle, SPSS organized 11 DQA data validation workshops with more than 100 participants from USAID and IPs. In line with the DQA TOs’ scopes, SPSS organized the Data Validation workshop to present DQA findings, initial conclusions, and recommendations for validation with USAID and IPs and discussions on the way-forward. The Data Validation workshops were organized using a participatory approach to allow IPs feedback on initial conclusions and recommendations. These sessions helped SPSS better contextualize the conclusions and recommendations.

RISK ASSESSMENT SCORE

Scoring in DQA tools

IMPLICATIONS FOR DATA

1–4 3 Data quality is well within acceptable parameters. No action needs to be taken to maintain data quality. Maintain systems as they are.

6–8 2

There are concerns regarding data quality that need to be addressed. Data quality is not critical; however, decisions based on this data may not result in anticipated outcomes. Data should be reported with caution.

9–12 1 Data is of very poor quality and should not be used for decision-making as it is. Data management processes and quality control of these processes require action prior to use of data.

16 0 Data has errors that render the data not fit for use in decision-making. Discard data and revise indicator.

43

7. Reporting

In total, SPSS prepared and submitted, each on time, 39 deliverables for the six DQAs implemented over the LOP. The deliverables were organized by USAID Somalia in the following seven categories: (1) Training/Sensitization materials, tools, and protocols to be used during fieldwork; (2) Training/ Sensitization Workshop Report; (3) Indicator Assessment Summary Matrix (input for PPR), which started in DQA 2017; (4) Data Validation Workshop

Materials, which was considered as deliverable only for DQA 2016, and prepared in all other DQAs as a prerequisite to organize the Data Validation Workshop, but not as a deliverable; (5) Data Validation Workshop Report, which in some instances (DQAs 2015, 2016, and 2018) included separate Data Analysis Workshop, and Data Validation Workshop reports; (6) Draft DQA Report; and (7) Final DQA Report.

The table below shows the details by DQA and type of deliverable.

Table 7. DQA deliverables by year

Deliverables DQA 2014 DQA 2015 DQA 2016 DQA 2017 DQA 2018 DQA 2020 Total Training/sensitization materials, tools, and protocols

1 2 1 1 1 1 7

Training/Sensitization Workshop Report 1 1 1 1 1 1 6

Indicator Assessment Summary Matrix (input for PPR)

0 0 0 1 1 1 3

Data Validation Workshop Materials 0 0 2 0 0 0 2

Data Validation Workshop Report 1 2 2 1 2 1 9

Draft DQA Report 1 1 1 1 1 1 6

Final DQA Report 1 1 1 1 1 1 6

Total 5 7 8 6 7 6 39

8. DQA Recommendations Follow-up

Following up on IP corrective actions taken to address data quality issues identified by SPSS DQAs is essential for USAID/Somalia to ensure that data reported by IPs are of quality, and also to ensure increased M&E capability of IPs mainly at local level and especially for government and local organizations’ staff. To translate this requirement into action, SPSS organized sessions on previous DQA recommendations during sensitization workshops and fieldwork from 2014 to 2017, and designed a DQA Recommendations follow-up tool in relation with USAID PO in 2018. A section on corrective actions taken to address data issues in previous DQAs in integrated into the DQA Draft and Final Reports.

44

The DQA Recommendations follow-up tool was used in DQAs 2018 and 2020 and helped better monitor corrective actions that IPs implemented to address data quality issues identified. Some A/CORs (e.g., EAJ) took initiative to organize with SPSS sessions on the DQA recommendations implementation plan immediately after the DQA Final Report was approved by the USAID PO. This was a very beneficial initiative for IPs and data quality over USAID portfolio.

EVALUATIONS & ASSESSMENTS

Over the SPSS LOP, USAID Somalia tasked IBTCI to design and manage performance evaluations of individual activities and/or projects implemented by USAID Somalia IPs. These evaluations included baseline, midline, and end-line evaluations. IBTCI was asked in different TOs to evaluate impact against intermediate results (IRs) and/or overall stabilization/ development objectives. These were accomplished through attitudinal surveys, focus groups discussions (FGDs), key informant interviews (KIIs), and other participatory methods applied to beneficiaries, communities and local organizations.

The following table provides a summary of evaluations and assessments over the SPSS LOP.

Table 8: List of evaluations and assessments done over the SPSS LOP


1 AID-623-TO-15-00001 05/04/2015 to 11/30/2015 Somali Youth Learners Initiative (SYLI) Baseline Evaluation

2 AID-623-TO-15-00006 07/16/2015 to 07/31/2016 Partnership for Economic Growth (PEG) Evaluation

3 AID-623-TO-15-00014 09/30/2015 to 07/30/2016 Transition Initiatives for Stabilization (TIS) Evaluation

4 AID-623-TO-15-00015 09/30/2015 to 01/31/2017 Baseline System Development for Strengthening Somali Governance (SSG)

5 AID-623-TO-16-00015 05/30/2016 to 07/25/2019 TIS+ Baseline and End-Line Assessment

6 TO#72062318F00003 02/01/2018 to 03/30/2018 Gender Analysis 2017 7 TO#72062320F00007 08/10/2020 to 06/30/2021 Citizen Consultation and Gender Assessment 2020

The SYLI Baseline Evaluation utilized a mixed-methods approach, which included a combination of document review, quantitative surveys, classroom observation, student examinations, FGDs, in-depth KIIs, and stakeholder meetings. The combination of methods established baseline measures on a number of indicators for SYLI from which to assess changes in outcomes achieved by the activity at end-line. The data collection process in 15 secondary school communities took place in Somalia from June 4 to 20, 2015, and was rolled out in six regions in 11 districts. SPSS contracted IPSOS, a French-affiliated survey and research company, to collect and enter data and carry out preliminary data analysis. IPSOS subcontracted Data and Research Solutions (DARS), a Somali-based research and consulting company, to carry out the field data collection in targeted schools.

45

The evaluation concluded (among other things) that there was agreement in both the quantitative and qualitative findings that the most significant factor positively influencing learning outcomes was teachers. The most significant community-level factor positively influencing learning outcomes was parental support and encouragement for their secondary school-age offspring. As parental education levels rise there is an increase in educational expectations for their children. There was an overwhelming consensus that the most significant community-level deterrent to learning outcomes was poverty and a lack of household income for educational expenses. The role of conflict and insecurity was also seen, in both qualitative and quantitative findings, as having a negative impact on learning outcomes. However, participants in FGDs and KIIs perceived that the security situation had improved recently and that this had led to increased secondary school enrollment.

The PEG Evaluation was conducted from 2015 evaluating the PEG activity that had been carried out for a period of 18 months (starting in March 2014) in South Central Somalia by Development Alternatives Inc. (DAI), through its local implementing partner, the Somalia Agriculture and Technology Group (SATG). It examined how successful the PEG models4 were in achieving their objectives, and how to inform future USAID economic growth programming. SPSS subcontracted HACOF, a local Somalia company, to collect both quantitative and qualitative data. The evaluation conducted 16 FGDs, 32 KIIs, and 481 farmer interviews all over Somalia. However, upon completion and presentation of the analysis of data, on September 24, 2015, USAID raised specific concerns with the data analysis and interpretation of results, and commented that the data analysis was very superficial, focused far too heavily on descriptive analysis, and had little or only generic context or interpretation. Subsequently, the USAID/ Somalia Office approved a modification to the TO on January 8, 2016, that enabled SPSS to implement a report improvement plan with an emphasis on collecting additional and/or augmented quantitative data that would provide accurate and comprehensive information in response to USAID/Somalia Office comments on the draft report. SPSS then subcontracted DARS, a more experienced Data and Research Company in Somalia, who conducted a fresh set of FGDs, KIIs, and farmer interviews.

The evaluation found that PEG interventions (input support and training about best practices) led to positive agricultural productivity. Evidence of impact was seen in the changes of mean yields among farmers who adopted new maize technologies with increases up to 300 percent. Farmers adopting new tomato technologies achieved a 32 percent increase in mean yields from 2,400 kg/Ha to 3,200 kg/Ha.

4 One of the models was an implementation model used by PEG, namely implementation of sub-activities through a local partner. The other was the cascade model which tested the effectiveness of the reach of the training and technology interventions.

46

The evaluation found that livestock interventions had a statistically significant, positive impact on the livestock farmers in the areas of fewer animals dying, ability to produce more milk, ability to own more animals, and increases in milk sales.

Both male-headed households and female-headed households indicated that PEG had a positive impact on women’s involvement in decision-making across production, resources, income, leadership, and time. While female respondents and female-headed households had equivalent increases to men, with regard to production increases, they were less likely to communicate technologies onward to other women.

PEG tested a model of onward transmission of innovations by the farmers directly reached, multiplying the use of innovations outward. The cascade model was found to have worked and PEG was able to reach 200 lead farmers and 2,810 farmers through direct contact, and indirectly reach 21,000 farmers.

The evaluation found that PEG had positive outcomes in terms of household income Seventy-five percent of the agriculture households who adopted maize production technologies indicated realizing more income while 65 percent of the livestock farmers who adopted milk hygiene technology indicated realizing more income.

The evaluation found that PEG improved food security among the affected households. The percent of households who reported food self-sufficiency for 10 months and above almost doubled from 9% before the PEG program to 17% after PEG inputs. The number of households who were food self-sufficient for two months or less reduced by half from 35% before PEG to 17% after PEG interventions.

Under the TIS Final Performance Evaluation, SPSS examined the TIS project, a five-year initiative implemented from 2011 to 2016 in Somaliland, Puntland, and South Central Somalia by two USAID IPs: Development Alternatives Incorporated (DAI) and the International Organization for Migration (IOM). From September 2015 to July 2016 SPSS assessed the extent to which USAID’s stabilization activities in Somalia have met the project goal to increase stability in Somalia through targeted interventions that foster good governance and economic recovery and reduce the appeal of extremism in selected settlements. SPSS also provided USAID the lessons about conducting stabilization activities in Somalia to improve future programming. The evaluation reviewed documents, conducted 87 FGDs, 221 KIIs, and a survey of sample size 1,719 across all of Somalia.

SPSS found that TIS had contributed to increasing confidence in the ability of local government to improve lives; the level of professionalism of local government; community satisfaction with basic services; collaboration between civil society, private sector, and local government; and stability in TIS communities. SPSS found that TIS processes of collaboration and transparency had been or would be adopted by several important entities and actors. The evaluation also found that most of the physical structures built by TIS continued to be put to good use, were used as intended, and were equitably accessible. Through the process of gender inclusive, community-based consultation, TIS had contributed to increasing the role of women in community-level decision-making.

Some of the challenges that SPSS identified included nepotism, which was seen to have increased everywhere except one area, though corruption was perceived to have decreased. A number of infrastructure quality-related issues were reported, with occasional concerns about their location.

47

Under the SSG Baseline Systems Development, SPSS carried out specific tasks which included the following:

1) Working in close collaboration with USAID and Chemonics to:

• Review and refine the theory of change (TOC) to articulate measurable relationships between SSG interventions, objectives, and goal; and identify measurable ways in which these contribute to USAID Somalia’s development hypothesis.

• Refine or revise indicators that can be used to test the TOC during the final evaluation, focusing on outcome-level indictors. These indicators will derive from scorecards or a similar methodology, to be developed by SPSS in close collaboration with Chemonics and USAID.

2) Identifying indicators to measure progress against SSG’s goal and identifying or developing a measurement tool or methodology to assess those indicators.

SPSS examined documents, refined the SSG TOC, and conducted KIIs with multiple stakeholders at USAID and other organizations. SPSS refined the Results Framework (RF) for SSG, going through six iterations, with each iteration producing a refined set of indicators. SPSS also used the following tools and methodologies to determine the baseline values for each of the outcome indicators in the SSG RF:

• Public Expenditure and Financial Accountability Marker (PEFA) Tool; using our Most Significant Change (MSC) Tool

• Advocacy Capacity Tool (ACT)

• A set of benchmarking tools

Each of the above tools were applied to a number of SSG outcome indicators and a number of baseline values were determined for future reference.

Coupled together with the SSG Baseline was the Somalia Citizen Perception Survey, which was mainly quantitative, and a questionnaire was administered to all of Somalia with a sample of 1,920 respondents. The questionnaire had a mix of closed-ended with questions containing predetermined answers and a few open-ended questions with no predetermined answers. All data collected was disaggregated by gender, region, geographic area, and age. The respondents were asked how much exposure they had to governance systems outside Somalia (through residence or travel elsewhere, or contact with diaspora). Questions focusing on the following additional areas of interest were also included:

• Political milestones including the upcoming elections, federalism, and constitutional referendum

• Knowledge and perception of Xeer Soomaali (customary law) • Role of Xeer Soomaali and Shariah in governance and building democratic

institutions • Women’s role in conflict resolution mechanisms

The TIS+ baseline and end-line assessment was conducted on the TIS+ activity, a

five-year, US$66.9 million project implemented by AECOM that ran from May 2015 to April

48

2020, with a one-year extension to May 2021. The assessment was carried out by SPSS to generate a better understanding of the extent to which TIS+ interventions had achieved their objectives and contributed to USAID’s goal of increased stability in Somalia through participatory processes that promote good governance and community cohesion. Specific clusters or geographic areas were strategically chosen to assess the influence that the TIS+ program activities had on mitigating the effects of violent extremism and insurgency in these high-risk areas of Somalia.

These clusters included Banadir, Lower Shabelle, Lower Juba, Gedo/Bay, and Bakool. The assessment was implemented in four phases (May 2016–July 2019): Phases 1 and 2 – baseline data collection (pre-intervention); Phase 3 – midline data collection (Wave I and II); and Phase 4 – cross cluster analysis. During the assessment, documents were reviewed throughout the assessment period, 137 FGDs were conducted, 178 KIIs took place, and 4,174 households were surveyed.

The assessment found the following:

Confidence in Government: All areas demonstrated increased scores since the baseline on the “Good Governance Index,” a composite ranking of three themes: (1) confidence in government, (2) service delivery, and (3) stakeholders’ partnership. Increased insecurity, unfulfilled election promises, victimization by Al-Shabaab, and poor service delivery contributed to decreased confidence in the government in most areas.

Increased Satisfaction of Service Delivery by the Government: In places where the government participated in the management/governance of service delivery, respondents had an increased perception of the government as a provider. The government did not have to provide anything as long as it was present.

Increased Community Ownership: Through the TIS+ community consensus processes, there was an increase in citizen participation throughout projects and because of equal participation of men, women, youth, and persons with disabilities the clan elders felt excluded because they traditionally dominated such forums.

Reduced Conflicts over Resources: TIS+ operated areas showed reduced conflicts over resources. There was a greater increase of reliance on local structures (clan elders, peace building and state administrators) to mediate these conflicts.

Al-Shabaab Service Delivery Threatens Stabilization Gains: Al-Shabaab is growing increasingly sophisticated in services it offers, building its legitimacy while undermining the legitimacy of the government.

The purpose of the 2018 Gender Analysis was to identify critical gender-related challenges and opportunities facing USAID Somalia’s Transition Strategy and Transition Objectives (TO) in all focus sectors and to make a series of recommendations to strengthen the gender aspects of USAID Somalia’s policies, practices, and programming.

49

SPSS reviewed materials that had been published between 2015 and 2018, with most material published in 2016–17 and several key documents published in 2018. These materials were arranged into a topical literature list to generate a gender knowledge base for USAID Somalia and its IPs. SPSS also conducted ten KIIs with USAID Somalia staff, IPs, and technical and gender experts to ground-truth information and obtain nuanced gender insights. An online survey was conducted to assess the extent to which USAID Somalia and its IPs implemented recommendations from the 2014 Gender Analysis.

The assessment documented general constraints, findings, opportunities, and recommendations across USAID Somalia TOs, including cross-cutting recommendations for processes, procedures, and practices. Specific findings, opportunities, and recommendations were obtained for TO1: systems and processes that enable inclusive governance strengthened; TO2: service delivery improved; and TO3: inclusive economic growth expanded.

The objective of the USAID Somalia 2020 gender assessment was to assemble and analyze evidence that identified macro-level gender issues, inequality, constraints, and opportunities relevant to USAID Somalia’s 2020–2024 Country Development Cooperation Strategy (CDCS) priorities. The qualitative gender assessment methodology included an extensive desk-based literature review and KIIs with a wide variety of Somali and international interlocutors, and especially individuals who are aware of the local context in Somalia and could provide unpublished information and insight. The assessment team reviewed about 235 existing information sources, especially documents published since 2018. The team also conducted 75 KIIs with a total of 106 individuals (65 women and 41 men) representing diverse organizations based on a standardized protocol, which was adapted as needed for different interlocutors.

The key findings were presented according to the five domains for gender analysis in the Automated Directives System (ADS) 205. Analysis under each domain was done referring to each of the two Development Objectives (DOs) in the new draft CDCS, namely countering violent extremism and building resilience.

Recommendations were then presented according to the draft CDCS DOs: DO1: Countering Violent Extremism; DO2: Building Resilience; and General (Cross-Cutting) Recommendations.

The Citizen Consultation sub-activity was carried out by IBTCI through a subcontract arrangement with the Africa Voices Foundation (AVF) from September 2020 to June 30, 2021, following a recommendation by USAID to work with AVF. The one-year pilot activity comprised a six-part radio series (divided into two series of three shows each) and an integrated social media (Facebook) amplification campaign, resulting in both qualitative and quantitative findings designed to inform initial recommendations for USAID on how to improve the design and implementation of its strategy and activities. The radio series and social media campaign broadly focused on citizens in Banadir and South West State of Somalia. AVF deployed its Common Social Accountability Platform (CSAP), which has an established radio “brand” in Somalia, known as Hage, meaning “Guidance” in Somali to deliver the interactive radio in two series.

During series 1 of the radio shows, which took place from November 5 to 19, the discussions focused on the extent to which young people engage in decision-making in their community and the social and civic engagement activities of youth in their communities.

50

The radio shows were aired across a network of ten radio stations (see Annex 2 of Citizen Consultation Deliverable #7 already submitted and approved by USAID). Each radio show included relevant guests to facilitate a discussion and answer questions in relation to the messages sent by participants to the radio promos and SMS ads sent before the show. Listeners were invited to share their views to the toll-free short code 384. The results from the shows in series 1 were documented in Deliverable #7 (Series 1 Report) submitted and approved by USAID.

During the three weeks of the series 1 interactive radio shows, AVF posted a number of items to two Facebook accounts (Atosh and Faisal Dhayow) in parallel with the radio shows to further the interaction of both radio and Facebook discussions. Each week, one post contained the question of the week, one post contained the radio show schedule, and a third post included a video containing youth voices responding to the question and one voice clip from the actual show for the three weeks of the interactive radio series. People engaged in various ways (commenting, clicking on and viewing content, sharing, or reacting through, for example, likes) but only comments were analyzed. Engagement figures in relation to Facebook content include only comments. For more detailed engagement metrics please see Annex 4 of Citizen Consultation Deliverable #7 already submitted and approved by USAID.

A similar approach was used by AVF to deliver series 2 citizen engagement over three weeks in February and March 2021. However, the questions for engagement were different from series 1. These questions were as follows:

• For the small businesses owned by women in your community, where do you think women currently find support to start these businesses?

• What challenges do women face in trying to make a small business successful? • What changes are needed to help women access financing (such as grants and

loans) for their small businesses?

The results of series 2 engagement were documented in deliverable #12, Series 2 Report that was submitted and approved by USAID.

In both series 1 and 2, USAID, AVF, and IBTCI participated in design workshops in which questions for engagement were crafted and approved by all parties before the engagements of citizens began.

Monitoring, Verification, and Reporting (MV&R)

USAID’s ADS 201 stipulates that USAID/Somalia is responsible for tracking Implementing Partners’ (IP) performance and ensuring that they are accomplishing the tasks they set out to achieve. Delays in completing tasks or producing outputs provide an early warning that results may not be achieved as planned. USAID/Somalia is also responsible for monitoring partner compliance with 22 CFR 216 environmental determinations, and their Branding and Marking Plans. Given that travel to Somalia for USG staff is currently restricted, USAID/Somalia utilized the services of IBTCI (a third-party M&E Contractor) to provide technical performance management support services and conduct monitoring and verification visits to all the USG-supported sites.

Over the course of SPSS, IBTCI received seven MV&R Task Orders, and conducted a total of 2,696 verifications across all regions of Somalia for the following USAID Somalia activities:

51

17. Somali Youth Learners Initiative (SYLI) – Mercy Corps 18. Partnership for Economic Growth – Development Alternatives INC (DAI). 19. Transition Initiatives for Stabilization (TIS) – International Organization for

Migration (IOM) 20. Transition Initiatives for Stabilization (TIS) – Development Alternatives INC (DAI) 21. Somalia Towards Reaching Resilience Project (STORRE) – CARE 22. Program to Enhance Resilience in Somalia (PROGRESS) Catholic Relief Services

(CRS) 23. Resilience and Economic Activity in Luuq (REAL) – World Vision International

(WVI) 24. Strengthening Somali Governance, led by Chemonics (SSG-Chemonics) 25. Alternative Basic Education for Pastoralists (ABE) – UNICEF 26. Transition Initiatives for Stabilization Plus (TIS+) – DT Global 27. Growth, Enterprise, Employment, and Livelihood (GEEL) – International Resource

Group (IRG) 28. Office of Transition Initiatives (OTI) – International Organization for Migration

(IOM) 29. Bringing Unity, Integrity, and Legitimacy to Democracy (BUILD) – Creative

Associates 30. Expanding Access to Justice (EAJ) – Freedom House/Pact 31. Strengthening Public Accountability in Parliament (Damal) – Chemonics 32. Bar ama Baro – “Teach or Learn” – Creative Associates International

52

Figure 3: Total SPSS Site Visits by States

Using the SPSS automated flagging system that allocates a flag to each verification, 2,257 of the 2,696 verifications (84 percent) were allocated a green flag because they met the agreed minimum implementation standards. One hundred and twenty-four verifications (5 percent) were allocated a yellow flag, which indicated some areas of concern that the IP needed to investigate further or clarify. Eighty-one (3 percent) were allocated an orange flag because these sites were not ready for verification during the data collection period despite their selection and approval prior to the site visits. Thirty-two verifications (1 percent) were allocated a red flag because the activities were reported as completed in the quarterly reports but were not verifiable by SPSS. Whenever SPSS introduced a new tool for MV&R, this tool would not have a flagging system and was piloted during its first use. As a result of piloting of new tools, 202 verifications (7 percent) were not allocated any flags. The table below summarizes the MV&R flag status by IP. Details of the flags allocated are found in the respective consolidated reports over the life of MV&R.

Table 9: SPSS Flagging Status by IP

Program / IP Green No Flag/ Tool Pilot

Orange Red Yellow Total

ABE–UNICEF 233 50 17 22 322 BAB–Creative Associates 3 3 BUILD–Creative Associates 141 4 16 6 167

Somaliland -

Puntland

Federal Member States –

Mogadishu

53

Program / IP Green No Flag/ Tool Pilot

Orange Red Yellow Total

Damal–Chemonics 28 28 EAJ–Freedom House/ Pact 61 61 GEEL–IRG 386 38 22 22 28 496 OTI–IOM 199 3 3 7 212 PEG–DAI 73 15 5 93 PROGRESS–CRS 70 28 1 99 REAL–WVI 69 16 1 86 SSG–Chemonics 161 6 3 170 STORRE–CARE 48 22 1 71 SYLI–Mercy Corps 258 9 22 289 TIS+–DT Global 324 10 23 7 18 382 TIS–DAI 100 4 6 110 TIS–IOM 103 4 107 Total 2,257 202 81 32 124 2,696

The 2,696 verifications were conducted across 16 different sub-activities as categorized by the Tools and Protocols developed by SPSS and revised over the life of the project. MV&R tools are designed for each type of sub-activity (training, construction, events, goods and supplies, etc.) as well as for USAID compliance issues in environmental monitoring for certain types of sub-activities. Different tools were designed within each sub-activity to reflect stages of implementation (e.g., ongoing construction is one tool, post-construction is a different tool) and type of respondent (e.g., the trainer is one tool, the trainees are a different tool). Each tool has its own protocol that guides the field monitor in administration of the tool. New tools categories were added to the introduction of new sub-activities for MV&R. There were instances when the tools were used across all USAID activities (Training, Supply of Goods, etc.) and there were instances when SPSS designed new tools that were specifically targeting certain unique sub-activities being implemented by a specific activity, e.g., the community savings and lending scheme tool that was designed for the ABE activity. Table 10 below summarizes the verifications conducted under various sub-activities over the life of SPSS, and Table 11 summarizes the verifications conducted for each IP across the years.

Table 10: SPSS Verifications by Sub-Activities over time

Sub-Activity

Yea

r 1

Yea

r 2

Yea

r 3

Yea

r 4

Yea

r 5

Yea

r 6

Yea

r 7

Tot

al

Benefits Mapping 13 86 116 68 11 28 322 Cash for Work 5 18 23 Community Education Committee 9 18 5 6 38 Community Savings and Lending Scheme 24 22 46 Construction 99 37 47 107 69 9 24 392 CtC Clubs Formation 15 8 5 28 Door-to-Door Campaigns 2 9 11 Environmental Monitoring 1 3 20 14 5 43

54

Sub-Activity

Yea

r 1

Yea

r 2

Yea

r 3

Yea

r 4

Yea

r 5

Yea

r 6

Yea

r 7

Tot

al

Event 22 54 206 90 91 14 80 557 Grant Management 1 19 9 2 20 51 Local Education Authority Supervision 10 12 3 25 Small Scale Infrastructure 14 7 12 3 14 50 Supply of Goods 64 40 82 51 40 7 23 307 Technical Assistance 13 1 12 45 22 7 34 134 Tender 8 12 8 4 6 38 Training 77 91 265 84 40 11 63 631 Total 296 253 787 594 394 75 297 2,696

Table 11: SPSS Verifications by USAID Somalia Activities over time

Implementing Partner

Yea

r 1

Yea

r 2

Yea

r 3

Yea

r 4

Yea

r 5

Yea

r 6

Yea

r 7

Total

Damal–Chemonics 28 28 EAJ–Freedom House/ Pact 10 51 61 GEEL–IRG 108 155 130 18 85 496 OTI–IOM 143 69 212 PEG–DAI 93 93 PROGRESS–CRS 24 75 99 REAL–WVI 14 72 86 SSG–Chemonics 55 88 27 170 STORRE–CARE 19 52 71 SYLI–Mercy Corps 74 53 162 289 TIS+–DT Global 128 107 71 17 59 382 TIS–DAI 59 51 110 TIS–IOM 70 37 107 BUILD – Creative Associates 25 44 41 10 47 167 ABE–UNICEF 77 118 83 20 24 322 BAB–Creative Associates 3 3 Total 296 253 787 594 394 75 297 2,696

SPSS DATA COLLECTION AND MANAGEMENT

Third-Party Monitoring (TPM) Induction Workshops

SPSS conducted periodic induction/orientation/refresher sessions for all the SPSS staff in Nairobi and Somalia who were using the suite of tools and protocols developed for all sub-activities. Each orientation workshop sought to refresh all participants’ knowledge on respondent selection procedures, interview protocols, quality control measures, and how to

55

use mobile devices and survey software, as well as improve problem-solving capabilities using scenarios and role playing.

During pre-deployment, SPSS sent approved tools for verifications to every monitor involved in the SPSS third-party monitoring before the induction workshop for their comprehension and understanding. At the TPM induction workshop, SPSS introduced the new tools to the TPM management and field monitors. Field monitors then practiced and role-played using the tools on their tablets and uploaded dummy data to the platform. The practice sessions generated issues related to language usage, the intent of questions, and the scripting of the tools. SPSS then made the final adjustments to the questions and scripting and uploaded the final forms to the Askia platform. SPSS finally instructed the field monitors to download the ready applications for data collection.

SPSS conducted these induction workshops through physical workshops either in Hargeisa or Mogadishu; however, due to the COVID-19 pandemic, SPSS could not hold physical workshops from MV&R VII R2 of verification through to the last round (MV&R VII R6). The team improvised and held online workshops through Google Meet. The online workshops included a joint session between all the TPM firms and SPSS, then a breakaway into individual TPM sessions virtually attended by a member of the Nairobi team.

1) Purpose of the SPSS MV&R Tool Induction Workshops The purposes of the workshops were to build the knowledge and strengthen the

practical skills of the local partner Monitoring Firms (doing the TPM), Team Leaders (TLs), and Field Monitors (FMs) about the tools; the changes in the scope of what each tool covers; compliance to protocols and standards; and other aspects. The following modules were always covered during the Induction Workshop:

Figure 4: Modules covered during the Induction Workshops

During the induction workshops, facilitators ensured that the sessions were conducted within allocated timelines. The discussions were lively and added value to participants’ learning. Each day included a 15-minute morning tea break, a 75-minute break for midday prayers and lunch, and 45-minute afternoon break for prayers and tea.

2) Training Expectations

Through the rounds, SPSS always ensured that the TPM participants’ expectations were “SMART” hence specific, measurable, and attainable. Thematic analysis revealed that most participants stated objectives that were classified as SMART objectives. As from Year 4 of MV&R, participants’ self-stated objectives were analyzed and classified according to

Module 1: Background to SPSS

Module 2: SPSS MV&R technical approach

Module 3: Key USAID Guidelines relevant to MV&R Activities

Module 4: SPSS Mobile Data Collection & Quality Control

Module 5: SPSS MV&R Tools & Protocols

Module 6: Ethical Principles & Practices for MV&R Activities

56

Bloom’s taxonomy5 (Knowledge, Understanding, Application, Analysis, Evaluation, and Creation). Examples included “To gain more experience about the tool and protocol reviews,” “To gain knowledge about MV&R and benefit from the knowledge and experienced of both IBTCI staff and other TPMs,” and “To learn the lessons learned from previous rounds and discover special areas that need greater attention during the next round of verifications.”

5 Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing, Abridged Edition. Boston, MA: Allyn and Bacon.

57

Figure 5: Bloom’s taxonomy used to classify participants’ self-stated expectations

3) Evaluation of the training

At the end of each training, participants were emailed a link to an evaluation survey and requested to complete it using their laptops or smartphones. The survey, developed in AskiaFace (a data management software), contained objective items (quantitative data) and open-ended items (qualitative data). Likert items were used to measure participants’ understanding of sessions presented as well as their understanding of SPSS tools and protocols, attitudes about how the sessions were conducted, and feedback on trainers’ clarity. The evaluation questions were as follows:

1. How well do you understand the purpose of Monitoring & Verification? 2. How well do you understand the SPSS approach for M&V? 3. How well do you understand the SPSS methodology for M&V? 4. How well do you understand the roles & responsibilities of Field Supervisors, monitoring firms,

and Field Monitors? 5. How well do you understand the organizational structure of SPSS for M&V? 6. How well do you understand the use of research ethics and guidelines for behavior when

conducting M&V activities? 7. How well do you understand the guidelines for observation on-site? 8. How well do you understand the guidelines for taking photographs? 9. How well do you understand the guidelines for taking GPS measurements? 10. How well do you understand the USAID Gender Policy and its application during SPSS M&V

activities? 11. How useful were the handouts provided to you? 12. How useful were the slide presentations to you?

Over and above the training evaluation, at the end of each training day, the team always had debrief sessions to discuss each team member’s feedback concerning the training days, the successes of each training day, and how to improve on any challenges faced on each training day.

58

Photos

Photo 1: Plenary session during the SPSS TPM Tool Induction workshop held at Aran

Guest House, Mogadishu on September 8, 2019.

Photo 2: Participants engage in role-playing during research sessions at the SPSS TPM Tool

Induction workshop held at Aran Guest House, Mogadishu on September 11, 2019.

59

IP Site Visit Pre-Deployment Meetings

During the first three years of MV&R, a growing challenge was observed in the relationship between Implementing Partners, SPSS, and Third-Party Monitoring Firms. There was frequently insufficient support or a lack of supporting documentation from IPs and their sub-grantees or subcontractors. In part this is because, while the Head Office staff of IPs knew about the mandate of the SPSS activity, their field teams did not. SPSS sought in Year 4 onwards to have the Nairobi-based MV&R team lead and physically attend monitors’ pre-deployment and IP pre-deployment workshops to foster a closer working relationship with IPs and their grantees. Attendance at these meetings introduced the Nairobi staff to the field-based IP staff and fostered a common understanding of the MV&R approach. As a result of this, there was improved coordination and collaboration, which led to increased efficiencies during data collection and reporting. Challenges with IPs and supporting documents significantly reduced from Year 4 onwards.

Data Collection

Data was collected through face-to-face interviews from MV&R Year 1 to MV&R VII R1. However, due to the COVID-19 pandemic outbreak in early 2020, SPSS changed its data collection methodology from face-to-face interviews to telephone interviews from MVR VII R2. This was informed by the recommendation from the World Health Organization (WHO) and the national government’s advice to maintain social distance to minimize the spread of the disease. However, some verifications such as construction, some benefit mapping sub-activities, supply of goods, and environmental monitoring sub-activities required physical observation and photographs as part of the data collection protocol. Monitoring partners were allowed to collect the observation data/photographs by visiting the sites while ensuring that their Field Monitors (FMs) observed the COVID-19 protection measures such as wearing face masks, hand washing/ hand sanitization, and maintaining adequate social distance.

Analysis and Reporting

SPSS analyzed and presented MV&R findings to USAID/Somalia in a manner that maximized its utility to COR/AORs and partners to improve results. The Consolidated Report presented all the MV&R findings for each round by each IP. Reports provided to USAID/Somalia included the following:

1. Summary reports for the site visit completed. The format of the report evolved over the course of MV&R and this was based on discussions between USAID/Somalia and SPSS. In Year 7 the format of the consolidated report changed into a simpler clearer one-pager per site visit. This one-pager at a glance was able to summarize the site visit as well as flag any issues arising that needed to be addressed or any responses received from IPS.

2. A tracker that provided status updates on the MV&R site universe (selected sites, sites verified, sites not verified and reasons for non-verification, non-verified sites to be considered in subsequent verifications).

3. PowerPoint Presentation for each round, as requested by USAID/Somalia. The format of the presentation was based on the discussions between USAID/Somalia and SPSS.

60

4. Detailed individual site visit reports, with photographs and GIS data for each site visit. Reports on findings that need USAID/Somalia action (for example those with red and orange flags), were provided to USAID/Somalia on a rolling basis.

5. An MV&R response tracking sheet for USAID/Somalia for site visit conducted, including flags, Action Points, and Learning Points.

Prior to consolidated report writing, SPSS does prepare individual site visit reports. This is a task that was handled by external report writing consultants. SPSS held induction sessions for Report Writers prior to the commencement of each round’s reporting. During this meeting, the Report Writers were taken through the analysis and reporting plans. The objectives of these sessions were to understand the components of MV&R, understand the generation of the M&V reports and any key changes from previous rounds, and strengthen Report Writers’ understanding and application of SPSS data quality control systems and MVR report writing quality standards in the box shown here. Each report writer was then

then expected to produce individual site verification reports as assigned to them. On a rolling basis, when SPSS finalized a critical mass of Verification Reports for a particular IP (generally a minimum of five reports), the MV&R team sent the reports to the IP to review. IPs were given five working days to provide their feedback, which was then incorporated verbatim into the Verification Reports. Based on the supplementary evidence provided by the IP, SPSS could then revise the flag allocated to an individual Verification Report, but documented the initial flag and subsequent flag for that report in the Dashboard Annex of the Consolidated report.

SPSS Flagging System

Below is a description of the analytical framework for the analysis of monitoring and verification data that was applied to each of the data collection tools designed by the SPSS MV&R team. The method describes the steps involved.

The first step begins with the selection of key questions for each of the data collection tools, followed by a response option to those key questions. Key questions are those listed in each tool that trigger attention based on the response. Key questions serve as predictors of emerging or small problems that could become more serious unless attended to. The selection of a response option trigger to more than one

key question indicates the likelihood that there is already a serious problem requiring immediate attention by the IP and USAID.

A revised flagging system was developed in Y3 to quickly communicate to USAID the status of a sub-activity based on the results of SPSS monitoring and verification. It relied more heavily on evidence-based data and on objective criteria for assigning a particular flag color. Scoring for both Yellow and Green Flags became automated into the data collection

Quality: The report should be based on facts, verifiable information, and valid proofs. It should be clear and easily understood, free from errors and duplication, well organized and structured, and adhere to ethical reporting style. Timeliness: The report should be prepared and submitted within a short span of time or the time stipulated. Information delayed is information denied. Accuracy of information should not suffer at the cost of achieving an objective of promptness. Consistency: A report should be prepared for many rounds from the same type of information and statistical data. SPSS used the same principles and concepts for data collection, classifying, tabulating, and presenting the information. . Diligence: Include only relevant and accurate data in the report. Go the extra mile to confirm information from other relevant documents. Content: A report should be precise, accurate, and specific. Ensure that the report is comprehensive, including the data and results. Provide a suitable title for each report according to the nature of contents.

61

system. Red, Yellow, and Orange-Flagged verifications have been reported immediately when encountered without wait until the report is consolidated or submitted. In Y4, this system was updated with questions from new tools and an expanded description of the green and red flag:

• A green flag indicates there are no issues or problems uncovered by SPSS during monitoring and verification and that the minimum implementation standards are met.

• A yellow flag indicates there is either an emerging problem or a small problem that should be addressed before it develops into a larger set of problems.

• An orange flag indicates that, although the activity/site was approved for verification, the actual activity did not happen during the SPSS data collection window. This is an indicator of delayed implementation.

• A red flag indicates a serious problem requiring immediate attention. These are problems that could conceivably hinder achievement of a result based on performance targets related to the sub-activity. If serious enough, it could derail the achievement of the activity’s purpose. The assignment of red flags should be rare. The red flag will be applied in instances when an activity is indicated as completed in Quarterly Reports, although it has actually not been done, or in cases of fraud. For OTI, the red flag is assigned for sites that SPSS teams cannot access.

Figure 6: SPSS Flagging System

RED FLAG ORANGE FLAG YELLOW FLAG GREEN FLAG

Serious problem identified at site visit, that requires immediate attention to resolve or No Activity Implementation

No verification done during data collection period

Emerging or small problem identified that should be addressed to prevent it becoming more significant (Issues for IP to Investigate)

No issues or problems identified, but valuable findings for program management / decision-making purposes (Minimum implementation standards met)

62

ANNEX 3: LIST OF EVENTS & MEETINGS

# Date Description of meeting/event Location

1 02/15/2015 USAID Somalia IP Meeting Zen Gardens, Kenya

2 02/13/2015 M&E Community of Practice Meeting SPSS Offices, Kenya

3 04/13/2015 One-day SYLI Workshop The Tribe Hotel, Kenya

4 04/15/2015 One-day Strategy Review Workshop The Tribe Hotel, Kenya

5 05/18/2015 Somalia Office Strategy Offsite Meeting to review draft strategy

6 06/02/2015 MV&R Year 1 - Presentation of findings from MV&R Rounds 1 & 2

The Tribe Hotel

7 06/03/2015 Communications Community of Practice Meeting

8 06/10/2015 USAID Somalia IP Meeting Zen Gardens, Kenya

9 08/10-13/2015 MV&R Year 1 - Presentation of Round 3 findings to PEG, SYLI, and TIS

10 09/14/2015 PEG Evaluation – Presentation of Preliminary Findings to USAID/Somalia, representatives from the State Department, DFID and the Danish Embassy.

The Tribe Hotel, Nairobi

11 09/15/2015 PEG Evaluation – Presentation of preliminary findings to the Ministry of Agriculture and Ministry of Livestock

Aran Guest-House, Mogadishu

12 10/6-7/2015 2015 DQA Data Analysis Workshop 13 12/01/2015 SSG Baseline - Presentation of Phase 1 findings

(December 1, 2015) Kempinski Hotel, Kenya

14 12/02/2015 MV&R Year 2 - Presentation of Round 1 findings to USAID/Somalia Tech Teams and IPs

Kempinski Hotel, Kenya

15 12/03/2015 USAID Somalia IP Meeting Kempinski Hotel, Kenya 16 03/02-03/2016 MV&R Year 2 - Presentation of Round 2 reports and

findings to USAID & IPs The Tribe Hotel, Kenya

17 03/17/2016 TIS Evaluation - Presentations of preliminary findings and conclusions to USAID

USAID Offices, Kenya

18 03/23/2016 TIS Evaluation - Presentations of preliminary findings and conclusions to key stakeholders


19 03/30/2016 SSG Baseline - Presentation of preliminary findings and conclusions of Phase 1 and Phase 2 to USAID and key stakeholders


20 05/19/2016 PEG Evaluation - Presentation of preliminary findings and conclusions of the PEG evaluation to key stakeholders

21 05/30/2016 MV&R Year 2 - Presentation of R3 findings to USAID and IPs

22 08/02/2016 Somalia Perception Survey - presentation of preliminary findings and conclusions to USAID and key stakeholders


23 08/09/2016 MV&R Year 2 - Presentation of R4 findings to USAID and IP

24 08/29-31/2016 2016 DQA Data Analysis Workshop 25 09/7-8/2017 SSG Baseline - Presentation of preliminary findings

and conclusions of Phase 2 to USAID and key stakeholders

26 02/13/2017 MV&R Year 3 - R1 Findings Presentation Lord Erroll, Kenya

63

# Date Description of meeting/event Location

27 02/14/2017 USAID/Somalia Implementing Partner Meeting Windsor Hotel, Nairobi 28 03/30/2017 MV&R Year 3 - Round 2 Presentations of findings UNICEF Offices, Nairobi 29 05/17-18/2017 USAID Somalia Mission retreat Zen Gardens, Nairobi 30 05/31/2017 TIS+ Presentation of preliminary findings from

baseline to USAID and stakeholders USAID Offices, Nairobi

31 06/20/2017 USAID Somalia Program Office Retreat Windsor Hotel, Nairobi 32 08/02/2017 MV&R Year 3 - Presentation of findings for Round 3 USAID Offices, Nairobi 33 09/2017 Training of TIS+-AECOM and ABE-UNICEF on

M&E concepts SPSS Offices, Nairobi

34 11/16/2017 Economic Growth APS Resilience Learning Workshop

Lord Erroll, Nairobi

35 02/6-14/2018 Data Validation Workshop with USAID Somalia and IPs

USAID Offices, Nairobi

36 03/23/2018 Gender Assessment 2017 - Presentation of findings and conclusions

USAID Offices, Nairobi

37 05/05/2018 MV&R Year 3: Presentation of findings from Round 1 and 2

38 05/16/2018 Review of the USAID/Somalia PMP Lord Erroll, Nairobi 39 07/24/2018 TIS+ Assessment - Wave 1 Data Collection Outbrief USAID Offices, Nairobi 40 09/25/2018 MV&R Year 3 – Presentation of findings from Round

3 & 4

41 10/24-25/2018 ABE Learning Event

Windsor Hotel, Nairobi

42 10/29/2018 TIS+ Phase 3 Presentation of Wave 1 Preliminary Findings

Trademark Hotel, Nairobi

43 11/20/2018 USAID Somalia - Implementing Partner’s Meeting

Windsor Hotel, Nairobi

44 11/29-30/2018 USAID Somalia – Staff Retreat Windsor Hotel, Nairobi 45 01/09/2019 TIS+ Phase 3 Out-brief with USAID USAID Offices 46 03/11-12/2019 Two-day USAID Somalia – Education Event Lord Erroll, Nairobi 47 03/20/2019 TIS+ Presentation of preliminary findings from Wave

2 USAID Offices

48 04/24-25/2019 MV&R Year 5 – presentation to USAID/SFO of findings from Round 1 and 2

49 08/06/2019 MV&R Year 5 - presentation to USAID/SFO of findings from Round 3

Trademark Hotel

50 12/06/2019 MV&R Year 5 - presentation to USAID/Somalia of findings from Round 4

51 05/11-15/2020 MV&R Year 7 - Round One Virtual presentations to USAID

Virtual – Google Meet

52 MV&R Year 7- Round Two Virtual presentations to USAID


53 10/5/2020 Gender Assessment 2020 - Inception Report Presentation


54 11/09/2020 MV&R Year 7 - Round 3 Virtual presentation to USAID Somalia


55 11/19/2020 Gender Assessment 2020 – Data Collection Out brief Virtual – Google Meet 56 12/01-03/2020 DQA 2020 – Data Validation Sessions Virtual – Google Meet 57 12/10/2020 Citizen Consultation – Series 1 Analysis Workshop Virtual – Google Meet 58 02/2/2021 Gender Assessment 2020 - Dissemination Workshop Virtual – Google Meet 59 03/31/2021 Citizen Consultation – Series 2 Analysis Workshop Virtual – Google Meet

64

ANNEX 4: SPSS INDICATOR PERFORMANCE TRACKING TABLE

Objectives/Results and Intermediate Results

Key Performance Indicators (KPIs) FY 2019 FY 2020 FY 2021

Objective: To improve USAID/Somalia activity effectiveness through the provision of a suite of appropriate, high-quality and timely performance management services

1- Overall performance of USAID/Somalia programs portfolio

54.76% TBD TBD

Sub-Objective 1: Provide USAID/Somalia with tailored monitoring, verification, and reporting, and performance management support services

2- Percentage of USAID/ Somalia Technical Officers who reported using SPSS MV&R information in decision-making processes

62.50% TBD TBD

IR-1.1: Timely and Useful Monitoring and Verification (MV&R) feedback provided

3- Percentage of satisfactory or above ratings across the Contractor Performance Assessment Reporting System (CPARS) standard categories (MV&R)

100% TBD TBD

Output 1.1.1: High-quality approaches and tools are used to perform timely monitoring and verification of USAID/ Somalia activities

4- Percentage of planned M&VR site visits completed as per TOs’ requirements

99.56% 100% 98%

Output 1.1.2: USAID/ Somalia is provided on a quarterly basis with high-quality MV&R data and meaningful feedback on its program portfolio

5- Percentage of MV&R deliverables submitted and accepted on time

100% 100% 100%

Output 1.1.3: High-quality engagement to ensure effective IPs feedback and monitoring of MV&R recommendations

6- Percentage of SPSS MV&R recommendations effectively implemented by IPs following MV&R presentations

74.19% 64.81% 97.80%

IR-1.2: High-Quality Performance Management Services Provided

7- Percentage of satisfactory or above ratings across the Contractor Performance Assessment Reporting System (CPARS) standard categories (Core Services)

100% TBD TBD

8- Percentage of satisfactory or above ratings across the Contractor Performance Assessment Reporting System (CPARS) standard categories (DQAs)

N/A N/A TBD

Output 1.2.1: Successful learning events and workshops are organized to support USAID/Somalia performance management

9- Number of learning events and workshops facilitated by SPSS that meet USAID standards

1

Output 1.2.2: Rigorous and unbiased on-demand DQAs are implemented to ensure

10- Percent of DQA recommendations implemented by IPs

93.33% N/A TBD

65



USAID/Somalia uses high-quality data

Output 1.2.3: Meaningful feedback is provided to USAID/Somalia through high-quality indicators and performance analyses

11- Percent of indicators fully verified and submitted on time

84.31% 100% 100%

12- Number of activity performance analysis performed and approved by USAID

25 22 10

Output 1.2.4: High-quality on-demand performance management support services are provided to USAID/ Somalia

13- Number of evaluations, assessments, and studies SOW developed or reviewed by SPSS team

1 1 0

14- Number of strategy documents, M&E plans including Mission PMP, IPs AMELPs, and other technical M&E documents developed or reviewed by SPSS

4 7 2

15- Number of USAID staff, IPs staff, and other USAID partners staff who received M&E capacity-building from SPSS

9 37 7

Sub-Objective 2: Provide USAID/Somalia with meaningful feedback through rigorous and unbiased evaluations and ad hoc studies

16- Percentage of USAID/ Somalia Technical Officers who reported using SPSS Evaluation feedback in decision-making processes

100% TBD TBD

IR-2.1: High Quality Evaluations and Assessments and other studies provided

17- Percentage of satisfactory or above ratings across the Contractor Performance Assessment Reporting System (CPARS) standard categories (Evaluations and other studies Task orders)

100% TBD TBD

Output 2.1.1: Robust and unbiased methods and techniques used to prepare and carry out high-quality evaluations of USAID/ Somalia programs

18- Percentage of Evaluation deliverables submitted and accepted on time

100% 100% 100%

Output 2.1.2: Rigorous methods and techniques used to prepare and carry out high-quality ad hoc studies and assessments on USAID/ Somalia programs

19- Number of assessments and other studies deliverables submitted and accepted on time

66.70% 100% 100%

66



Output 2.1.3: Useful feedback from evaluations, assessments, and studies is provided to USAID/Somalia and its partners through learning sessions and workshops

20- Number of learning events, workshops, or other meetings organized to share findings and conclusions from evaluations, assessments, and other studies

4 3 4

67

ANNEX 5: PHOTOS FROM SITE VISIT MONITORING

GOOGLE DRIVE LINK: PHOTOS DIVIDED BY IPS

https://drive.google.com/drive/folders/1Vwwf3T117b-Kcb3iwJ7NB54DM4oKHjsB?usp=sharing

68

ANNEX 6: SUMMARY OF USAID PROJECTS PERFORMANCE DATA6

On target Below target Above target Within 10% either side of the target 11% or more below the target 11% or more over the target

1. Summary of ABE performance data Life of activity: September 30, 2015–September 29, 2020

Update end of Q2-FY 2021

Indicator #

Results Framework Indicators

Achievement to Date Gaps and issues on target

setting, data collection, and reporting

ABE GOAL: Alternative Basic Education provides a viable alternative to formal school for pastoralist and other out-of-school children to complete a cycle of basic education.

1

Percent of learners who demonstrate reading fluency and comprehension of grade- level text at the end of grade 2 with USG assistance (ES.1)

• Baseline: 0% (2015) • Target by end of project: 75% • Cumulative achievement: 82% • Percent achievement: 109% (on

target)

The Early Grade Reading Assessment (EGRA) study was delayed. Data was only available at the end of the project. USAID did not have the opportunity to assess quality of this indicator.

Activity Purpose 1: Pastoralist and other out-of-school children have access to quality Alternative Basic Education.

2

Number of learners in primary schools or equivalent non-school- based settings reached with USG assistance (ES 1-3)

• Baseline: 0 (2015) • Target by end of project: 21,000 • Cumulative achievement:

20,248 • Percent achievement: 96.44%

(on target)

Availability of source documents was a key issue that was resolved by the IPs and support from the MV&R site visits.

3

Number of primary or secondary classrooms built or repaired with USG assistance (ES 1-14)

• Baseline: 0 (2015) • Target by end of project: 192 • Cumulative achievement: 194 • Percent achievement: 101% (on

target)

4 Number of gender-sensitive sanitation

• Baseline: 0 (2015) • Target by end of project: 192

6 For achievement over/under 11% of the target, review the latest IPTT for more details (deviation narratives, achievement for specific FYs, etc.).

69

Indicator #




facilities provided by ABE

• Cumulative achievement: 182 • Percent achievement: 95% (on

target)

5

Number of reusable sanitary towels provided to qualifying ABE adolescent girls


14,944 • Percent achievement: 75%

(below target)

6

Number of primary or secondary textbooks and other teaching and learning materials (TLM) provided with USG assistance (ES-1.10)


48,334 • Percent achievement: 98.75%

(on target)

Availability of source documents was a key issue that was resolved by the IPs and support from the MV&R site visits. Involvement of local IPs in the procurement and distribution processes greatly improved the achievement of results.

Activity Purpose 2: ABE Classrooms provide quality education to pastoralist and other out-of-school children.

Number of education administrators and officials who complete professional development activities with USG assistance (ES1-12)

• Baseline: 0 (2015) • Target by end of project: 296 • Cumulative achievement: 675 • Percent achievement: 228%

(above target)

Alignment of PIRS definition with the standard definition and clarification of the project approaches were required to improve data collection.

7

Number of educators who complete professional development activities with USG assistance (ES 1-6)


(below target)

Alignment of PIRS definition with the standard definition and clarification of the project approaches were required to improve data collection.

8 Number of teachers trained on IRI

• Baseline: 0 (2015) • Target by end of project: 148 • Cumulative achievement: 0 • Percent achievement: 0% (below

target)

9

Number of Parent–Teacher Associations (PTAs) or community governance


target)

70

Indicator #




structures engaged in primary or secondary education supported with USG assistance (ES 1-13)

10 Number of Child-to-Child (CtC) Clubs established


target)

Activity Purpose 3: State and regional authorities adopt policies and practices that promote quality Alternative Basic Education as a viable alternative to formal school for pastoralist and other out-of-school children to complete a cycle of basic education.

Number of consultations held on NFE draft policy


(below target)

11

Number of joint monitoring visits to ABE centers with government officials


(below target)

2. Summary of BUILD performance data Life of activity: March 31, 2016–March 30, 2021 (TBC)


Indicator #

Results Framework Indicators Achievement to Date

Gaps and issues on target setting, data

collection, and reporting

BUILD Goal: Increase participation and trust in political processes among Somali citizens.

1

% of respondents reporting that the electoral process was fair

• Baseline (Somaliland): N/A • Target by end of project

(Somaliland): 71% • Cumulative achievement

(Somaliland): 76% • Percent achievement: 107%

(on target)

This indicator was measured only in Somali-land, because elections did not take place in FGS. However, the indicator was disaggregated into the registration process and elections. BUILD did not update on the registration process in FGS.

71

Indicator #




2

% change in respondents’ participation (as voters, poll agents, and observers) in electoral and political processes

• Baseline (Somaliland): N/A • Target by end of project

(Somaliland): 10% • Cumulative achievement

(Somaliland): 7% • Percent achievement: 70%

(below target)

This indicator was measured only in Somaliland because elections did not take place in FGS.

Objective 1: Citizens’ understanding of political and electoral processes increased.

3

Number of individuals receiving voter education through USG-assisted programs (DR 3.2-4)

• Baseline: 0 (2016) • Target by end of project:

455,666 • Cumulative achievement:


(above target)

4a

# of individuals receiving civic education through USG-assisted programs (DR 3.2-5)


1,700,000 • Cumulative achievement:

12,934,001 • Percent achievement: 761%

(above target)

The overall achievement (761%) was inflated by counting the number of “Likes” in the social medias for BUILD electoral messages shared. It was recommended to clarify “Civic education” in the indicator definition to avoid validity issues.

4

Percentage of respondents reporting increased knowledge of electoral and political processes (voter registration)

• Baseline: N/A • Target by end of project:

25% • Cumulative achievement:

12% • Percent achievement: 48%

(below target)

• Sub-objective 1.1: Civil society organizations’ capacity to inform citizens and monitor political and electoral processes increased.

5

Percent of local CSOs receiving USG assistance that show improved capacity to promote political participation and voter education

• Baseline: 0% (2016) • Target by end of project:



(on target)

The Institutional Capacity Assessment (ICA) tool was initially applied. However, there were no clear criteria established to assess improvement of grantees’ capacity to promote political participation. Dropped in FY 2019 due to the complexity of measuring improvement in capacity of CSOs replaced by two better F

72

Indicator #



collection, and reporting indicators: current indicator 7 (advocacy CSOs) and 8 (media CSOs).

6a

Number of USG-supported activities designed to promote or strengthen the civic participation of women (DR.4-1)


(on target)

7a

Number of CSOs receiving USG assistance engaged in advocacy interventions (DR.4.2-2)


(on target)

16

Percent of USG-assisted organizations with improved performance




(above target)

Sub-objective 1.2 (Output): Media capacity to cover and disseminate information on electoral and political processes increased.

6

Number of USG-supported media outlets with improved content quality on elections-related topics


(below target)

Dropped in FY 2019 and replaced by a similar F indicator: # of USG-assisted media-sector CSOs and/or institutions that serve to strengthen the independent media or journalists (DR.5.3-2).

8a

Number of USG-assisted media-sector CSOs and/or institutions that serve to strengthen the independent media or journalists (DR.5.3-2)


(on target)

Objective 2: Capacity of election management bodies (EMBs) and other relevant government bodies to design and manage transparent, credible elections, and political processes increased.

9

Number of election officials trained with USG assistance [National Electoral Commission (NEC) in Somaliland and National Independent Electoral Commission (NIEC) in EFS]




(on target)

Sensitivity of high-ranked official signing the sign-in sheet with their details for security reasons influenced the quality of this indicator at certain point. It was recommended to adopt a more commensurate

73

Indicator #



collection, and reporting approach to evidence participation of high-ranked officials in USAID activities to avoid data issue while ensuring data protection and security of participants.

10a

Number of laws, regulations, and electoral administrative procedures and systems strengthened with USG support


(above target)

17 Number of women and marginalized trained as election monitors


(below target)

8

Extent to which key electoral stakeholders believe EMB is engaging them

• Baseline: TBD • Target by end of project:



(on target)

Dropped due to political sensitivity and objections from the National Independent Electoral Commission (NIEC, a Somalia EMB) that asking this undermines them. (The M&E Manager received a threat and left the project as a result of trying to organize focus group discussions.)

Objective 3: Political parties are more competitive, inclusive, and representative of citizens' interests.

11

Number of USG-assisted political parties implementing initiatives to increase the number of candidates and/or members who are women, youth, and from marginalized groups


(above target)

There was a deviation between what was measured (number of political parties participating in training designed to support initiatives that promote women’s participation) and what the indicator intended to measure/monitor according to the PIRS definition. It was recommended to revise the measurement approach, data collection methods, and tools.

74

Indicator #



collection, and reporting For more details, see DQA 2020 report.

12

Number of clear and transparent processes and procedures developed by political parties


(on target)

13

Number of BUILD-supported women and youth who attain a formal role or position in their party or on campaign teams


(above target)

14

Number of issue-based platforms and/or policy documents that address citizens’ (women, youth, marginalized groups) concerns developed by USG-assisted political parties (Somaliland)


(above target)

15a

Number of individuals who receive USG-assisted political party training (DR 3.3-1)




(above target)

18

Number of women candidates and campaign staff who receive USG-assisted training


(below target)

19

Percentage of women candidates who received USG-assisted training who get elected and/or appointed to a political office


TBD • Cumulative achievement:


(below target)

75

3. Summary of Damal performance data Life of activity: September 16, 2019–May 15, 2021


Indicator #




Program Goal: Strengthen the Federal Parliament of Somalia’s ability to fulfill its legislative and oversight responsibilities, particularly regarding financial governance and oversight, as well as increase public participation in the public financial management process.

1

Number of media broadcasts or posts to extend MP outreach to public provided with USG assistance

• Baseline: 0 (2019) • Target by end of project: 12 • Cumulative achievement:

25 • Percent achievement: 208%

(above target)

2

Percent of USG-assisted organizations with improved performance (CBLD-9)




(below target)

No data were reported under this indicator with the exception of baseline qualitative information on Upper and Lower houses performance. The indicator is new, and the implementation of some related activities was delayed due to circumstances linked to the COVID-19 pandemic.

Expected Result 1: Inclusive legislative process promoted

3

Number of consensus-building forums (multi-party, civil/ security sector, and/or civil/ political) held with USG assistance (DR.3.1-3)



(above target)

Expected Result 2: Financial oversight practices improved

4

Number of laws, policies, or procedures drafted, proposed, subject to substantive amendment, or adopted for public financial management or audit


(above target)

There was need to provide more details on how data is generated through quarterly reviews (DQA 2020 report).

5 Number of MPs and staff trained in oversight duties (oral/written


DQA 2020 identified Reliability and Integrity issues (for further

76

Indicator #




questioning and interpellation techniques) with USG assistance

• Cumulative achievement: 58

• Percent achievement: 77% (below target)

details, see DQA 2020 report).

6

Number of executive oversight actions taken by the legislature receiving USG assistance (DR.2.1.1.)



(above target)

Expected Result 3: Institutional independence and interdependence improved

7

Number of draft bills subject to substantive amendment or final vote in legislatures receiving USG assistance


(above target)

8

Proportion of benchmarks for legislative processes achieved by targeted committees




(on target)

Benchmarks are very clearly defined. There is a need to define a more consistent way of setting target (for further details, see DQA 2020 report).

Expected Result 4: Capacity of MPs and staff on key committees strengthened

9

Number of parliamentary committee staff trained to draft legislative amendments



(on target)

DQA 2020 identified Reliability and Integrity issues (for further details, see DQA 2020 report).

10

Number of MPs, staff, and fellows provided with technical assistance (committee report writing, research and analysis, logistical planning, etc.)



(on target)

While data was entered manually from participant lists to the reports submitted to USAID, following data verification, no discrepancies were noted (for further details, see DQA 2020 report).

77

4. Summary of EAJ performance data

Life of activity: 2018–2023 (TBC)


Indicator #



collection, and reporting EAJ PROGRAM GOAL: Lasting improvements in access to justice and effective mechanisms to address grievances for stability in Somalia.

1G

% of citizens reporting improvement on access to justice assessment elements (HRSM Indicator 03-R2-02)


40.4% • Cumulative achievement:


(below target)

2G

% of beneficiaries (citizens) experiencing improved awareness of justice options


TBD • Cumulative achievement:


(above target)

3G

# of commitments made by justice actors to broaden access to justice for women and marginalized population


(below target)

4G

# of justice institutions that modified their practices to be more inclusive


(below target)

A data issue was identified. For details, see EAJ DQA 2020 report.

5G

# of procedures, policies or laws proposed, drafted, or adopted to promote improved practices in access to justice


(below target)

Objective 1: Support and improve inclusive community engagement in justice solutions.

1.1.OPT

# of events held to directly address barriers to access to justice and stability in Somalia


(below target)

1.2.OPT # of local women participating in a substantive role or

• Baseline: 0 (2019) • Target by end of project: 90 • Cumulative achievement: 23

78

Indicator #



collection, and reporting position in a justice and or peacebuilding process supported with USG assistance


Objective 2: Strengthen justice services

2.1.OPT # of justice legal service providers trained




(below target)

2.2.OTC

# of initiatives created through USG funding dedicated to strengthening justice services


(below target)

2.3.OTC

# of access to justice actors who have expanded services and or locations where they offer services


(on target)

There might be a possibility of undercounting: EAJ counts only partners who expand locations or services when the expansion is directly through EAJ activities; however, partners expanding services or location as a result of EAJ indirect support are not counted.

Objective 3: Improve navigation of justice pathways by aggrieved parties.

3.1.OTC # of clients assisted




(below target)

3.2.OTC % of clients satisfied with legal aid services




(below target)

A data issue was identified. For details, see EAJ DQA 2020 report.

3.3.OTC

% of USG-assisted organizations with improved performance (OPI)


78%

79

Indicator #



collection, and reporting • Cumulative achievement:

66,7% • Percent achievement: 85%

(below target)

OPT 3.2.1

Network strength of targeted access to justice networks (Edited: Network strength of targeted CSOs)




(below target)

Administrative/cross-cutting indicators

4.1.OPT # of events held under EAJ



(below target)

4.2.OPT # of people reached through EAJ




(below target)

5. Summary of GEEL performance data

Life of activity: September 29, 2015–July 31, 2021 (TBC)


Indicator #



collection, and reporting GEEL Goal: Promoting inclusive economic growth in Somalia by stimulating production, employment, and enterprise development.

1

Number of microenterprises supported by USG enterprise assistance (EG.5-3)




(above target)

A data issue was identified. For details, see DQA 2020 report

7

Number of promotional or competitiveness events organized or supported


(above target)

80

Indicator #




16 Number of individuals directly benefiting from the project




(on target)

IR1: Business enabling environment strengthened through access to finance and support to policy and regulation

6

Number of visits organized for Somali government officials in other parts of Somalia or within the region


(above target)

10

Value of new private capital sector investment in the agriculture sector or food chain leveraged by the project




(above target)

When setting targets for PPR indicators in the AMELP revision process, GEEL must seek confirmation from the COR about what target has been set in the mission PPR report.

19

Number of laws, policies, regulations, or standards to enhance energy sector governance formally proposed, adopted, or implemented as supported by USG (EG.7.3-1)


(below target)

26

Total number of clients benefiting from financial services provided through USG-assisted financial intermediaries, including non-financial institutions or actors (EG.4.2-1)


(below target)

A data issue was identified. For details, see DQA 2020 report.

IR2: Enterprise development promoted through business development services

2

Number of private sector firms that have improved management practices as a result of USG assistance (EG.5.2-2)




(above target)

A data issue was identified. For details, see DQA 2020 report

81

Indicator #




12 Number of trainings organized by project



(above target)

13

Number of vocational training programs in target population centers


(on target)

17

Number of food security private enterprises (for-profit), producer’s organizations, water user’s associations, women’s groups, trade and business associations, and community-based organizations (CBOs) receiving USG assistance (EG 3.2-4)



(below target)

20

Number of firms receiving USG-funded technical assistance for improving business performance (EG 5.2-1)


(above target)

21bis

Percent of USG-assisted organizations with improved performance (CBLD-9)



69.8% • Percent achievement: 140%

(above target)

23

Percent of individual with improved soft skills following participation in USG-assisted workforce development program (EG 6-13)




(above target)

IR3: Improved production, employment, and incomes in select VCs (agriculture, fisheries, and renewable energy)

3 Increase in production of sector/value chain commodities targeted


50%

Baseline values are dynamic and depend on the number of number of firms receiving support in

82

Indicator #



collection, and reporting by the project by direct project beneficiaries

• Cumulative achievement: 36%


the current FY based on USAID priority value chains and the task order’s objectives. This is not a one-time baseline setting. That should be factored in IPTTs and other performance data platforms.

4

Value of targeted agricultural commodities exported with USG assistance (EG.3.2-23) - USD

• Baseline: 12,828,540 (2015) • Target by end of project:



(above target)

9

Number of new or more efficient technologies introduced by GEEL that will lead to increase in production in a sector value chain (EG.3.2-7)


(on target)

11

Number of full-time equivalent (FTE) jobs created with USG assistance (EG.3-9)




(on target)

14

Number of individuals who have received USG-supported short-term agricultural sector productivity training (EG.3.2-1)




(above target)

15 Number of sustainable civil infrastructure works completed


(on target)

18

Number of beneficiaries with improved energy services due to USG assistance (EG.7.1-1)


(below target)

83

Indicator #




21

Number of people trained in technical energy fields supported by USG assistance (EG.7.3-2)



(above target)

22

Number of natural resource management/biodiversity conservation techniques promoted in the target sectors as a result of USG assistance (EG.10.2-3)


(below target)

22bis

Percent of individual with new employment following participation in USG-assisted workforce development program (EG 6-12)




(above target)

24

Percent of individual who complete USG-assisted workforce development program (EG 6-14)




(below target)

IR4: Greater participation by women and youth in the economy as entrepreneurs, employers, and employees

5

Percentage of female participants in USG-assisted programs designed to increase access to productive economic resources (assets, credit, income or employment) (GNDR-2)




(below target)

8

Proportion of female participation in promotional and competitiveness events




(on target)

6. Summary of TIS+ performance data

84

Life of activity: May 2015–July 2021 (TBC)


Indicator #




TIS+ Goal: Somalia’s stability increased through participatory processes that promote good governance and community cohesion.

G1

Percent of citizens surveyed in targeted local government areas reporting increased confidence in local government officials

• Baseline: 90% (2017) • Target by end of project: 97% • Cumulative achievement: 86% • Percent achievement: 89%

(below target)

As per TIS+ AMELP PIRS section, these indicators are collected through the TIS+ Assessment implemented by SPSS. The project had difficulties in updating the indicators because the Assessment Report did not include a table that shows the indicator values. The evaluation team was requested to analyze the dataset and provide values for the indicators.

G2

Percent of citizens surveyed in targeted local government areas reporting increased participation in decision-making


target)

G3

Percent of citizens surveyed in targeted local government areas reporting improved experience with personal security


(below target)

Objective 1: Increase confidence in government based on equitable participation in decision-making and management of community assets.

1.1

Number of USG-assisted consensus-building processes resulting in agreement (DR.3.3-1)


(on target)

Sub-objective 1.1: Improve governance processes supporting communities (Cross-cuts with Objective 2).

1.1.1

Number of consensus building forums (multi-party, civil/security sector, and/or civil/political) held with USG Assistance (DR.3.1-3)


(above target)

Sub-objective 1.2: Improve effectiveness of shared assets management.

1.2.1 Proportion of supported infrastructure facilities

• Baseline: 0% (2015) • Target by end of project: 100% • Cumulative achievement: 87%

DQA 2018 identified validity, reliability, and precision gaps which

85

Indicator #




being used for the intended purpose


TIS+ resolved in FY 2019 and FY 2020 by setting up a log book for each infrastructure facility and training field monitors. However, data issues were identified in FY 2020. For details, see DQA 2020 report.

1.2.2

Proportion of supported civil infrastructure projects maintained as per TIS+ operations and maintenance manual


(below target)

Sub-objective 1.3: Improve delivery of basic services.

1.3.1

Kilometers of roads improved or constructed as a result of USG assistance (EG.3.1-1)

• Baseline: 0 (2015) • Target by end of project: 56.9 • Cumulative achievement: 40 • Percent achievement: 70%

(below target)

1.3.2

Number of civil infrastructure projects completed with USG funds


(above target)

1.3.3

Average % change in number of weekly flights at airstrips receiving USG assistance


(below target)

Objective 2: Empower community and government representatives to engage with the private sector and development actors in a collaborative process for community growth.

2.1

Number of successful financial or in-kind partnerships developed with private sector or development partners towards achieving medium- or long-term community growth


(on target)

Sub-objective 2.1: Increase leadership, design, and implementation of development activities by Somali stakeholders.

2.1.1

Number of mechanisms for external oversight of public resource use supported by USG activities (DR 2.4-2)


(below target)

Sub-objective 2.2: Improve citizen confidence in governance skills of Somali officials (cross-cuts with Objective 1)

2.2.1 Number of joint monitoring visits and


86

Indicator #




assessments done collaboratively with government officials

• Cumulative achievement: 152 • Percent achievement: 137%

(above target)

Objective 3: Increase Somali engagement in creating a more stable future.

3.1 Community cohesion increased


target)

Sub-objective 3.1: Increase community leadership of and participation in social reconciliation, peace-building, peace messaging, and stability activities.

3.1.1

Number of new groups or initiatives created through USG funding dedicated to resolving the conflict or the drivers of the conflict (PS.6.2-1)


target)

3.1.2

Number of people participating in USG-supported events, trainings, or activities designed to build mass support for peace and reconciliation (PS.6.2-4)




(below target)

A data issue was identified. For details, see DQA 2020 report.

Sub-objective 3.2: Increase identification with shared Somali values

3.2.1

Number of USG supported events, trainings, or activities designed to build support for peace or reconciliation among key actors to the conflict (PS.6.2-3)


(below target)

Objective 4: Support inclusive, sustainable development by reducing gender gaps in stabilization and development.

4.1

Percent of citizens surveyed in targeted local government areas reporting increased women participation in stabilization and development


(above target)

•

Sub-objective 4.1: Increase women’s leadership in peacebuilding and governance.

87

Indicator #




4.1.1

Number of local women participating in a substantive role or position in a peacebuilding process supported with USG assistance (GNDR 10)



(on target)

•

Sub-objective 4.2: Increase equitable access to and control over assets across genders.

4.2.1 Proportion of women in oversight committees


target)

•

88

ANNEX 7: SPSS FACT SHEET

89

ANNEX 8: MV&R GEO-TAGGED SITES

GOOGLE DRIVE LINK: MV&R GEO TAGGED SITES

https://drive.google.com/file/d/1Vtu_frrop6cs_7VaorjvSX2ykTdu_GIR/view?usp=sharing

90

ANNEX 9: EVALUATION DATASETS

GOOGLE DRIVE LINK: EVALUATION DATA SETS

https://drive.google.com/drive/folders/1XdvffoFlJ_rbN_AZRmmMGBLrsPuEgVYm?usp=sharing

91

ANNEX 10: POWERPOINT OVERVIEW OF LIFETIME OF SPSS

SPSS OVERVIEW

93

CORE SERVICES

94

MONITORING, VERIFICATION AND REPORTING (MV&R)

97

DATA QUALITY ASSESSMENT (DQAs)

100

EVALUATIONS AND ASSESSMENTS

105

ANNEX 11: LIST OF DELIVERABLES

The period of Performance for SPSS was from August 1, 2014 to December 31st, 2019. SPSS received an extension of the original IDIQ for December 31st, 2019 to June 30th, 2021.

Attached are all the deliverables under the IDIQ (#IDIQ AID-623-I-14-00009):

Task Order#1 Task Order/ Contract Number: #AID-623-TO-14-00008

Contract Section: F.5. CONTRACT DATA REQUIREMENTS LIST

Name: TASK ORDER 01

Period of Performance: August 1, 2014 – July 31, 2019. Second Modification: Till June 30, 2021.

# Deliverable Date Due

Date Submitted

Date Approved or invoiced

Core Services and Operational Support

Annual Work Plan - Year 1 Within 30 days after award 09/12/2014 N/A

Annual Work Plan - Year 2 October 1, 2015 10/01/2015 N/A Annual Work Plan - Year 3 October 1, 2016 10/01/2016 N/A Annual Work Plan - Year 4 October 1, 2017 10/01/2016 N/A Annual Work Plan - Year 5 October 1, 2018 10/01/2016 N/A Annual Work Plan - Year 6 October 1, 2019 10/01/2016 N/A Annual Work Plan - Year 6 October 1, 2020 10/01/2016 N/A

Quarterly Progress Report Q4 FY14 October 31, 2014

10/30/14 (Quarterly) N/A

Quarterly Progress Report Q1 FY15 January 31, 2015 01/31/2015 N/A

Quarterly Progress Report Q2 FY15 April 30, 2015 04/30/2015 N/A Quarterly Progress Report Q3 FY15 July 31, 2015 07/31/2015 N/A

Quarterly Progress Report Q4 FY15 October 31, 2015 10/31/2015 N/A

106

















Quarterly Progress Report Q2 FY21 April 30, 2021 04/27/2021 N/A

Quarterly Progress Report Q3 FY21 July 31, 2021 Will be included in the final report

N/A

Weekly Progress Reporting Submitted weekly N/A N/A

Exit Plan 90 days prior to the contract’s completion

03/26/2021 05/27/2021

Final Contract Completion Report Contract end date 05/28/2021 06/21/2021

Quarterly Financial Report Q4 FY14

30 days following the end of the quarter

Submitted N/A

Quarterly Financial Report Q1 FY15 October 31, 2014 Submitted N/A

107

Quarterly Financial Report Q2 FY15 January 31, 2015 Submitted N/A

Quarterly Financial Report Q3 FY15 April 30, 2015 Submitted N/A Quarterly Financial Report Q4 FY15 July 31, 2015 Submitted N/A








QuarterlyFinancial Report Q2 FY18 January 31, 2018 Submitted N/A







Quarterly Financial Report Q3 FY20 April 30, 2020 Submitted N//A Quarterly Financial Report Q4 FY20 July 31, 2020 Submitted N/A



Quarterly Financial Report Q3 FY21 April 30, 2021 Submitted N/A Task Order #2 Task Order/ Contract Number: #AID-623-TO-14-00010

Contract Section: F.4. CONTRACT DATA REQUIREMENTS LIST

Name: TASK ORDER #2

Period of Performance:

One year from the date of the award, for the reasons stated in Section C.1.1 of this TO. (Date of Award: 08/11/2014)

108


Date Submitted


Monitoring, Verification and Reporting

1 Performance monitoring data collection/survey protocols refined

December 9, 2014 12/09/2014 01/28/2015

2 LIst of 20 target activity sites selected for January 6, 2015-February 10,2015 January 6, 2015 01/05/2015 01/28/2015

3 Training materials for field supervisors and TPM field monitors developed January 9, 2015 01/12/2015 01/28/2015

4 Monitoring & verification reports provided to USAID for January 6, 2015-February 10,2015

February 24, 2015 02/24/2015 05/20/2015

5

Monitoring & verification reports provided to USAID for Somali regional governments for sites visited during January 6, 2015-February 10,2015

February 24, 2015 02/24/2015 05/20/2015

6 Performance monitoring data collection/survey protocols reviewed and updated

February 1, 2015 02/02/2015 07/21/2015

7 List of 150 target activity sites selected for February 11, 2015 - May 10, 2015

February 1, 2015 02/02/2015 07/21/2015

8

Monitoring & verification reports provided to USAID for Somali regional governments for sites visited during February 11, 2015 - May 10, 201

May 25, 2015 05/26/2015 07/21/2015

9 Monitoring & verification reports provided to USAID for February 11, 2015 - May 10, 201

May 25, 2015 05/26/2015 07/21/2015

10 Performance monitoring data collection/survey protocols reviewed and updated

May 1, 2015 05/04/2015 09/16/2015

11 List of 190 target activity sites selected for May 11, 2015 - August 10, 2015 May 1, 2015 05/01/2015 09/16/2015

12 Monitoring & verification reports provided to USAID for May 11, 2015 - August 10, 2015

August 10, 2015 08/10/2015 09/16/2015

13

Monitoring & verification reports provided to USAID for Somali regional governments for sites visited during May 11, 2015 - August 10, 2015

August 10, 2015 08/10/2015 09/16/2015

14 Train USAID staff to conduct DQAs on USAID’s behalf, if needed

TBD (once during the TO period of performance)

N/A N/A

Task Order #3 Task Order/ Contract Number: #AID-623-T0-14-00020

109

Contract Section: F.4 DELIVERABLES Name: COMMUNICATION SERVICES

Period of Performance: One year from the date of the award, but may be re-issued annually. (Date of Award: 09/26/2014)


Date Submitted

Date Approved

or Invoiced Outreach Information

System to produce and written guidance to constantly update all products below in place Q2

Quarter 2 03/23/2015 12/11/2015

System to produce and written guidance to constantly update all products below in place Q4

Updated based on experience in Q4

12/11/2015

Fact sheets for all current activities updated Q1 Quarter 1 11/072014 12/11/2015

Fact sheets for all current activities updated Q2 Quarter 2 03/16/2015 12/11/2015



Sectoral fact sheets created/updated Q2 Quarter 2 03/24/2015 12/11/2015 Sectoral fact sheets created/updated Q4 Quarter 4 09/22/2015 12/11/2015

Cross-cutting fact sheets created/updated Q1 Quarter 1 12/29/2014 12/11/2015

Cross-cutting fact sheets created/updated Q3 Quarter 3 06/26/2015 12/11/2015

Regional profiles created or updated Q1 Quarter 1 12/29/2014 12/11/2015 Regional profiles created or updated Q3 Quarter 3 06/26/2015 12/11/2015

C-COP meetings facilitated with IP outreach staff to share best practices, build capacity and collect story leads Q1 - 1-3 TBD by USAID

Quarter 1 December, 2014 12/11/2015


Quarter 2 March, 2015 12/11/2015


Quarter 3 4/15/15 12/11/2015


Quarter 4 9/10/15 12/11/2015

3 Transforming lives/success stories produced Q1 Quarter 1 12/11/2015

110

3 Transforming lives/success stories produced Q2 Quarter 2 02/19/2015

&03/20/ 2015 12/11/2015

3 Transforming lives/success stories produced Q3 Quarter 3 06/25/2015 12/11/2015

3 Transforming lives/success stories produced Q4 Quarter 4 09/25/2015 12/11/2015

Speeches, talking points, or other briefing materials Q1 TBD 12/11/2015




Content for social media posts (24) Q1 Quarter 1 12/29/2014 12/11/2015

Content for social media posts (24) Q2 Quarter 2 03/18/2015 & 3/19/2015 12/11/2015

Content for social media posts (24) Q3 Quarter 3 06/26/2015 12/11/2015 Content for social media posts (24) Q4 Quarter 4 09/25/2015 12/11/2015 Photos for the products above Q1 Quarter 1 12/29/2014 12/11/2015 Photos for the products above Q2 Quarter 2 03/24/2015 12/11/2015 Photos for the products above Q3 Quarter 3 06/26/2015 12/11/2015 Photos for the products above Q4 Quarter 4 09/25/2015 12/11/2015 2 videos for the website and other use Q2 Quarter 2 03/16/2015 12/11/2015 2 videos for the website and other use Q4 Quarter 4 09/25/2015 12/11/2015 Task Order #4 Task Order/ Contract Number: #AID-623-TO-14-00022 Contract Section: F.4 DELIVERABLES Name: Clearinghouse

Period of Performance: Nine months from the date of the award. (Date of Award: 09/26/2014)


Date Submitted

Date Approved

or Invoiced Clearinghouse

Identify the most cost-effective FEDRAMP-certified Cloud Service Provider to host the clearing house

Within 30 days of task order award

09/5/2014 09/26/2014

Assume full operation of the Somalia Clearinghouse in current hosting environment


10/26/2014 06/30/2015

Transfer Clearinghouse to new hosting environment

Within 30 days of approval from USAID

10/26/2014 06/30/2015

Maintain full operation of the Somalia Clearinghouse


10/01/14 06/30/2015

111

Timely provision of standard reporting data from the Clearinghouse to USAID and/or IPs

By November 3, 2014 for the annual performance report, and upon request, thereafter.

11/02/2014 06/30/2015

Provision of periodic training for new USAID and/or IP staff on use of the CH

As needed, no more than quarterly.

02/04/2015 - Mercy Corps 02/10/2015 (TIS Staff) 02/10/2015 (DAI) 2/10/15 DAI COP DCOP) 02/26/2015 USAID STAFF 03/25/2015 USAID Staff

06/30/2015

Administrative support for users of the system provided

As needed, on-going. N/A 06/30/2015

Task Order #5 Task Order/ Contract Number: #AID-623-TO-14-00018 Contract Section: F.4 DELIVERABLES Name: Support for Somalia Stabilization Strategy

Period of Performance: Nine months after the date of award. (Date of Award: 09/22/2014)


Date Submitted

Date Approved

or Invoiced Strategy Support

1 SWOT workshop with USAID (USG Offsite Meeting) conducted by contractor. 11/12/2014 11/12/2014 04/18/2015

2 Summaries (powerpoint slides) of all documents in Annex 3. 12/06/2014 12/06/2014 04/18/2015

3 Stakeholder consultations conducted in Nairobi.

12/08/2014-12/19/2014

12/08/2014-12/19/2014 09/16/2015

4

New Senior Strategy Specialist reads Summary of all docs in Annex 3, report from SWOT, Nairobi consultation Notes, and the full documents listed in Annex 4 (attached). SSS also meets with USAID staff for orientation, to have any questions answered, and conducts 2-3 consultations in Nairobi.

01/26/2015-02/13/2015

01/28/2015-02/13/2015 09/16/2015

5 Stakeholder consultations conducted in Somalia.

02/15/2015-02/28/2015

02/15/2015-02/28/2015 09/16/2015

6 First Expert Seminar held at USAID 02/13/2015 03/06/2015 09/16/2015

112

7 Second Expert Seminar held at USAID 03/06/2015 03/06/2015 09/16/2015 8 Third Expert Seminar held at USAID 03/13/2015 03/20/2015 09/16/2015 9 Fourth Expert Seminar held at USAID 03/20/2015 03/27/2015 09/16/2015

10 Summaries of consultations and draft RF diagram, to be presented and used in RF workshop, shared with USAID

03/06/2015 03/06/2015 09/16/2015

Contractor works with USAID/PPL Officer to plan RF workshop

03/09/2015- 3/11/2015 N/A N/A

11 RF Workshop conducted by contractor 03/11/2015-03/12/2015

03/09/2015-03/18/2015 06/08/2015

12 Contractor provides a brief summary of workshop highlights and decisions to USAID.

03/30/2015 03/30/2015 09/25/2015

13

SSS submits a 2-4 page word document on background, development hypothesis and results framework, along with a powerpoint presentation of the results framework for use during the Washington DVC.

03/30/2015 03/30/15 09/25/2015

14

SSS participates in the Washington DVC presentation of the results framework. On or before April 8, as determined by W/AFR front office.

04/08/2015 04/08/2015 09/25/2015

15 SSS submits strategy (first draft) to USAID. On or about April 22 (or two weeks after the DVC)

04/22/2015 04/20/2015 09/25/2015

16 Somalia office Strategy offsite meeting to review draft strategy with SSS participation facilitated by IBTCI.

04/30/2015 04/29/2015 09/25/2015

17 SSS submits a second draft strategy. 04/29/2015 04/29/2015 09/25/2015

18 SSS submits a third draft strategy to USAID. 06/22/2015 06/22/2015 07/28/2015

Task Order #6 Task Order/ Contract Number: #AID-623-TO-14-00014 Contract Section: F.3 DELIVERABLES Name: DATA QUALITY ASSESSMENT

Period of Performance: 2 months from the effective date of the award. (Date of Award: 09/23/2014)


Date Submitted


Data Quality Assessment

1 Training materials and tools for use during site visits 09/25/2014 09/25/2014 12/09/2014

2 Training Workshop 09/30/2014 09/30/2014 09/30/2014

3 Data analysis workshop/meeting with DQA teams 10/16/2014 09/30/2014 09/30/2014

4 Draft DQA Report 10/30/2014 10/30/2/14 12/09/2014

113

5 Final DQA Report 11/16/2014 11/16/2014 12/08/2014 Task Order #7 Task Order/ Contract Number: #AID-623-TO-15-00001 Contract Section: F.4 Deliverables Name: SYLI Baseline

Period of Performance: 6 months from the effective date of the award. (Date of Award: 05/04/2015)


Date Submitted

Date Approved

or Invoiced Data Quality Assessment

1

Initial meeting between the SYLI COR, Mercy Corps, USAID/EA/Somalia M&E Specialist, other relevant USAID personnel, and contractor evaluation team key personnel. Meeting will clarify roles and responsibilities, logistical issues, and timelines.

No later than two days after team arrives in Kenya

05/12/2015 05/12/2015

2

A draft work plan, which provides a projected timeline and describes in detail the final data collection methods (including draft interview questions and data collection tools) to be used

No later than 5 days after #1. 05/04/2015 11/05/2015

3 Meeting with USAID and partner to refine and finalize the work plan and tools

Before field work begins 05/12/2015 05/12/2015

4 In-brief with education ministry Before field work begins 05/12/2015 05/12/2015

5 Data collection

Contractor to propose based on scheduling limitations highlighted in this statement of objectives

11/05/2015

6

Updates on progress to date, in person or via email as agreed at the initial meeting, including any issues or problems encountered

weekly Weekly N/A

7 Out-brief with education ministry Before departing the field

08/02/2015 08/02/2015

8 Presentation of Preliminary Findings and Conclusions of the evaluation to USAID and key stakeholders.

No later than 12 days after completion of fieldwork.

08/12/2015 11/05/2015

9 Draft Evaluation Report for review. No later than 12 days after #8. 09/18/2015 11/05/2015

114

10 USAID and implementing Partner Comments on Draft Report due to contractor

No later than 12 days after #9. 10/15/2015 11/05/2015

11 Final Evaluation Report submitted to USAID along with all raw data, as stated above

No later than 7 days after receiving comments from USAID.

11/03/2015 11/05/2015

12 USAID Comments on or approval of Final Report due to contractor

No later than 10 calendar days after #11

10/15/2015 11/05/2015

Task Order #8 Task Order/ Contract Number: #AID-623-TO-15-00006 Contract Section: F.4 DELIVERABLES

Name: PARTNERSHIP FOR ECONOMIC GROWTH (PEG) EVALUATION

Period of Performance: July 16, 2015 to August 16, 2016


Date Submitted

Date Approved

or Invoiced PEG Evaluation

1 Literature review Prior to team arriving in Kenya

N/A 09/06/2016

2

Initial meeting between the PEG COR, USAID/KEA/Somalia M&E Specialist, other relevant USAID personnel, and contractor evaluation team key personnel to clarify roles and responsibilities, logistical issues, and timelines.

July 22, 2015 07/22/2015 07/22/2015

3

Draft work plan, which provides a projected timeline and describes in detail the final data collection methods (including draft interview questions and data collection tools) to be used.

July 24, 2015 07/24/2015 09/06/2016

4 Meeting with USAID and partners to refine and finalize the work plan and tools. July 27, 2015 07/27/2015 08/05/2015

5 In-brief with Ministry of Agriculture and Benadir University. August 2, 2015 08/02/2015 08/02/2015

6 Data collection August 11 - 31, 2015

08/11/2015 -08/31/2015

7

Updates on progress during data collection, in person or via email as agreed at the initial meeting, including any issues or problems encountered.

Weekly Weekly 09/06/2016

8 Out-brief with Ministry of Agriculture September 15, 2015 09/15/2015 08/15/2015

115

9 Presentation of preliminary findings and conclusions of the evaluation to key stakeholders, if requested by USAID

September 14, 2015 09/14/2015 09/14/2015

10 Draft Evaluation Report submitted to USAID for review.

September 23, 2015 09/23/2015 09/06/2016

11 USAID and DAI Comments on Draft report due to contractor

October 05, 2015 10/05/2015 09/06/2016

12 Pilot-test report and finalize tool/approval of final tool March 29, 2016 01/31/2016 03/29/2016

13 Presentation of preliminary findings and conclusions of the evaluation to USAID and DAI Regional Director

May19, 2016 05/19/2016 05/19/2016

14 Presentation of preliminary findings and conclusions of the evaluation to other donors, partners, and GEEL

May 20, 2016 05/20/2016 07/01/2016

15 Draft Evaluation Report submitted to USAID for review. June 28, 2016 07/27/2016 09/06/2016

16

Final evaluation Report submitted to USAID along with cleaned datasets and an info-graphic based on the evaluation report submitted to USAID, as well as raw data.

July 20, 2016 07/20/2016 12/06/2016

17 Fact sheet on data collection and quality lessons learned during this evaluation to share with stakeholders

July 20, 2016 07/20/2016 12/06/2016

Task Order #9

Task Order/ Contract Number: # AID-623-TO-15-00007 Contract Section: F.4 DELIVERABLES

Name: MONITORING, VERIFICATION & REPORTING SERVICES Year Two

Period of Performance: Beginning August 11, 2015 and ending August 10, 2016.


Date Submitted

Date Approved

or Invoiced MONITORING, VERIFICATION & REPORTING

1 Finalize Indicator Tracker Format with USAID Aug. 28, 2015 08/26/2015 09/04/2015

2

Updated Indicator spreadsheet, disaggregated by PPR and non-PPR indicators, and Tracker to include 3rd quarter FY 2015

Sept 09, 2015 09/09/2015 09/15/2015

3

Updated Indicator spreadsheet, disaggregated by PPR and non-PPR indicators, and Tracker to include 4th quarter FY 2015

Nov 06, 2015 11/06/2015 09/15/2015

116

4

Updated Indicator spreadsheet, disaggregated by PPR and non-PPR indicators, and tracker to include the 1st quarter FY 2016.

Feb 29, 2016 02/29/2016 09/15/2015

5

Updated Indicator spreadsheet, disaggregated by PPR and non-PPR indicators, and Tracker to include 2nd quarter FY 2016

May 29, 2016 05/29/2016 06/30/2016

6 Modified tracker to include data needed for AIDT+ and DIS

Within 60 days of when template is provided by USAID

11/11/2015 09/15/2015

7 Indicator Spreadsheet and ad hoc reports recurrent Multiple submissions 12/14/2015

8 Performance monitoring data collection tools reviewed and updated Aug 30, 2015 08/30/2015 09/15/2015

9 List of activity sites for M&V Round 1 (Sept - Nov 2015) Aug 30, 2015 08/30/2015 09/15/2015

10 Training materials for field supervisors and TPMs developed Aug 30, 2015 08/30/2015 12/16/2015

11 MV&R Reports - provided to USAID for Round 1 Nov 15, 2015 11/16/2015 12/16/2015

12 MV&R Reports for Somali regional governments provided to USAID for Round 1

March 7, 2016 11/16/2015 12/16/2015

13 Performance monitoring data collection tools reviewed and updated Nov 30, 2015 11/30/2015 12/11/2015

14 Powerpoint presentation to USAID of findings from MV&R visits Round 1 Nov 30, 2015 11/30/2015 12/11/2015

15 List of activity sites for M&V Round 2 (December 2015 - February 2016) Nov 30, 2015 11/30/2015 12/11/2015

16

MV&R Reports, including a summary table of all flags, Actions and Learning Points from that Round, provided to USAID for Round 2

Feb 15, 2016 02/15/2016 03/30/2016


Feb 15, 2016 02/16/2016 03/30/2016

18 Performance monitoring data collection tools reviewed and updated Feb 29, 2016 02/25/2016 03/14/2016

19 Powerpoint presentation to USAID of findings from MV&R visits Round 2 Feb 29, 2016 02/29/2016 03/14/2016

20 List of activity sites for M&V Round 3 (March - May 2016) Feb 29, 2016 03/07/2016 03/14/2016

21

MV&R Reports, including a summary table of all flags, Action and Learning Points from that round,- provided to USAID for Round 3

May 15, 2016 05/16/2016 06/21/2016

117


May 15, 2016 05/16/2016 06/21/2016

23 Consolidate into a tracker all IP responses detailed in their respective 2nd Qtr Reports to flags, Action and Learning points.

May 16, 2016 (or 15 days after receipt of IP Reports)

05/16/2016 06/21/2016

24 Performance monitoring data collection tools reviewed and updated May 30, 2016 06/07/2016 06/21/2016

25 Powerpoint presentation to USAID of findings from MV&R visits Round 3 May 30, 2016 05/25/2016 06/21/2016

26 List of activity sites for M&V Round 4 (June - Aug 2016) May 30, 2016 05/30/2016 06/21/2016

27 MV&R Reports - provided to USAID for Round 4 Aug 9, 2016 08/09/2016 08/31/2016


Aug 9, 2016 08/09/2016 08/31/2016

29 Powerpoint presentation to USAID of findings from MV&R visits Round 4 Aug 26, 2016 08/09/2016 08/31/2016

Task Order #10 Task Order/ Contract Number: #AID-623-TO-15-00008 Contract Section: F.3 Deliverables Name: 2015 Data Quality Assessment Period of Performance: August 24- November 30, 2015.


Date Submitted

Date Approved

or Invoiced Data Quality Assessment 1 Training materials 09/03/2015 09/01/2015 09/03/2015 2 Tools for use during site visits 09/03/2015 09/01/2015 09/03/2015 3 Training workshop 09/09/2015 09/09/2015 09/09/2015

4 Data analysis workshop for two DQA teams

10/06/2017- 10/07/2015

10/06/2015- 10/07/2015 10/072015

5 Validation Workshops with IPs 10/09/2015 10/08/2015 10/08/2015 6 Draft DQA Report 10/23/2015 10/23/2015 11/20/2015 7 Final DQA Report 11/16/2015 11/30/2015 12/02/2015 Task Order #11 Task Order/ Contract Number: #AID-623-TO-15-00014 Contract Section: F.4 DELIVERABLES Name: TIS Evaluation Period of Performance: September 30, 2015 to April 29, 2016.


Date Submitted

Date Approved

or Invoiced Evaluation Services

118

1

A framework for categorization of grants by sub-activity and any other sub-categorizations, as appropriate (including the coding system) to be used to answer key evaluation questions

10/19/2015 11/09/2015 12/08/2015

2

A draft Work Plan which provides a projected timeline and describes in detail the final data collection methods (including draft interview questions and data collection tools) to be used

11/16/2015 11/09/2015 12/10/2015

3 Report on the pilot test, and all revised instruments 12/04/2015 11/30/2015 06/10/2016

4 Presentations (2) of preliminary findings and conclusions of the evaluation to (1) USAID and (2) key stakeholders

03/17/2016 & 03/23/2016 03/23/2016 06/10/2016

5 Draft Evaluation Report for USAID review and feedback 04/07/2016 04/07/2016 06/06/2016

6

Draft Infographic/One-page summary, if requested by USAID for USAID review and feedback (the format and content of this deliverable to be finalized in consultation with USAID)

04/13/2016 04/13/2016 06/6/2016

7

Final Evaluation Report & Infographic/One paper submitted to USAID along with all source data, as well as Draft Summary Documents

05/9/2016 05/09/2016 06/06/2016

8 Synthesis of final evaluation report, maximum of three different versions

NTL 5 days after receiving USAID approval of Final Evaluation Report

06/05/2016 06/06/2016

9 Submission of final report on the DEC and data on Open Data

NTL 5 days after receiving USAID approval of Final Evaluation Report

06/07/2016 06/08/2016

Task Order #12 Task Order/ Contract Number: #AID-623-TO-15-00015 Contract Section: F.4 DELIVERABLES Name: Baseline System Development for SSG Period of Performance: September 30, 2015 to October 31, 2016.


Date Submitted

Date Approved

or invoiced

119

baseline system development services

1

A draft Work Plan for Phase 1, with final data collection and analysis methods to be used for Phase I and a timeline for both Phase 1 and Phase 2.

11/12/2015 11/10/2015 01/20/2016

2 Presentation of Phase 1 findings 12/01/2015 01/19/2016 01/20/2016

3 Submission of Tool Kits developed in Phase 1 02/11/2016 02/11/2016 05/18/2016

4 Submission of Phase I Report 01/19/2016 01/19/2016 02/25/2016

5

A draft work plan for Phase 2, a Somalia Perception Survey (SPS), with a detailed timeline and the final data collection methods

02/22/2016 01/06/2016 05/11/2016

6 Pilot Test Report, along with revised tools 06/22/2016 06/22/2016 07/07/2016

7 Presentation of preliminary findings and conclusions of Phase 2 to USAID 09/07/2016 09/07/2016 09/07/2016

8 Draft SPS Report (comprised of one report for Somalia and one report for Somaliland) for USAID review and feedback

12/14/2016 10/10/2016 1/20/2015

9

Final SPS Report (comprised of one report for Somalia and one report for Somaliland) and Draft Synthesis Documents (1) two-page; and (2) one four-to-six page synthesis document, submitted to USAID

7 Business days after USAID comments on draft

01/13/2017 03/03/2017

10

Submission of final SPS Synthesis Reports and posting of Final SPS Report and data - the final Synthesis Reports must be submitted to USAID; the final Baseline Report must be posted on the DEC; and the raw quantitative data and codebook must be submitted on the USAID Open Data website. SPS synthesis documents: (1) a two-page and (2) a four-to-six page synthesis document.

01/15/2017 01/13/2017 03/03/2017

Task Order #13 Task Order/ Contract Number: #AID-623-TO-15-00013 Contract Section: F.4 Tasks and Deliverables

Name: Design, Production and Distribution Services of USAID 2016 Calendars

Period of Performance: September 29, 2015 to December 10, 2015.


Date Submitted

Date Approved

Calendar 1a Shortlist photographs (electronic version) 10/06/2015 10/0920/15 01/13/2016

1b First draft of English text (electronic version) 10/15/2015 10/15/2015 10/22/2015

1c Draft calendar design and layout (printed version) 11/06/2015 11/06/2015 01/14/2016

120

2

USAID selected photographs and second draft of English text embedded in the USAID approved calendar design and layout (printed version)

11/23/2015 11/23/2015 12/02/2015

3 Final English text and draft Somali text (electronic version) 12/10/2015 12/10/2015 12/11/2015

4 Final English and Somali text and photograph placement in USAID approved calendar layout (printed version)

12/14/2015 12/14/2015 12/22/2015

5 Proof of delivery of calendars to sites requested by USAID 01/13/2016

01/04/2016(Hargeisa) 01/07/2016 (Mogadishu) 01/08/2016 (Garowe)

01/13/2016

Task Order #14 Task Order/ Contract Number: # AID-623-TO-16-00005 Contract Section: F.4 DELIVERABLES Name: 2016 DATA QUALITY ASSESSMENT Period of Performance: June 30, 2016 to November 22, 2016.


Date Submitted

Date Approved

or Invoiced Data Quality Assessment

1 Training materials and tools for use during site visits 07/12/2016 07/12/2016 07/19/2016

2 Training workshop 07/19/2016 07/19/2016 07/19/2016

3 Phase 3 Report: Based on meetings with each DQA team 07/27/2016 07/27/2016 07/27/2016

4 Phase 4&5 Report: based on Site Visits (at least two per indicator) 09/02/2016 09/02/2016 10/07/2016

5 Data analysis workshop/meeting with DQA teams

08/29/2016 - 08/31/2016

08/29/2016 0-8/31/2016 10/07/2016

6 Five-day data validation workshop with USAID and IPs

09/26/2016 - 09/30/2016

09/26/2016 - 09/30/2016 10/07/2016

7 Draft DQA Report 10/28/2016 10/28/2016 11/10/2016 8 Final DQA Report 11/18/2016 11/18/2016 12/13/2016 Task Order #15 Task Order/ Contract Number: #AID-623-T0-16-00015 Contract Section: F.4 DELIVERABLES

Name: TIS+ BASELINE & END-LINE ASSESSMENT CLUSTER ONE

Period of Performance: September 30, 2016 to July 25, 2019.


Date Submitted

Date Approved

or Invoiced TIS+

121

Phase 1

1

Draft cluster-level toolkits for each of the five selected clusters, including both qualitative and quantitative tools for baseline and end-line data collection within each cluster and cluster-level data

By January 2017 01/31/2017 02/06/2017

2

Draft work plan, which provides a projected timeline for the duration of the task order and describes in detail the final data collection methods to be used and communities covered by the evaluation

By January 2017 01/31/2017 02/06/2017

3 Final work-plan and cluster-level toolkits By February 2017 02/28/2017 03/06/2017

Phase 2 - Baseline Data Collection

4 Report on the pilot test and any revised data collection instruments By March 2017 03/31/2017 04/19/2017

5 Presentation of (2) preliminary findings from baseline to USAID and stakeholders By May 2017 04/10/2017 10/13/2017

6 Draft evaluation report submitted to USAID covering baseline findings for all five clusters

By July 15,2017 07/17/2017 10/13/2017

7 Final baseline report submitted to USAID along with all raw data

September 1, 2017 09/01/2017 09/23/2017

8 Approved final baseline report submitted to the DEC with all raw data

By September 2017 09/28/2017 10/13/2017

9 Evaluation Synthesis document submitted to USAID

By September 2017 09/28/2017 10/13/2017

Phase 3- Endline Evaluation Wave 1

10 Updated draft end-line tools incorporating revisions emerging from baseline lessons learned for all five clusters

No later than December 2017 12/24/2017 12/16/2017

11 Conduct in-brief meeting with TO COR (SPSS and TIS+ Evaluation Team) and AECOM

By February 2018 02/28/2018 04/20/2018

12 Updated end-line tools incorporating revisions emerging from baseline lessons learned for all five clusters

April 15, 2018 05/01/2018 05/02/2018

13 Pilot-test report and final revised data collection instruments and data analysis plan

By May 2018 05/30/2018 06/06/2018

14 Conduct out-brief with USAID By August 2018 07/24/2018 07/25/2018

15 Presentation of preliminary findings from Wave 1 cluster-level end-lines to USAID

By October 2018 10/29/2018 10/30/2018

16 Draft end-line reports for each Wave 1 cluster submitted to USAID for review

No later than 15 days after Wave 1 findings presentation

11/26/2018 12/21/2018

122

17 Final end-line reports for each Wave 1 cluster submitted to USAID along with all raw data

No later than 15 days after receiving comments from USAID

01/7/2019 01/30/2019

Phase 3- Endline Evaluation Wave 2

18 Conduct out-brief with USAID By December 2018 01/07/2019 1/11/2019

19 Presentation of preliminary findings from Wave 2 cluster-level end-lines to USAID

No later than March 2019 01/07/2019 03/21/2019

20 Draft end-line reports for each Wave 2 cluster submitted to USAID for review

No later than April 2019 04/30/2019 05/20/2019

21 Final end-line reports for each Wave 2 cluster submitted to USAID along with all raw data

No later than 15 days after receiving comments from USAID

05/31/2019 06/13/2019

22 Updated analysis plan incorporating lessons learned from cluster-level evaluations submitted to USAID for review

No later than 15 days following submission of final reports on last cluster and no later than May 15, 2019

05/15/2019 05/15/2019

23 Presentation of preliminary findings from synthetic, comparative analysis across clusters

No later than 30 days following completion of the updated analysis plan

06/17/2019 06/19/2019

24 Draft report of synthetic, comparative analysis across clusters submitted to USAID for review

No later than 30 days following presentation of preliminary findings and no later than July 1, 2019

07/01/2019 07/01/2019

25 Final end-line evaluation report submitted to USAID along with all raw data

No later than August 12, 2019

07/15/2019 8/13/2019

26 Approved final end-line evaluation report submitted to the DEC along with all raw data

No later than August 19, 2019

08/26/2019 9/12/2019

Task Order #16 Task Order/ Contract Number: # AID-623-TO-16-00006 Contract Section: F.5 DELIVERABLES

Name: MONITORING, VERIFICATION & REPORTING SERVICES III

Period of Performance: August 16, 2016 to August 15, 2017.

123


Date Submitted

Date Approved

or Invoiced MONITORING, VERIFICATION & REPORTING SERVICES III

1

Provide an individual detailed briefing to each new IP (GEEL, TIS+, ABE and BUILD) and their AOR/CORs on SPSS MV&R process and reporting requirements of this Task Order, and their role in that process.

Not later than November 30, 2016

11/30/2016 11/7/2016

2

Updated Indicator Performance Tracking Table (IPTT) spreadsheet, disaggregated by PPR and non-PPR indicators, and Tracker to include Q3FY2016

Within 15 days of receipt of all IP data for Q3FY2016

11/30/2016 01/25/2017

3 Synthesis and Analysis Report (SAR) of all SPSS M&V findings for TIS-DAI

September 30, 2016 09/30/2016 01/25/2017

4 Performance monitoring data collection tools review and updated for Round 1

September 9, 2016 09/09/2016 01/25/2017

5 List of activity sites for M&V Round 1 (September-November 2016)

September 9, 2016 09/09/2016 01/25/2017

6 Training materials for Field Supervisors and TPMs developed

September 9, 2016 09/09/2016 01/25/2017

7

MV&R Reports, including a summary table of all flags, Action and Learning Points from that Round, provided to USAID for Round 1

November 25, 2016 11/25/2016 01/25/2017

8 MV&R Reports for Somali Regional governments provided to USAID for Round 1

November 25, 2016 11/25/2016 01/25/2017

9 Powerpoint presentation to USAID of findings from MV&R Round 1 visits

November 29, 2016 11/29/2016 01/25/2017

10

Updated Indicator Performance Tracking Table (IPTT) spreadsheet disaggregated by PPR and non-PPR indicators, and Tracker to include Q4 FY 2016

Within 15 days of receipt of all IP data for Q4 FY 2016

12/15/2016 02/22/2017

11 Performance monitoring data collection tools reviewed and updated for round 2

December 6, 2016 12/06/2016 12/19/2016

12 List of activity sites for M&V Round 2 (December 2016 - February 2017)

December 6, 2016 12/06/2016 12/16/2016

13 Synthesis and Analysis report (SAR) of all SPSS M&V findings for SYLI-Mercy Corps

By the end of the SYLI activity if no final evaluation, or by two weeks before an evaluation team deploys if one is done

05/31/2017 07/19/2017

124

14


Within 15 days of all IP data for Q1FY2017

03/03/2017 04/11/2017

15


March 24, 2017 03/24/2017 04/11/2017


March 24, 2017 03/24/2017 04/11/2017

17 Powerpoint presentation to USAID of findings from MV&R Round 1 visits March 29, 2017 03/27/2017 03/29/2017

18 Performance monitoring data collection tools reviewed and updated for Round 3 April 7, 2017 04/05/2017 04/06/2017

19 List of activity sites for M&V Round 3 (March - May 2017) April 7, 2017 04/05/2017 04/06/2017

20


Within 15 days of receipt of all IP data for Q2FY2016

08/17/2017 09/25/2017

21a

Consolidate into a tracker all IP responses to flags, Action and Learning Points detailed in their Q3FY2016 quarterly reports

Within 15 days of receipt of all IP quarterly reports for Q3FY2016

08/29/2017 09/07/2017`

21b



03/03/2017 03/28/2017

21c



03/03/2017 03/28/2017

21d



08/11/2017 09/06/2017

22


July 21, 2017 07/21/2017 09/12/2017


July 21, 2017 07/21/2017 09/06/2017

24 Powerpoint presentation to USAID of findings from MV&R Round 1 visits July 27, 2017 07/26/2017 08/02/2017

125

25 Modify IPTT to include data needed for AIDT+ and DIS

Within 60 days of when template is provided by USAID

09/08/2017

26 Ad hoc reports Upon request by USAID N/A 09/08/2017

27

Consolidated report of process learning for the full year, including best practices established, challenges of working within a dynamic environment and mitigation plans against those challenges

August 1, 2017 08/01/17 09/06/2017

Task Order #17 Task Order/ Contract Number: TASK ORDER # AID-623-TO-17-00008 Contract Section: F.5 DELIVERABLES

Name: MONITORING, VERIFICATION & REPORTING SERVICES (MV&R) IV

Period of Performance: October 1, 2017 to September 30, 2018.


Date Submitted

Date Approved

or Invoiced MONITORING, VERIFICATION & REPORTING SERVICES (MV&R))

1

Provide an individual detailed briefing to IOM (OTI) and any new IP and their AOR/CORs on SPSS MV&R processes including reporting requirements and their role in that process.

For IOM, within one month of this Task Order being awarded and for new awards, within three months of the activity award.

10/10/2017 &10/23/2017 12/13/2017

2

Google calendar with milestone dates relevant to AOR/CORs

Within one month of this Task Order being awarded-

10/13/2017 10/13/2017

3

Updated Indicator Performance Tracking Table (IPTT) spreadsheet, disaggregated by PPR and non-PPR indicators, and Tracker to include 3rd Quarter FY 2017

Within 15 days of receipt of all IP data for Q3 FY 2017 quarterly reports-

10/26/2017 02/15/2018

126

4

IPTT summary for 3rd Quarter FY 2017 PowerPoint highlighting performance against targets, disaggregated by PPR and all indicators

Within 15 days of receipt of all IP data for Q3 FY 2017 quarterly reports

11/03/2017 02/15/2018

5

Consolidate into a tracker all IP responses to flags, Action and Learning Points as detailed in their respective Q3 FY 2017 Reports.

TBD by Contractor-

11/03/2017 11/06/2017

6 SAR of all SPSS MV&R findings for all closing activities.

Two months before activity closure.-

07/31/2018 10/12/2018

7 Performance monitoring data collection tools reviewed and updated for Round 1 (Q1 2018)

TBD by Contractor.- 10/19/2017 11/02/2017

8

Development and submission of Data Analysis Plan.

At least one month before round 1 data collection begins

10/31/2017 11/02/2017

9 List of activity sites for MV&R Round 1 (Q1 2018)

TBD by Contractor 10/23/2017 11/6/2017

10 Training materials for field supervisors and TPMs developed


11

MV&R Reports, including a summary table of all flags, Action and Learning Points from that Round, and MV&R site universe tracker as an annex provided to USAID/SFO for Round 1 (Q1 2018)

TBD by Contractor

01/18/2018 02/26/2018

12 MV&R Reports for Somali regional governments provided to USAID/SFO for Round 1 (Q1 2018)


13

Updated IPTT spreadsheet, disaggregated by PPR and non-PPR indicators, and Tracker to include Q4 FY 2017


02/09/2018 03/09/2018

127

14

IPTT summary for Q4 FY 2017 PowerPoint highlighting performance against targets, disaggregated by PPR and all indicators

Within 15 days of receipt of all IP data for Q4 FY 2017quarterly reports

02/09/2018 03/09/2018

15

Consolidate into a tracker all IP responses to flags, Action and Learning Points, as detailed in their respective Q4 FY 2017 Reports.

TBD by Contractor

02/09/2018 03/09/2018

16 Performance monitoring data collection tools reviewed and updated for Round 2 (Q2 FY 2018)


17

Data Analysis Plans for Round 2 At least one month before round 2 data collection begins

01/30/2018 06/13/2018

18 List of activity sites for MV&R Round 2 (Q2 FY 2018)


19

MV&R Reports, including a summary table of all flags, Action and Learning Points from that Round and MV&R site universe tracker as an annex provided to USAID/SFO for Round 2 (Q2 FY 2018)

TBD by Contractor

04/19/2018 06/04/2018

20 MV&R Reports for Somali regional governments provided to USAID/SFO for Round 2 (Q2 FY 2018)


21 Power-point presentation to USAID/SFO of findings from MV&R visits Round 1 & 2 (Q1 & Q2 FY 2018)


22

Updated IPTT spreadsheet, disaggregated by PPR and non-PPR indicators, and Tracker to include Q1 FY 2018


04/03/2018 04/017/2018

128

23

IPTT summary for Q1 FY 2018 PowerPoint highlighting performance against targets, disaggregated by PPR and all indicators


04/03/2018 04/17/2018

24

Consolidate into a tracker all IP responses to flags, Action and Learning Points, as detailed in their respective Q1 FY2018 reports

TBD by Contractor

04/03/2018 04/17/2018

25 Performance monitoring data collection tools reviewed and updated for Round 3


26

One day workshop presenting the tools, protocols and flagging.


04/16/2018 04/19/2018

27 List of activity sites for MV&R Round 3 (Q3 FY 2018)

TBD by contractor 04/13/2018 04/30/2018

28

MV&R Reports, including a summary table of all flags, Action and Learning points from that round, and MVR site universe tracker as an annex provided to USAID for Round 3 (Q3 FY 2018).

TBD by contractor

06/29/2018 08/27/2018

29 MV&R Reports for Somali regional governments provided to USAID/SFO for round 3 (Q3 FY 2018)


30

Updated indicator Performance tracking table (IPTT) spreadsheet, disaggregated by PPR and non-PPR indicators, and tracker to include Q2 FY 2018.

Within 15 days of receipt of all IP data for Q2 FY 2018 quarterly reports.

06/21/2018 07/23/2018

31

IPTT summary for Q2 FY 2018 powerpoint highlighting performance against targets, disaggregated by PPR and all indicators.

Within 15 days of receipt if all IP data for Q2 FY 2018 quarterly reports

06/21/2018 07/23/2018

129

32

Consolidate into a tracker all IP responses to flags, action and learning points as detailed in their respective Q2 FY 2018 reports.

TBD by Contractor

06/21/2018 07/23/2018

33 Performance monitoring data collection tools reviewed and updated for Round 4 (Q4 FY 2018


34

One day workshop presenting the tools, protocols and flagging system


07/16/2018 08/27/2018

35 List of all activity sites for MVR round 4 (Q4 FY 2018)


36

MVR Reports, including a summary table of all flags, Action and Learning Points from that round, and MV&R site universe tracker as an annex provided to USAID/SFO for Round 5 (Q4 FY 2018)

TBD by Contractor

09/19/2018 10/30/2018

37 MV&R Report for SomalI regional governments provided to USAID/SFO for Round 4 (Q4 FY 2018)


38 Power-point presentation to USAID/SFO of findings from MV&R visits Round 3&4 (Q3 & 4 FY 2018).


39

Consolidated report of process learning for the full year, including best practices established, challenges of working within a dynamic environment and mitigation plans against those challenges.

TBD by Contractor

09/25/2018 10/12/2018

40 Ad hoc reports Upon request by

USAID/SFO N/A N/A

Task Order #18 Task Order/ Contract Number: TO#72062318F00002 Contract Section: F.4 DELIVERABLES Name: 2017 DATA QUALITY ASSESSMENT Period of Performance: November 16, 2017 to March 12, 2018.

130


Date Submitted

Date Approved

or Invoiced Data Quality Assessment (DQA)

1 Tools and Protocols for use during site visits

Week two of the TO award 11/20/2017 02/15/2018

2 Data Validation Workshop with USAID/Somalia and IPs

Week twelve of the TO award 03/29/2018 3/29/2018

3 Draft DQA Reports (IP Specific) Week fourteen of the TO award 02/23/2018 3/14/2018

4 Final DQA Reports (IP Specific and consolidated)

Week sixteen of the TO award 03/19/2018 03/26/2018

Task Order #19 Task Order/ Contract Number: TO#72062318F00003 Contract Section: F.4 DELIVERABLES Name: Gender Analysis 2017 Period of Performance: February 1, 2018 to March 30, 2018.


Date Submitted

Date Approved

or Invoiced Gender Analysis

1 Work Plan/Schedule and draft questions February 7, 2018 02/07/2018 02/13/2018

2 Status update to USAID/Somalia February 14, 2018 02/14/2018 03/12/2018

3 Powerpoint presentation

February 20, 2018 (draft) February 23, 2018 (Presentation)

02/27/2018 03/02/2018

4 Draft report March 9, 2018 03/09/2018 05/31/2018

5 USAID/SFO comments on the draft report February 28 - March 6, 2018 05/31/2018

6 Final Report March 14, 2018 03/14/2018 05/31/2018 Task Order #20 Task Order/ Contract Number: TO# 72062318F00005 Contract Section: F.4 DELIVERABLES Name: 2018 DATA QUALITY ASSESSMENT Period of Performance: September 18, 2018 to February 8, 2019


Date Submitted

Date Approved

Data Quality Assessment (DQA)

1 Training/sensitization materials, tools and protocols for use during site visits 10/01/2018 10/01/2018 10/11/2018

2 Training/Sensitization Report/Minutes/Agenda 10/15/2018 10/15/2018 10/11/2018

3 Indicator Assessment Summary Matrix (input for PPR) 11/22/2018 11/22/2018 11/28/2019

131

4 Data Analysis Report/Minutes/Agenda 11/25/2018 11/25/2018 11/28/2019 5 Data Validation Report/Minutes/Agenda 12/03/2018 12/03/2018 11/20/2018 6 Draft DQA Report 12/20/2018 12/19/2018 01/11/2019 7 Final DQA Report 01/20/2019 01/20/2019 01/31/2019 Task Order #21 Task Order/ Contract Number: TO # 72062318F00006 Contract Section: F.5 DELIVERABLES

Name: Monitoring, Verification, Reporting (MV&R V)

Period of Performance: October 1, 2018 to July 31, 2019.


Date Submitted


Monitoring, Verification and Reporting (MV&R)

1

Provide an individual detailed briefing to new IP(s) and their AOR/CORs on SPSS MV&R processes including reporting requirements and their role in that process

Within one month of the activity award

10/22/2018 10/22/2018

2 Google calendar with milestone dates relevant to AOR/CORs

Within one month of the activity award

10/29/2018 11/05/2018

3

Updated Indicator Performance Tracking Table (IPTT) spreadsheet, disaggregated by PPR and non-PPR indicators, and Tracker to include 3rd Quarter FY 2018


10/05/2018 10/12/2018

4

IPTT summary for 3rd Quarter FY 2018 Powerpoint highlighting performance against targets, disaggregated by PPR and all indicators


10/05/2018 10/12/2018

5

Consolidate into a tracker all IP responses to flags, Action and Learning Points as detailed in their respective Q3 FY 2018 Reports


6 [Removed] N/A N/A N/A



8 Training/sensitization on the tools, protocols and flagging system




10 Training materials for field supervisors and TPMs developed


132

11

MV&R Reports, including a summary table of all flags, Action and Learning Points for Round one, and MV&R site universe tracker as an annex provided to USAID/SFO for Round 1 (Q1 2019)




13

Updated Indicator Performance Tracking Table (IPTT) spreadsheet, disaggregated by PPR and non-PPR indicators, and Tracker to include Q4 FY 2018


01/07/2019 01/23/2019

14

IPTT summary for Q4 FY 2018 Powerpoint highlighting performance against targets, disaggregated by PPR and all indicators


01/07/2019 01/23/2019

15









19





21 Powerpoint presentation to USAID/SFO of findings from MV&R visits Round 1 & 2 (Q1 & Q2 FY 2019)


22



03/22/2019 04/10/2019

133

23



03/22/2019 04/10/2019

24









28





30 Powerpoint presentation to USAID/SFO of findings from MV&R visits Round 3 (Q3 FY 2019)


31



06/28/2019 07/29/2019

32



06/28/2019 07/29/2019

33



34

Consolidated report of process learning for the full year, including best practices established, challenges of working within a dynamic environment and mitigation plans against those challenges


134

35

Provide an annual compilation of environmental compliance findings for each relevant partner against their Environmental Mitigation and Monitoring Plan (EMMP)


36 Ad hoc reports Upon request by USAID/SFO N/A N/A

Task Order #22 Task Order/ Contract Number: TO #TO # 72062319F00001 Contract Section: F.5 DELIVERABLES

Name: Monitoring, Verification, Reporting (MV&R VI) 2019

Period of Performance: August 1, 2019 to December 31, 2019.


Date Submitted

Date Approved

or Invoiced Monitoring, Verification and Reporting (MV&R)

1

Provide an individual detailed briefing to new IP(s) and their AOR/CORs on SPSS MV&R processes including reporting requirements and their role in that process

As and when required N/A N/A

2

Updated Indicator Performance Tracking Table (IPTT) spreadsheet, disaggregated by PPR and non-PPR indicators, and Tracker to include Quarter 3 FY 2019


10/17/2019 10/22/2019

3

IPTT summary for Quarter 3 FY 2019 Powerpoint highlighting performance against targets, disaggregated by PPR and all indicators


10/17/2019 10/22/2019

4

Consolidate into a tracker all IP responses to flags, Action and Learning Points as detailed in their respective Quarter 3 FY 2019 Reports


5 Performance monitoring data collection tools reviewed and updated for Round 4 (Quarter 4 2019)




7 List of activity sites for MV&R Round 4 (Quarter 4 2019)


8



135

9 Powerpoint presentation to USAID/SFO of findings from MV&R visits Round 4 (Q4 FY 2019)


10



12/03/2019 12/11/2019

11



12/03/2019 12/12/2019

12



Task Order #23

Task Order/ Contract Number: TO # 72062320F00004

Contract Section: F.5 DELIVERABLES

Name: Monitoring, Verification, Reporting (MV&R VII)

Period of Performance: December 19, 2019 to June 30, 2021


Date Submitted

Date Approved

or Invoiced Monitoring, Verification and Reporting (MV&R)

1.

Provide an individual detailed briefing to new IP(s) and their AOR/CORs on SPSS MV&R processes including reporting requirements and their role in that process.

As and when required. N/A N/A

2

Updated Indicator Performance Tracking Table (IPTT) 2 spreadsheet, disaggregated by PPR and non-PPR indicators, and Tracker to include Quarter 1 FY 2020


04/03/2020 04/06/2020

136

3

Consolidate into a tracker all IP responses to flags, Action and 3 Learning Points as detailed in their respective Quarter 1 FY 2020 Reports.

TBD by Contractor

04/03/2020 04/06/2020

4 Performance monitoring data collection tools reviewed and 4 updated for Round 1 (Quarter 2 2020)

TBD by Contractor

01/31/2020 02/03/2020


TBD by Contractor

01/24/2020 02/13/2020

6

MV&R Reports, including a summary table of all flags, Action and Learning Points for Round one, and MV&R site universe tracker as an annex provided to USAID/Somalia for Round 1(Quarter 2 2020)

TBD by Contractor

04/09/2020 04/28/2020

7 Power 7 -point presentation to USAID/Somalia of findings from MV&R visits Round 1 (Quarter 2 FY 2020)

TBD by Contractor

04/29/2020 05/07/2020

8 Updated Indicator Performance Tracking Table (IPTT) spreadsheet, disaggregated by PPR and non-PPR indicators, and Tracker to include Quarter 2 FY 2020


07/10/2020

07/21/2020

9 Consolidate into a tracker all IP responses to flags, Action and Learning Points as detailed in their respective Quarter 2 FY 2020 Reports.

TBD by Contractor

07/10/2020

07/21/2020


TBD by Contractor

04/17/2020

04/20/2020


TBD by Contractor

04/09/2020

04/26/2020

12 MV&R Reports, including a summary table of all flags, Action and Learning Points for Round one, and MV&R site universe tracker as an annex provided to USAID/Somalia for Round 2 (Quarter 3 2020)

TBD by Contractor

06/26/2020

07/20/2020

13 Power-point presentation to USAID/Somalia of findings from MV&R visits Round 2 (Quarter 3 FY 2020)

TBD by Contractor

07/30/2020 07/30/2020

14



10/02/2020 10/15/2020

137

15

Consolidate into a tracker all IP responses to flags, Action and Learning Points as detailed in their respective Quarter 3 FY 2020 Reports.

TBD by Contractor

10/02/2020 10/15/2020


TBD by Contractor

07/13/2020 07/20/2020


TBD by Contractor

07/06/2020 07/20/2020

18

MV&R Reports, including a summary table of all flags, Action and Learning Points for Round one, and MV&R site universe tracker as an annex provided to USAID/Somalia for Round 3 (Quarter 4 2020)

TBD by Contractor

09/25/2020 10/15/2020

19 Annual Environmental Mitigation and Monitoring Plan (EMMP)

TBD by Contractor

11/20/2020 01/05/2021


TBD by Contractor

10/13/2020 01/05/2021

21



01/12/2021 02/10/2021

22


TBD by Contractor

01/12/2021 02/10/2021


TBD by Contractor

09/29/2021 10/01/2020


TBD by Contractor

09/29/2020 10/15/2020

25


TBD by Contractor

01/08/2021 02/04/2021


TBD by Contractor

02/17/2021 02/24/2021

27


Within 15 days of receipt of all IP data for

03/16/2021 03/18/2021

138

Q1 FY 2021 quarterly reports

28


TBD by Contractor

03/16/2021 03/18/2021


TBD by Contractor

03/16/2021 01/15/2021


TBD by Contractor

01/08/2021 01/21/2021

31


TBD by Contractor

03/26/2021 04/21/2021


TBD by Contractor

05/07/2021 05/20/2021

33



06/04/2021 06/07/2021

34 Consolidate into a tracker all IP responses to flags, Action and Learning Points as detailed in their respective Quarter 2 FY 2021Reports.

TBD by Contractor

06/04/2021 06/07/2021


TBD by Contractor

03/24/2021 03/25/2021


TBD by Contractor

03/26/2021 03/31/2021

37


TBD by Contractor

06/03/2021 06/14/2021


TBD by Contractor

06/03/2021 06/14/2021

Data Quality Assessments (DQA)

1 Training/sensitization materials, tools and protocols for use during site visits

TBD by Contractor

08/26/2020 09/08/2020

2 Training/Sensitization Agenda TBD by Contractor

08/26/2020 09/08/2020

3 Indicator Assessment Summary Matrix (input for PPR)

TBD by Contractor

11/12/2020 12/07/2020

139

4 Data Validation Report/Minutes/Agenda TBD by Contractor

12/10/2020 01/25/2021

5 Draft DQA Report TBD by Contractor

01/08/2021 03/02/2021

6 Final DQA Report TBD by Contractor

03/08/2021 03/31/2021

Task Order #24 Task Order/ Contract Number: TO#72062320F00007 Contract Section: F.5 DELIVERABLES.

Name: CITIZEN CONSULTATION AND GENDER ASSESSMENT 2020

Period of Performance: Aug 10, 2020 to June 30, 2021


Date submitte

d

Date Approved

or Invoiced Citizen Consultation

1

Work plan detailing process for gathering Mission input, drafting the Results Framework and other related documents, and preparing final documents for the Phase 2 DVC

TBD by Contractor

09/04/2020 09/11/2020

2 Coordination and facilitation of meetings and Results Framework workshop with USAID

TBD by Contractor

10/23/2020 11/17/2020

3 Radio Show #1 (Series 1) TBD by Contractor

12/04/2020 12/11/2020


12/04/2020 12/11/2020


12/04/2020 12/11/2020

6 Series 1 Analysis Workshop TBD by Contractor

12/04/2020 12/11/2020

7 Series 1 Report TBD by Contractor

01/13/2021 03/13/2021


03/01/2021 04/12/2021


03/09/2021 04/12/2021


03/16/2021 04/12/2021

11 Series 2 Analysis Workshop TBD by Contractor

03/31/2021 04/12/2021

12 Series 2 Report TBD by Contractor

04/29/2021 05/17/2021

13 Final Report TBD by Contractor

05/28/2021 Pending

14 Dissemination Workshop TBD by Contractor

05/20/2021 05/20/2021

Gender Assessment

140

1

Inception Report: Preliminary desk review, assessment questions, preliminary data collection and analysis plan, list of potential interviewees, limitations to the assessment design, workplan.

TBD by Contractor

08/24/2020 09/08/2020

2 Inception Report Presentation TBD by Contractor

10/05/2020 10/05/2020

3

Final Assessment Design Report: In addition to what is in the Inception Report, should include fleshed-out desk review, finalized KII list, and finalized data collection instruments

TBD by Contractor

09/11/2020 10/05/2020

4 Data Collection Out Brief TBD by Contractor

11/13/2020 12/01/2020

5 Final Report TBD by Contractor

10/30/2020 01/19/2021

6 Dissemination Workshop TBD by Contractor

02/02/2021 02/24/2021