Date: February 2018Prepared for: Concern Worldwide Consortium Coordinating Unit Name: Mo Ali, Kate Hutton, Julia Lewis.Contact Details: info@aidworks.org.uk

Final Evaluation of Building Resilience in Chad and Sudan

The contents of this report are the sole responsibility of Aid Works and do not necessarily reflect the views of Concern Worldwide

This material has been funded by UK aid from the UK government; however the views expressed do not necessarily reflect the UK government’s official policies.

© Abdalla Mohamed Yahia

About the evaluation team Kate Hutton (Team Leader with focus on Sudan) has considerable experience in fragile states (Sierra Leone, South Sudan, Darfur), particularly in health, nutrition, livelihoods and community-based programmes. Kate also has substantial experience leading reviews and evaluations of multi-sector programmes using mixed approaches. All her evaluations have ensured a gender equity lens on data collection and results, and have been based on the criteria of the Organisation for Economic Co-operation and Development-Development Assistance Committee. Kate has a deep understanding of the disaster risk reduction approach through working with various non-governmental organisations (NGOs).

Julia (Evaluator with focus on Chad) is a cross-sectoral emergency response specialist with experience in conflict, post-conflict and development settings, predominantly across Central and West Africa and the Horn of Africa (English- and French-speaking countries). Julia has significant experience in strategic planning, management, programme design, and monitoring and evaluation (M&E). She has implementation experience across sectors including health and nutrition, food security and livelihoods, non-food items/shelter; education, water, sanitation and hygiene (WASH), and integrated resilience programming.

Mo Ali (Quality Advisor) has extensive experience in monitoring and evaluation, learning, systems and strategy development. He has led a large number of evaluations and reviews for NGOs, UN agencies, the Department for International Development (DFID) and USAID, and designed a wide range of research pieces, tools and analysis methodologies. He is Aid Works’ lead for quality assurance and value for money.

Toby Gould (WASH Expert) and John Spilsbury (Agriculture and Livelihoods Expert) provided technical advice.

Aid Works is a high-impact social enterprise that provides aid organisations with tailored project support and learning services. Worldwide, Aid Works provides support through our strong, solution-focused partnerships with aid organisations. Aid Works designs projects, develops systems, evaluates organisations’ work, and designs and facilitates training courses. We also provide leadership coaching for international and local development workers.

Acknowledgements Kate Hutton and Julia Lewis wrote the report, supported by Mo Ali, with critical review by Toby Gould and John Spilsbury. The authors would like to thank Concern’s Consortium Coordinating Unit, in particular Gretta Fitzgerald for technical leadership and Rory Crew for operational support. We would also like to thank the consortium members, Concern’s country offices’ support, and all those who participated in the interviews and focus groups.

Cover photo credit: Geneina, West Darfur taken by Abdalla Mohamed Yahia

© Aid Works Limited | www.aidworks.org.uk | Registered in England and Wales Company No. 8950939

Executive summary Building Resilience in Chad and Sudan (BRICS) was part of a three-year, £140m global resilience programme known as BRACED (Building Resilience and Adaptation to Climate Extremes and Disasters). BRICS ran from April 2015 to December 2017 in Chad and Sudan. In Chad, the programme followed on from a resilience project: Community Resilience to Acute Malnutrition (CRAM), an operational research programme. In Sudan this was the first time Concern had ventured into an integrated resilience programme, modelled on CRAM.

A consortium of partners implemented BRICS: Concern Worldwide (consortium lead), Feinstein International Center at Tufts University (Tufts), the World Agroforestry Centre (ICRAF) and Concern’s partner in Sudan, Al Massar. The Consortium Coordinating Unit (CCU) based in Concern’s London office, managed the consortium and was responsible for coordinating the four partners. BRICS programme activities fell into six intervention packages: climate-smart agriculture (CSA); disaster risk reduction and early warning systems (EWS); water, sanitation and hygiene; health and nutrition; gender equality; and learning and advocacy.

The purpose of the final evaluation was to provide an assessment of the achievements and successes of, and challenges to, the BRICS programme. The evaluation provides a better understanding of how and why the programme worked, or did not, documenting lessons learnt. The evaluation was conducted between September 2017 and February 2018, with questions based on evaluation criteria defined by the Organisation for Economic Co-operation and Development (OECD). The questions were:

Relevance and impact

1. To what extent have particular interventions led to anticipated changes and results? 2. Were there any unintended attributable results and why?

Effectiveness

3. How and why have particular intervention packages led to observed results and changes? 4. What evidence is there of knowledge and capacity transfer between partners and local

stakeholders?

Efficiency

5. How did the consortium model enable or constrain delivery of the project’s Theory of Change (ToC)?

6. To what extent did the project represent value for money (in terms of efficiency)?

Sustainability

7. What impact (if any) did the project have on internal (and external) thinking about resilience and did it help to crystallise this?

8. What evidence is there that benefits from the project will continue after donor funding has ceased?

9. What evidence is there that interventions and the mechanisms that support them have the potential to deliver ‘amplified’ results?

Lessons learnt

10. What has the project learnt about delivering these packages of interventions?

Relevance and impact

The BRICS programme increased the resilience of vulnerable populations to withstand predicted climate change shocks and extremes, by improving their capacity to absorb, adapt to and anticipate them. The design of BRICS was described in its ToC, which was updated to better reflect activities

© Aid Works Limited | www.aidworks.org.uk | Registered in England and Wales Company No. 8950939

and processes. However, it was not a tool the country teams used and did not fully align with the logframe.

CSA activities have helped target households to better absorb and adapt to the effects of climate change by increasing their year-round food security, dietary diversity and income, while also preventing desertification through farmer-managed natural regeneration (FMNR) techniques and tree planting. Local seed banks, availability of climate-smart seed varieties, and livestock health were important factors in improving results.

Improved information and communication about weather patterns and rainfall have been important in reducing the risk of disasters and helping communities to better plan agricultural activities. Establishing community grain stores provided a safety net in case household stocks were exhausted.

Ensuring access to improved water sources was central to the BRICS programme. Communities now have greater access to improved water sources. This saves time for women, who can spend it productively elsewhere, and reduces disease and illness in the community, thereby increasing resilience. Water user committee (WUCs) have successfully ensured that water points continue to function and have the potential to ensure the reliability of water sources, provided they have a consistent and guaranteed income. However, understanding of the link between improved health, sanitation and hygiene practices in the communities targeted is limited and mixed.

Training of health volunteers and lead mothers has improved mothers’ and caregivers’ understanding of nutrition and health practices, improved referrals and highlighted the importance of seeking care at health centres. Concern’s support to health and nutrition centres has improved quality of care and health indicators, but progress is unlikely to be maintained without Concern’s ongoing support. Health centres could have been better prepared for peaks in malnutrition if they had implemented the community-managed acute malnutrition (CMAM) surge model.

Gender equality activities had a positive impact on the other intervention packages. Evidence showed that women were participating more in decisions at community level, which in turn has contributed to community resilience. The programme has made more progress in Chad than Sudan, where communities are more conservative and access more restricted.

There were challenges related to the context and resourcing of BRICS advocacy work. However, the programme showed evidence of contributions to and influence on state, national and international initiatives, policies and wider climate change platforms.

The BRICS consortium sought new opportunities to engage externally and communicate results at federal level, which produced unintended benefits. These included the use of improved seeds by other agencies in Sudan and Concern’s contribution to climate-smart policies. Other unintended results of the programme included: community members or groups actively seeking opportunities to invest resources made available to them through BRICS; and the unexpected reduction of conflict between pastoralists and agricultural groups over water and land, due to increasing availability or reducing their use of these resources.

Effectiveness

Providing technical support and training were two key change processes the ToC outlined that have supported the outputs and outcome. The introduction of two new intervention packages – CSA and gender equality – gave the programme greater capabilities to build resilience and was an opportunity to implement new activities, which also increased staff knowledge and skills. Similarly, the introduction of new techniques and technologies increased staff skills and their confidence to employ and use new methods in a given context.

Relationships with key stakeholders and the consortium members provided access into the community (and greater reach), and strengthened the intervention packages: CSA, EWS, gender equity, and learning and advocacy. Improved access to water was core to the WASH intervention

© Aid Works Limited | www.aidworks.org.uk | Registered in England and Wales Company No. 8950939

package, supported better health and livelihoods, and allowed quicker buy-in from communities for other activities.

There is evidence of knowledge and capacity transfer between partners and local stakeholders. Staffing levels and inexperience slowed delivery at the start of the programme. However, programme teams had opportunities to learn about new technologies and observe other country projects. The mid-term review provided a process whereby partners and field teams learnt about and adapted the programme. Training of beneficiaries led to increased knowledge in hygiene practices, nutrition, WASH and CSA. Low literacy rates slowed progress of knowledge transfer and functionality of committees, which was addressed with visual tools and practical support.

Efficiency

The consortium model enabled partners to bring their specific expertise to the programme. Concern’s previous experience of CRAM made it a core member of the consortium. The participation of Al Massar and Tufts gave the programme an opportunity to deliver activities to pastoralist communities, increasing the confidence of programme staff to be able to work with this vulnerable group in future programmes.

Expertise from ICRAF provided an opportunity to trial new agricultural techniques, enabling staff to gain new skills and confidence. Tufts provided expertise in monitoring and research, producing a large amount of data and publications. Initially lack of communication and sharing of learning constrained the consortium, but the workshop in London made significant changes in how consortium members related to one another.

The evaluation was not designed to analyse financial data for the project. It is therefore difficult to say whether BRICS represents value for money in terms of efficiency without deeper investigation of cost drivers. Co-funding from Irish Aid means DFID gets better results for its investment. One of the main cost drivers was staff salaries. Given that the majority of staff were locals, it is more likely the knowledge they gained through the programme will remain in-country. The use of technology supported the scale-up of data collection, but the right staff capacity had to be in place.

The evaluation noted several efficiency gains in specific activities, which will have saved money. The added function of communication and burdensome reporting to the knowledge manager (KM) meant senior staff spent less time on programme management. The benefits of providing information to the KM were not evident at country level. With BRICS working in two countries, there is an opportunity for a value for money analysis comparing the two operations.

Sustainability

Internal and external stakeholders achieved a greater understanding of what resilience is, why it is necessary and how to achieve it. This included an understanding of which resilience activities are possible to implement in post-conflict and vulnerable communities, and that increasing resilience is as important for pastoralist as agro-pastoralist and agricultural households. It has also changed how field teams perceived Concern and its role beyond emergency response.

In contexts used to emergency response programmes, BRICS programme showed that responding to emergencies is not only a goal in itself, but also a means of protecting those gains made in longer-term programming during crises. The programme has underlined that small, practical actions and community-level changes can have an impact on resilience alongside– and to the same extent as – more expensive and intensive activities.

Given the short duration of the programme, there is evidence that benefits will continue. However, this evidence is weak, particularly in relation to behaviour change where evidence is mixed that increasing knowledge has led to changes in behaviour. An exit strategy would have helped to map how the benefits might continue after the end of the programme.

© Aid Works Limited | www.aidworks.org.uk | Registered in England and Wales Company No. 8950939

As mentioned above, the knowledge and skills the programme has strengthened are likely to stay in-country, as the majority of staff recruited for the programme were local. Activities that communities reported would be most likely to be sustainable were those that focused on building knowledge, created demand for goods and services, or had already produced tangible results. It is essential that the government implements the climate-smart policies to which BRICS has contributed for any benefits to be gained.

The BRICS programme has succeeded in reaching communities beyond those originally targeted to receive support, which are independently copying programme interventions. These include latrine construction in neighbouring villages, establishment of tree nurseries in Chad and construction of seed banks in Sudan. In Sudan, the interest other states have shown in using community radio to disseminate weather information, and that organisations have shown in using early-maturing seeds, have further amplified results.

The BRICS programme has also shown the potential for developing savings schemes, something that may fare better in more established programme locations with pre-established groups. Finally, the success of the livestock pharmacy, widespread use of good-quality livestock medicine in targeted villages, and Concern’s existing relationship with the Pastoralists’ Federation network provide an opportunity to promote livestock health and other resilience strategies further afield.

Lessons learnt

The evaluation report documents lessons on delivering the packages, resilience, context, structure and gender equality. They are detailed fully in section 5; some key lessons are:

• Taking an integrated approach and implementing all intervention packages in targeted villages will contribute to the success of each and of the overall project. However, the approach needs to be delivered in a realistic manner and not be overambitious in the number of communities it targets;

• Interventions do not build resilience equally and often depend on specific change agents, individuals or a mixture of change processes and levels of community involvement; and

• Gender equality activities are not expensive to implement in comparison to other packages. Their success depends on choice of community, local cultural acceptance, community buy-in, time, and local staff members’ own awareness, understanding and buy-in.

Recommendations

• Strengthen the intervention packages • Implement additional activities • Improve the design of the project • Strengthen learning and communication • Improve collaboration with wider stakeholders