study on legal instruments and lessons learned from...

Study on Legal Instruments and Lessons Learned from the Evaluat ions Managed by the

Jo int Evaluat ion Unit

Final Report

Volume 3 The Review of Evaluation Reports

July 2011

Evaluation for the European Commission

PARTICIP GmbH Germany

Framework contract for Multi-country thematic and regional/country-level strategy evaluation studies and synthesis in the area of external co-

operation

Aide à la Décision

Economique Belgium

LOT 5: Evaluation of EC main policies and strategies in the areas of

external cooperation

Italy

Ref.: EuropeAid/122888/C/SER/Multi Request for Service: 2010/247813

Deutsches Institut für Entwicklungspolitik

Germany

Study on Legal Instrument and Lessons

Learned from the Evaluations Managed by the Joint Evaluation Unit

European Centre for Development

Policy Management Belgium

Overseas Development

Institute, United Kingdom

FINAL REPORT

A consortium of

Particip-ADE–DRN-DIE–ECDPM-ODI

c/o Particip GmbH, leading company:

Headquarters Merzhauser Str. 183 D - 79100 Freiburg / Germany Phone: +49-761-79074-0 Fax: +49-761-79074-90 [email protected]

Belgium office Avenue des Arts 50 (5th floor) B-1000 Bruxelles / Belgium Phone: +32-2-5501160 Fax: +32-2-5501169 [email protected]

July 2011

This evaluation was carried out by The European Centre for Deve lopment

Pol i cy Management

The evaluation team was composed by: James Mackie (Team leader), Niels Keijzer, Faten Aggad, Alisa Herrero-Cangas, Simone Görtz, Eunike Spierings, Sophie Papalexiou, Christopher Veit, Véronique Girard. James Mackie was the evaluation manager. The evaluation is being managed by the Joint Evaluation Unit. The authors accept sole responsibility for this report, drawn up on behalf of the Commission of the European Union. The report does not necessarily reflect the views of the Commission

Study on Legal Instruments and Lessons Learned from the Evaluations Managed by the Joint

Evaluation Unit

Final Report

The report consists of 3 volumes: Volume 1: Draft Final Summary Report Volume 2: The Intervention Logics Volume 3: The Review of Evaluation Reports VOLUME 1: DRAFT FINAL REPORT 1. Introduction 2. Conceptual Framework 3. Methodology 4. Phases 1 and 2: The Intervention Logic of the Legal Instruments 5. Phase 3: Linking Evaluation Results (2006-2011) to the Legal Instruments Objectives 6. Phase 4: Recommendations for the Future Legal Instruments and their Evaluation 7. Concluding Remarks 8. Bibliography Annex: Reconstructed Intervention Logics VOLUME 2: THE INTERVENTION LOGICS 1. Introduction: The Intervention Logics for the Legal Instruments 2. Phase 1: Constructing the Faithful Intervention Logics

2.1 Step 1: Analysis of the Legal Instruments / Intervention Logic Diagrams 2.2 Step 2: Comparison of Legal Instruments

3. Phase 2: Analysing the Reconstructed Intervention Logics 3.1 Evidence Collected 3.2 Synthesis of the Evidence from Phase 2 3.3 Some Critical Questions Emerging from Phase 2

Annex 1: Comparing the 2004 DCECI Proposal with the 2006 DCI Regulation Annex 2: Comparing the 2004 ENPI Proposal with the 2006 ENPI Regulation Annex 3: Comparing the 2004 DCECI Proposal with the 2006 ICI Regulation Annex 4: Comparing the Original EC Proposal on the EIDHR (2006) with the Changes Introduced to the Final Text of the Regulation (2006) Annex 5: Comparing the Nuclear Safety Provisions in the 2004 IfS Proposal with the 2007 INSC Regulation Annex 6: Faithful Intervention Logics Annex 7: Reconstructed Intervention Logics

VOLUME 3: THE REVIEW OF EVALUATION REPORTS About this Volume 1. Analysis of the Evaluation Evidence 2. Synthesis of the Phase 3 Evidence on the Six Legal Instruments 3. Review of Evidence on Implementation Issues Annex 1: Grid for Analysing Evaluation Reports Annex 2: Trends in Management, Modality, Approach and Performance

STATUS AND VERSION OF THE DOCUMENT

Version Date provided Status Validation/

comments by Date of validation/

comments

1 25/03/2011 Draft

2 26/07/2011 Final

3

4

Study on Legal Instruments and Lessons Learned from the Evaluations Managed by the Joint Evaluation Unit PARTICIP-ADE-DIE-DRN-ECDPM-ODI

Final Report July 2011 Page i

TABLE OF CONTENTS

About This Volume ............................................................................................................... 1

1 Analysis of the Evaluation Evidence ............................................................................... 11.1 INTRODUCTION .............................................................................................................................. 11.2 THE OVERALL PICTURE ................................................................................................................ 31.3 FURTHER ANALYSIS OF THE OVERALL DATA SET .................................................................... 41.4 THE DEVELOPMENT COOPERATION INSTRUMENT (DCI) ...................................................... 6

1.4.1 Introductory Remarks ........................................................................................................... 61.4.2 Response to SQ 5: What are the overall results and effects pertinent to the instruments identified in the evaluations? ....................................................................................... 71.4.3 Response to SQ6: On the basis of results identified (in Q5) are there any overall trends or lessons to learn to be identified from the evaluations and that might be pertinent to the instruments? .......................................................................................................................... 17

1.5 THE EUROPEAN NEIGHBOURHOOD POLICY INSTRUMENT (ENPI) ................................... 191.5.1 Response to SQ5: Overall results and effects pertinent to the ENPI instrument identified in the evaluations ............................................................................................................ 191.5.2 Response to SQ6: Overall trends or lessons to learn identified from the evaluations that are pertinent to the ENPI instrument .................................................................................. 25

1.6 INSTRUMENT FOR STABILITY (IFS) ............................................................................................ 261.6.1 Response to SQ5: What are the overall results and effects pertinent to the instruments identified in the evaluations? .................................................................................... 261.6.2 Response to SQ6: On the basis of results identified (in Q5) are there any overall trends or lessons to learn to be identified from the evaluations and that might be pertinent to the instruments? .......................................................................................................................... 27

1.7 INDUSTRIALISED COUNTRIES INSTRUMENT (ICI) .................................................................. 281.7.1 Introduction: ........................................................................................................................ 281.7.2 Response to SQ5 : What are the overall results and effects pertinent to the instruments identified in the evaluations? .................................................................................... 281.7.3 Response to SQ6 : On the basis of results identified (in SQ5) are there any overall trends or lessons to learn to be identified from the evaluations and that might be pertinent to the instruments? .......................................................................................................................... 30

1.8 EUROPEAN INSTRUMENT ON DEMOCRACY AND HUMAN RIGHTS (EIDHR) ................... 301.8.1 Response to SQ5: What are the overall results and effects pertinent to the instruments identified in the evaluations? .................................................................................... 301.8.2 Response to SQ6: On the basis of results identified (in SQ5) are there any overall trends or lessons to learn to be identified from the evaluations and that might be pertinent to the instruments? .......................................................................................................................... 33

1.9 INSTRUMENT FOR NUCLEAR SAFETY COOPERATION (INSC) .............................................. 341.9.1 Response to SQ5 : What are the overall results and effects pertinent to the instruments identified in the evaluations? .................................................................................... 341.9.2 Response to SQ6: On the basis of results identified (in SQ5) are there any overall trends or lessons to learn to be identified from the evaluations and that might be pertinent to the instruments? .......................................................................................................................... 35

2 Synthesis of the Phase 3 Evidence on the Six Legal Instruments ................................ 372.1 RESPONSE TO SQ5: WHAT ARE THE OVERALL RESULTS AND EFFECTS PERTINENT TO THE INSTRUMENTS IDENTIFIED IN THE EVALUATIONS? ................................................................. 372.2 RESPONSE TO SQ6: ON THE BASIS OF RESULTS IDENTIFIED (IN SQ5) ARE THERE ANY OVERALL TRENDS OR LESSONS TO LEARN TO BE IDENTIFIED FROM THE EVALUATIONS AND


Final Report July 2011 Page ii

THAT MIGHT BE PERTINENT TO THE INSTRUMENTS? ....................................................................... 40

3 Review of Evidence on Implementation Issues ........................................................... 423.1 INTRODUCTION ........................................................................................................................... 423.2 GLOBAL COUNT OF EVALUATIONS OF EXTRACTS USED FOR THE ANALYSIS ................... 423.3 EFFECTIVENESS AND THE 3CS .................................................................................................. 43

3.3.1 Alignment ............................................................................................................................. 433.3.2 Coherence ............................................................................................................................ 443.3.3 Complementarity ................................................................................................................ 443.3.4 Coordination ........................................................................................................................ 443.3.5 Synergy ................................................................................................................................. 45

3.4 PROGRAMME MANAGEMENT CYCLE ....................................................................................... 453.4.1 Programming ....................................................................................................................... 463.4.2 Delays ................................................................................................................................... 463.4.3 M&E / Exit Strategy / Sustainability .............................................................................. 473.4.4 Learning ................................................................................................................................ 47

3.5 ACTOR BASED DIMENSION ........................................................................................................ 483.5.1 Participatory Approach ...................................................................................................... 483.5.2 Partners Capacity ................................................................................................................ 48

3.6 MODALITY .................................................................................................................................... 493.6.1 Project Approach ................................................................................................................ 493.6.2 Budget Support ................................................................................................................... 503.6.3 Modality Mix ........................................................................................................................ 51

3.7 CONCLUSION ................................................................................................................................ 52

Annex 1: Grid for Analysing Evaluation Reports ................................................................ 53

Full Listing of Evaluation Reports ..................................................................................... 54

Annex 2: Trends in Management, Modality, Approach and Performance ........................ 55

List of Tables: Table 1: Types and sub-types of evaluations synthesised ...................................................................... 1Table 2: Dates of evaluations synthesised ................................................................................................ 2Table 3: Number of results linked to expected results of the intervention logics .............................. 4Table 4: Sectoral distribution of identified results by instrument ......................................................... 5Table 5: Number of ex-post evaluation results related to the expected results of the DCI

Intervention Logic ............................................................................................................................... 7Table 6: Positive, mixed and negative results as related to the DCI .................................................. 10Table 7: Summary table of evaluation results in relation to the DCI ................................................. 12Table 8: DCI related results per sector ................................................................................................... 13Table 9: Results from EDF countries and their links to DCI Result areas ....................................... 14Table 10: Evaluation results as related to the DCI expected impact .................................................. 15Table 11: Overview of unique results linked to the ENPI .................................................................. 19Table 12: Evaluation results for each of the ENPI intervention logic Results ................................. 22Table 13: Ex-post evaluation results that can be related to the ICI instrument ............................... 28Table 14: Evaluation results related to the expected Results of the EIDHR Intervention Logic . 31Table 15: Number of evaluations and extracts used for each theme ................................................. 42Table 16: Number of evaluations and extracts used for each sub-theme .......................................... 43


Final Report July 2011 Page iii

List of Figures: Figure 1: Number of unique results linked to the main expected Results categories of the DCI

Intervention Logic ............................................................................................................................... 8Figure 2: Excerpt from the faithful DCI intervention logic ................................................................ 15Figure 3: Relative distribution of evaluation results to impact DCI (% of total results) ................. 16Figure 4: Sectoral distribution of evaluation results for the ENPI ..................................................... 20Figure 5: Extract from the ENPI intervention logic ............................................................................ 21Figure 6: Link between the evaluation results and the intervention logic results ............................. 32Figure 7: Results per sector ....................................................................................................................... 32


Final Report July 2011 Page iv

ACRONYMS

3Cs Coherence, Complementarity and Coordination AAP Annual Action Programme ACP African, Caribbean and Pacific countries ASEAN Association of Southeast Asian Nations ASEC ASEAN Secretariat BS Budget support CBC Cross Border Cooperation CFSP Common Foreign and Security Policy COM European Commission Communication CSO Civil Society Organisation CSP Country Strategy Paper DAC Development Assistance Committee DCECI Development Cooperation and Economic Cooperation Instrument DCI Development Cooperation Instrument DEVCO Directorate General Development and Co-operatoin DG RELEX Directorate General External Relations EC European Commission ECHO Directorate General Humanitarian Aid and Civil Protection EDF European Development Fund EEA European Economic Area EEAS European External Action Service EIA Extended Impact Assessment EIB European Investment Bank EIDHR European Instrument on Democracy and Human Rights ENP European Neighbourhood Policy ENPI The European Neighbourhood Policy Instrument EP European Parliament ERDF European Regional Development Funds ESDP European Security and Defense Policy ETP European Training Programme EU European Union EUD European Union Delegation Euratom European Atomic Energy Community FDI Foreign Direct Investment FIL Faithful Intervention Logic GBS General Budget Support IAEA International Atomic Energy Agency ICC International Criminal Court ICI Industrialised Countries Instrument IFI International finance institution IfS Instrument for Stability IL Intervention Logic INSC Instrument for Nuclear Safety Cooperation IP Investment Promotion IPA Pre-Accession Assistance JEU Joint Evaluation Unit JRC Joint Research Centre LICs Local Information Commission M&E Monitoring & Evaluation


Final Report July 2011 Page v

MDGs Millennium Development Goals MEDA Instrument of economic and financial cooperation under the Euro-

Mediterranean partnership MEP Member of Parliament MICs Monitoring and Information Centre MS Member State MTR Mid-Term Review NA Not Applicable NGO Non-Governmental Organisation NRM Natural resource management NSA Non-State Actor ODA Official Development Assistance OECD Organisation for Economic Co-operation and Development OJ Office Journal of the European Communities OSA On-site assistance and design safety PCA Partnership and Co-operation Agreement PCD Policy Coherence for Development PCM Project Cycle Management PEGASE Mécanisme Palestino-Européen de Gestion de l'Aide Socio-Economique NIP National Indicative Programme RIP Regional Indicative Programme PKO Peace keeping operations PMU Programme Management Unit PR Progress Report PRSPs Poverty Reduction Strategy Papers RG Reference Group RWM Radioactive waste management and safeguards SEC Secretariat of the Council of the European Union SIGMA Support for Improvement in Governance and Management SMEs Small and Medium Enterprises SQ Study Question TACIS Technical Assistance to the Commonwealth of Independent States TAIEX Technical Assistance and Information Exchange TDCA Trade, Development and Cooperation Agreement TEC Treaty establishing the European Community TEU Treaty on European Union TORs Terms of Reference UN United Nations WTO World Trade Organisation


Final Report July 2011 Page 1

ABOUT THIS VOLUME

This third Volume of the final report relates to the evidence collected from the Evaluation Reports and makes the link between this and the Results and Impacts identified in the faithful intervention logics for each of the six financial instruments. While key findings and implications for the Study are presented in Volume 1, this Volume presents the full analysis of the results, findings and analysis of the evaluations listed in annex 1 to the TOR. These results, findings and analysis were validated further through different means, including the Reference Group’s meeting about this desk report, and finalised and completed during the fourth and final phase of this study.

1 ANALYSIS OF THE EVALUATION EVIDENCE

1.1 Introduction

The third objective of this Study, as mentioned in the TOR, is to provide a “synthesis of the evaluations mentioned in Annex 1.” This work constitutes Phase 3 of the Study. The synthesis should describe the results identified in the evaluations and how they relate to the expected effects identified in Phases 1 and 2 of this study. Then, on the basis of a comparative analysis of the findings and conclusions of all these three phases, the Study will identify lessons learned from the evaluations and assess how the findings can be used to improve the legal instruments in future. In the Inception Report, it was noted that the synthesis could potentially cover a total of 65 evaluation reports. Of these 65, 20 evaluations were noted in the TOR as “Foreseen 2010 and beginning 2011”. At the end of the data collection process for this study, only 12 of the 20 expected extra studies had been completed, which meant that the analysis of the results as identified has been done on the basis of 57 evaluation reports (full listing found in Annex 1, after the analysis grid). These reports can be divided into the following three groups:

Table 1: Types and sub-types of evaluations synthesised

Country evaluations1 Regional evaluations Thematic evaluations

ACP 17 ACP 5 Channel/modality evaluations 4

DCI 12 ENPI (MEDA II and TACIS) 2 Cross-cutting issues and

sectors 8

ENPI 5 DCI (ASEAN2 and Central America) 2 Aid effectiveness/

Europeanisation 1

Other (statistical support) 1

Total country evaluations 34 Total regional evaluations

9 Total thematic evaluations

14

Given the Joint Evaluation Unit’s (JEU) own systems for quality control of the evaluations it commissions, the study team worked on the assumption that the

1 This differentiation is based on the grouping in which the respective countries are located. It may however be that

the geographic evaluation concerned covers interventions funded through various legal instruments (e.g. both DCI thematic programmes and EDF geographic instruments).

2 ASEAN : Association of Southeast Asian Nations



evaluation reports are sufficiently valid and reliable to be considered as independent evaluation evidence, and has not made its own examination of this issue. Several of the evaluations which were examined were finalised several years ago, as shown in Table 5 below. This has consequences for the relevance of the results in terms of the instruments this study is focusing on. The Study however assumes that that there is a high degree of continuity of the external action work of the Commission. Thus these results, even from 2005 or before, and even if evaluations themselves refer to instruments that no longer exist, are still relevant to the Study. These can therefore still be related to the appropriate current instruments to which they can be considered relevant in a thematic or geographic sense, or otherwise.

Table 2: Dates of evaluations synthesised

Year Country evaluations Regional evaluations Thematic evaluations 2006 7 2 5 2007 6 3 3 2008 1 2 4 2009 11 2 1 2010 7 - - 2011 2 - 1

As the principal means for conducting the data collection for producing the synthesis, a specific analysis grid has been developed which can be found in Annex 8 to this desk report. Given the large variety of evaluation reports to be studied, the study required a rather broad and flexible definition of results to be sure that it collects relevant evidence from the reports. The JEU’s glossary on its web pages defines a ‘result’ as “The initial impact of an intervention” and ‘impact’ as “A general term used to describe the effects of an intervention on society”.3 Building on this definition, the study team considered that the literature synthesis needed to provide an overview of societal effects in the countries concerned, which can to some extent be contributed or attributed to EC-funded interventions.4 The efforts made by the JEU to ensure both a standard methodology as well as approach in the evaluations it commissions means that the TORs it uses include standard requirements for the structure of the various reports produced in the context of an evaluation. This practice facilitated the work of the study team, as it meant that the evidence of societal effects is commonly located in the chapter in which the consultants respond to the Evaluation Questions. In studying the evaluation reports this chapter was therefore the prime focus of the team’s attention, based on the assumption that the findings on the contribution made to societal results was found here. At the same time, the grid proposed by the study team also allows for an overall examination of the full reports and the possibility to identify and include additional results during the data collection in case these results are located in other chapters. Given the high number of results identified in each evaluation, the results were grouped by ‘labelling’ them with specific sectors or areas of work (e.g. rural development,

3 Source of the definitions quoted:

http://ec.europa.eu/europeaid/evaluation/methodology/glossary/glo_en.htm 4 In case that an evaluation report does not include any evidence on societal effects, it will still be covered in the data

analysis and a second or third volume of the final report in the form of the filled-in grid. In this case, however, the study would not appear in the synthesis produce in phase 3 given the absence of these results in terms of societal effects.



democracy and human rights). In addition, the grid includes columns to relate the results to the third and fourth column of the intervention logics (respectively Results5 and ‘Impacts’). The grid was tested during the inception phase for three country level evaluations (Uganda, Nicaragua, Moldova) and was adjusted following comments from Reference Group (RG) members based on an earlier version that was included in the first draft of the Inception Report. The aim to link to evaluation results with the intervention logics assumes that evaluations do indeed focus on the Results and Impacts as defined in the intervention logics. However, this assumption should take into account that if linkages are not found, this might well be caused by the fact that the evaluation itself didn’t look for these specific Results and Impacts, while in reality the relevant societal results might still be created by the interventions. Quite some results could be linked to different ‘results’ and ‘impacts’ as defined in the same intervention logic (especially in the case of the DCI), and some results were linked to ‘results’ and ‘impacts’ of different intervention logics. When counting the number of results in the evidence as presented below we will make use of the number of unique results. This means each result is only counted once, although it is linked to several ‘results’ and ‘impacts’ of one intervention logic, or to different ‘results’ and/or ‘impacts’ of different intervention logics. Once completed, the grids were processed into an Excel spreadsheet to permit further analysis (e.g. the nature of the results) and to help the team aggregate the results into different groups by means of specific filters (e.g. displaying all the results in relation to one of the expected impacts of the ENPI). This Excel document is a useful data-set that might also be used for other purposes, and hence will be made available to the RG members as a CD-ROM annexed to the final report of this study.

1.2 The Overall Picture

Out of the 57 evaluation reports examined, a total of 503 results were identified, of which 263 unique results. Of these 503 results, 356 were linked to one or more ‘result’ and ‘impact’ areas of one or more intervention logics, and 147 were not linked to any intervention logic. The following table displays the number of results that could be related to each of the legal instruments.

5 The word results is used in this chapter both in terms of the messages extracted from the evaluation reports and in

terms of the Results in the intervention logics. To avoid confusion the Results and Impacts from the intervention logics will be spelt with capitals.



Table 3: Number of results linked to expected results of the intervention logics

Number of results

Number of unique results in relation to

the legal instruments

Development cooperation instrument (DCI) 231 147 European instrument for the promotion of democracy and human rights (EIDHR)

29 28

European Neighbourhood and Partnership Instrument (ENPI) 74 74

Instrument for cooperation with industrialised and other high-income countries (ICI)

4 (all from the regional ASEAN evaluation) 4

The Instrument for Stability (IfS) 5 (two in the civil society channel evaluation, one in

the Liberia evaluation) 5

Instrument for Nuclear Safety Cooperation (INSC)

13 5

Total number of results linked to an instrument 356 2636

Total number of results which could not be related to any of the instruments

147 (of which 112 in EDF

countries)

Total number of results 503 263 A few conclusions can be drawn from this overview: • Insufficient ex-post evaluation results in relation to what has been achieved through

operations financed under the ICI, IfS and INSC were identified to allow for any solid analysis.

• Around 60 percent of the evaluation results for the DCI can be related to more than one Result of the DCI intervention logic (in several cases DCI results were linked to more than two Results of the intervention logic). One of the reasons to explain this pattern is the fact that the DCI regulation contains a higher quantity of expected results compared to other legal instruments, including disaggregated results for five regions and six specific thematic programmes.

• The EDF countries are over-represented in the results that cannot be linked to any of the instruments. Those results found in geographic evaluations covering EDF countries which could be linked were very largely related to the thematic Results of the DCI intervention logic, which covers all developing countries. Furthermore, in some instances EDF countries’ results were linked to the IfS and the EIDHR legal instruments.

1.3 Further Analysis of the Overall Data Set

The following table presents an overview of the distribution of all unique ex-post results in relation to a total of 36 sectors (or cooperation areas7) to which they were related by the study team members. The table also indicates to which of the legal instruments the

6 The sum of the number of results linked to instruments is high since several results are linked to more than one legal

instrument. 7 Whereas there is no agreed definition of a ‘sector’ particularly ‘EU visibility’, ‘Policy Coherence for Development’

and ‘Ownership’ are more often regarded as cooperation areas.



sector results could be related, either at the level of expected results or impact or both. A few of the sector labels as used by the reviewers were merged for this overview, given that their purpose in this analysis is primarily to describe the areas or fields in which the various ex-post results as identified occurred.

Table 4: Sectoral distribution of identified results by instrument

Sector DCI EIDHR ENPI ICI IfS INSC No links Total8 Unique

total9 1 Economic sector &

trade 15 - 16 1 - - 11 43 41

2 Governance & decentralisation

14 4 1 - 1 1 9 30 26

3 Education 19 - 2 - - - 6 27 27 4 Civil society 8 12 2 - - - 2 24 22 5 Environment and

NRM10 14 - 2 - - - 8 24 24

6 Regional integration 6 - 2 2 - - 13 23 21 7 Rural development 8 - 4 - - - 11 23 22 8 Multiple issues/

sectors 11 3 10 - 1 - 1 26 16

9 Transport & communication

1 - - - - - 21 22 22

10 Water & sanitation 5 - 4 - - - 8 17 16 11 Democracy & Human

Rights 2 9 3 2 16 14

12 Energy and nuclear safety

4 - 7 - - 4 - 15 10

13 Health 8 - - - - - 7 15 15 14 Social sector 4 - 7 - - - 2 13 12 15 Food security 7 - 2 - - - 2 11 9 16 Poverty reduction 4 - 2 - - - 5 11 10 17 Private sector - - 6 - - - 4 10 10 18 Peace & security 1 - 1 - 2 - 5 9 8 19 Gender 4 - - - - - 3 7 7 20 Infrastructure - - - - - - 7 7 7 21 Statistical

programmes and projects

1 - - - - - 5 6 6

22 GBS 1 - - - - - 4 5 5 23 Institutional, legal &

admin. reform - - 2 - - - 3 5 5

24 Capacity building 4 - - - - - - 4 4 25 Alternative

development 4 - - - - - - 4 4

26 Natural disasters - - - - - - 3 3 3 27 Policy Coherence for

Development (PCD) 1 - - - - - 2 3 3

28 SME 2 - - - - - - 2 2 29 Land reform - - - - 1 - 1 2 1 30 Culture 1 - - - 1 1 31 Drugs trafficking - - 1 - - - - 1 1 32 EU visibility - - - 1 - - - 1 1 33 Fisheries 1 - - - - - - 1 1 34 Ownership 1 - - - - - - 1 1 35 Reducing disparities

between member states

1 - - - - - - 1 1

36 Refugees 1 - - - 1 1

The last 9 sectors only appear once or very few times and only in some specific evaluations, the evaluations about ASEAN and Namibia, Mauritius and the Seychelles are the main contributors of these extra categories.

8 This total includes the same results that were linked to different instruments. 9 This total gives the unique number of results per sector. 10 NRM: natural resource management



The results in the areas of ‘infrastructure’ and ‘natural disasters’ had no link to any of the intervention logics. But also ‘transport and communication’ showed only one link to DCI, out of 22 results, and also in the case of peace and security, the number of results with no link to the intervention logics outweights those with a link. As was stressed in the Inception Report, the above distribution of the results as identified in relation to the six legal instruments and the sectors as indicated, merely presents an overview of the available evidence in evaluation reports in relation to the EC’s past performance in these areas. This body of empirical evidence should not be seen as direct data on the performance of the six legal instruments and the degree to which their objectives have been realised. However, the practicalities of the EU policy processes in which the legal instruments are to be reviewed and revised before the results of their implementation are fully known11 means that the database as put together here can be a useful tool in this review and revision process. The subsequent sections go into the results as identified for each of the six legal instruments, the areas of the intervention logics to which these results relate and the degree to which these results can be judged as successful. Answers are also formulated for the two study questions agreed for Phase 3 (SQ5 and SQ6). For each of the instruments, some conclusions and lessons learned which may inform the review process are subsequently identified on the basis of the evaluation results.

1.4 The Development Cooperation Instrument (DCI)

1.4.1 Introductory Remarks

The findings gathered in the previous two phases of this study in relation to the Development Cooperation Instrument all point to this instrument leaving a particularly large ‘footprint’ in EU External Action in several respects, including: 1. Its long negotiation period and subsequent differences of views between the

European Parliament and the European Commission, as reflected in the 2009 Mid Term Review;

2. Its broad geographic and thematic scope, each with specific objectives all of which required to contribute to an ambitious global impact objective;

3. Its substantial financial envelope, hence requiring significant investment in evaluation to analyse the results achieved in all regions, including ACP regions covered by DCI thematic interventions.

A total of 231 results (roughly 60% of all results, of which 147 are unique results and 22 unique results are linked to expected impacts only12) can be related to the expected results and impacts of the DCI Intervention Logic. These results are found in all three types of evaluations that this study has looked into: country-level (23 out of 34), regional (4 out of 9) and thematic (10 out of 13). At the country level, 13 ACP country-level evaluations (out of 34) were found to contain 31 unique results which could be related to some or more of the thematic programmes of the DCI , because explicit reference was

11 It is noted that for some areas, e.g. regional integration, it is most often not realistic to expect results and impacts to

occur within the full duration of an EU budget cycle. 12 These 22 evaluation results could be linked to the impact level of the DCI intervention logic, but were not linked to

any of the result levels of the DCI intervention logic.



made to these thematic programmes or thematic instruments pre-dating the DCI with comparable scope and focus. As the 2009 Mid-Term review acknowledged that spending in thematic programmes was progressing at a slower pace compared to the geographic programme, the additional EDF results are relevant and useful to find out about the EC‘s performance in these thematic areas. Given this large volume of results, the following sub-section will illustrate the geographic and thematic distribution of these results in relation to the faithful intervention logic of the DCI as developed in phase 1 of this study. It will subsequently describe and analyse the evidence gathered as a basis for identifying overall trends and lessons.

1.4.2 Response to SQ13 5: What are the overal l resul ts and e f f e c t s per t inent to the instruments ident i f i ed in the evaluat ions?

Descriptive / Factual Evidence on Results and Effects Identified As identified in the ex-ante analysis done of the DCI, one key characteristic of the regulation is the definition of specific objectives for five geographic regions, as well as specific objectives for six thematic programmes which are implemented in these and other regions of developing countries. The following table shows the spread of the results in relation to the expected results for the geographic programmes as well as for the thematic programmes.

Table 5: Number of ex-post evaluation results related to the expected results of the DCI Intervention Logic

Expected Result identified in the DCI Intervention Logic

Number of unique results Number of results

R1 Latin America 4114 49 R1 Latin America 7 7 R1.1 Social cohesion 2 3 R1.2 Regional integration 3 4 R1.3 Good governance 5 5 R1.4 Higher education 2 3 R1.5 Sustainable development 26 27 R2 Asia 42 50 R 2 Asia 3 3 R2.1 MDGs (health/education) 9 9 R2.2 Governance 11 11 R2.3 Control of epidemics 0 0 R2.4 Sustainable development 26 26 R2.5 Drugs trafficking 1 1 R3 Central Asia 5 5 R3 Central Asia 1 1 R3.1 Constitutional reforms 1 1 R3.2 Market economy/WTO 3 3 R3.3 Border management 0 0 R3.4 Drugs trafficking 0 0 R3.5 Hiv/aids 0 0 R3.6 Regional cooperation 0 0 R4 Middle East 3 3 R 4 Middle East 1 1

13 SQ: Study Question 14 The (sub)totals in the unique results column are not neccesarily the sum of the sublevels, since the unique results are

counted, and some results refer to different sublevels of the DCI intervention logic.



R4.1 Social cohesion 0 0 R4.2 Economic diversification/WTO 1 1 R4.3 Regional cooperation 0 0 R4.4 Implementation international law 0 0 R4.5 Governance, fragile states 1 1 R5 South-Africa 1 1 R5.1 Democratic society 0 0 R5.2 Adjustment in TDCA 0 0 R5.3 Fight against poverty 0 0 R5.4 HIV/Aids 0 0

R6 A thematic programme shall be subsidiary to programmes 9 9

R7 Investing in people 34 36 R8 Environment 23 23 R9 NSAs and local authorities 18 18 R10 Food Security 11 11 R11 Migration and asylum 3 3 R12 ACP Sugar protocol 1 1 TOTAL 147 20915 The below graph visualises the rather uneven distribution of the number of unique results that have been identified ex-post in relation to those in the intervention logic:

Figure 1: Number of unique results linked to the main expected Results categories of the DCI Intervention Logic16

The graph indicates that the large majority of results found in evaluations can be related to two of the five regions (R1 Latin America and R2 Asia), while only few results can be found in evaluations that can be related to R3 Central Asia, R4 the Middle East and R5 South-Africa. For the Middle East and South-Africa the available results that can be

15 This total does not include the evaluation results that were linked to the expected impacts of the DCI intervention

logic, if those are include too, the total number of evaluation results linked to both expected results and impacts of the DCI intervention logic would be 231.

16 Please see the previous table for explanation of the abbreviations used on the X-axis.



related to their expected Results are so few that little can be said about the degree to which these have been taken forward. Findings that can be related to DCI thematic instruments mainly relate to R7 ‘investing in people’ (social sectors) and R8 ‘environment, sustainable resources’. While analysing the results, it was observed that a number of the results in relation to the NSA thematic instrument (n=5) describe results achieved through NSAs which relate to the democratic process, governance and related aspects. Given this focus, the results can also be related to the EIDHR, and are analysed in further detail in the sub-section dedicated to this regulation. Findings in relation to R10 ‘Food security’, R11 ‘Migration and asylum’ and R12 ‘ACP sugar protocol’ were found to be sparse. The results that can be related to their expected Results are so few that little can be said about the degree to which these have been taken forward. Beyond the coverage of the results, as shown above, a number of additional observations can be drawn in relation to: 1. The nature of the results: are results achieved through the interventions of a positive

nature, negative, or neutral?

2. The relevance and spread of the results, in terms of their distribution and the ways in which they link to specific impacts or the DCI’s envisaged impacts: do certain areas of impact (the DCI distinguishes five of them) seem better covered than others?

3. The assumed link between the impacts achieved and the wider global impacts defined for the DCI to achieve: based on the evaluation evidence, to what extent have the results in relation to the objectives succeeded in taking further the global impacts identified?

These three criteria for further analysing and understanding the results identified are addressed in the next sub-sections. Interpreting the Nature of the Results The ex-post evaluation results can be differentiated in accordance with the nature of the results achieved which they describe. An analysis of all evaluation results which can be related to the DCI regulation results in the largest group of unique results (75 out of 147) considered as positive, in the sense that these results can be seen as effectively contributing to the achievement of the specific Results as reflected in the faithful intervention logic (notwithstanding their relative sustainability). These positive results can be related both to the geographic programmes (all regions) as well as to the different thematic programmes. They cover both productive and non-productive sectors and were achieved through interventions using various modalities. Despite this high number of positive results, the results as observed by the evaluators are frequently (estimated 1 out of 4) accompanied by critical comments in relation to global impact (unknown or unsure) and concerns in relation to sustainability. A further 52 unique results related to the DCI regulation results were judged as neutral or ‘mixed’, in the sense that they were judged by the evaluators as unsuccessful or ineffective in taking these expected results forward. In most cases these results were found to be relevant though ineffective (e.g. project-level results in terms of gender equality not translating into societal change, increases in school enrolment while the



quality of education remains too low to expect literacy rates to change), or were too small scale in terms of coverage or investment to really make a difference. In quite a few cases too, results were not achieved but this was not seen as having any significant negative implications in relation to expected DCI Results (e.g. social cohesion, poverty, economic growth). Finally, 3 unique results that related to several DCI Result areas were considered as negative in the sense that what was achieved was counterproductive to the Results as described in the DCI regulation. In these limited cases, the reason for the observed failure was related to the design and investment in otherwise justified specific means (e.g. inappropriate or insufficient investment in interventions to strengthening accountability in education), or to overpowering internal factors in the partner country (e.g. government measures against gender equality in Nicaragua). In one case, evaluators suggested that the needs of the poorest have not been met and that, while there may have been poverty reduction impacts, gaps may actually have widened as a result of EC projects. The distribution of positive, mixed and negative results also differs between the different result areas of the DCI. When leaving out those Result areas that feature too few results to be of interest for such a comparison (R4, 5, 11 and 12) see earlier in this section for the total of results for each of them), the following overview indicates this distribution:

Table 6: Positive, mixed and negative results as related to the DCI

DCI Result areas

Positive results

Mixed results

Negative results

Total results

Positive as % of

total

Mixed as % of

total

Negative as % of

total

R1.Latin America 20 18 1 39 19 25 20

R2. Asia 23 17 2 42 22 23 40

R3.Central Asia 3 2 0 5 3 3 0

Sub total – Geographic

46 37 3 86

R6. Thematic, subsidiary to

R1-R5 7 2 0 9 7 3 0

R7.Investing in people 21 11 2 34 20 15 40

R8. Environment and natural resources

10 13 0 23 10 18 0

R9.Non-State Actors and local

authorities 12 6 0 18 12 8 0

R10. Food Security 7 4 0 11 7 5 0

Sub total – Thematic 57 36 2 95

Total 103 73 5 181 100 100 100

Unique total 75 52 3 130

As shown in the table, the distribution of types of results between the areas is relatively even, while the thematic programmes seem to show slightly higher scores on average



than the geographic programmes (54% average positive results for R1-3 compared to 63% for R6-10). Numbers however, do not say everything, and the next table may help the reader interpret the overall picture of results by summarising what a representative selection of these describe. 17

17 All results are summarised for R3, R6 and R8, while a representative selection is summarised for result areas R1, R2,

R7 and R9.



Table 7: Summary table of evaluation results in relation to the DCI

Result areas Positive results Mixed results Negative results

R1.Geographic: Latin America

Life standards poor communities strengthened [36, 37, 193]; access to bursaries/higher education [44]; regional integration promoting

economic and social development [71]; public sectors supported [194, 304]; public finance management strengthened [195]; increased primary-

secondary transition rates, secondary education supported [358, 359]

Ineffective due to mix of internal/external factors [32]; labour diversification poor results [39]; low results in increasing competitiveness

[73]; low performance in cross-cutting areas [75]; no strong role in strengthening CSOs as regional agents of change [76]; project results, no

system change [113, 198, 306];

No positive movement in quality indicators some showing

deterioration [196]

R2.Geographic: Asia

EU best practice energy and biodiversity introduced in ASEAN [29]; access to social services supported [118]; decentralisation and policy

formulation supported [124, 125]; trade integration supported [139, 140, 299, 300, 374]; ten year key support to health sector Thailand [284];

good policy dialogue [285]; democratic change supported [375]

Ineffective support through training [26, 30]; insufficient resource/means allocation [110, 111]; lack of visibility and/or ownership [115, 116]; some projects improved food security, others didn’t [137]; low sustainability and withdrawal from sector [298]; gender-progress only at project level [306]

Poverty gaps might have widened due to EC projects [303]; neglect of accountability in education meant

that situation worsened [123]

R3.Geographic: Central Asia

Positive contributions EC through CSOs [84]; increases trade/investment [126]; successful interventions in support of

productive sector [219]

Devolution problems hampering sector approaches [128]; interventions too small to make a difference [218] 0

R6. Thematic, subsidiary to R1-R5

Successful support to sector reform [68]; effective policies formulated through dialogue [211]; successful use of UN as channel [296];

infrastructure reinforced [343]; support to social sectors [344, 345]

Unsustainable training interventions with poor supervision [69]; unsustainable results through micro-projects [157] 0

R7.Thematic: Investing in people

Decentralisation and policy formulation supported [124, 125]; access to justice improved [194]; integration of women in projects strengthened [223]; access to and completion of education strengthened [352, 356,

361]; EC contributed to construction and rehabilitation in post-conflict or post-emergency situations [353]; EC has joined forces to abolish school fees in countries [355]; cost-effective pro-poor support to

secondary education [358, 359]

Health sector reform slower than expected [3]; no impact of gender activities [48]; progress at skills/individual level, policies remain

unchanged [213]; outcomes of EC projects not sufficiently translated into political dialogue, low impact on policy [287]; education progress not

sufficient to achieve 2000 targets [357]

Neglect of accountability in education meant that situation worsened [123]; gender equality deteriorated despite assistance

provided [197]

R8.Thematic: Environment and natural resources

Huge and visible impact through environment budget line [49]; positive results achieved [146, 221, 285]; strengthened policy formulation

capacity and sustainable economic production [215]; environmental protection increased through interventions [216]; improved data

production and sensitisation [257]; positive and successful contribution to water supply [304]; management information systems supported

[368]

Structural weaknesses on partner government side [46]; insufficient resource/means allocation [110, 113, 127]; impact and visibility hampered through use call for proposals [111]; low effectiveness [115]; unsuccessful mainstreaming [169]; unsuccessful sensitisation [249]; unsuccessful setting up of solid waste management system [373]; progress low on increasing

quality of education and literacy [360, 365, 366]

0

R9.Thematic: Non-State Actors and local authorities

Post-war recovery supported [4]; capacity development of NSAs contributed to so that they can play their roles effectively [56, 208, 209,

234, 293]; various positive contributions by working through CSOs [84]; projects worked as bridges to facilitate quality improvement of

services [361]

Contributed to the consolidation of peace, but dialogue remains poor [1]; Devolution problems hampering sector approaches [128]; activities

limited in scope, poor dialogue CSO-government [147]; only modest progress due to persisting issues [280]; isolated improvements, no

systemic change [295, 363];

0

R10.Thematic: Food Security

Food security improved, reactive capacity increased [4, 294]; improved nutrition rates linked to infrastructure [41, 42]; strengthened production

systems [156]; strengthened co-management of regional fish stocks [214];

Unsustainable results due to low participation [40]; short-term food security support likely not resulting in sustainable results [327]; lack of

investment in communication hampering results [249] 0



While reflecting what has been covered JEU evaluations, the overview of results does confirm the different nature, scope and focus of the EC cooperation programmes in the different DCI regions, which would seem to justify the existence of the separate objectives at the Results level: • A strong focus on education and related social sectors in Latin America, with minor focus on

trade-related interventions; • Interventions in Asia predominantly focusing on the economic sector and regional integration in

that area; • A focus on the productive sector and state building in Central Asia. Looking into the mixed results, what jumps out is that evaluations covering countries in all three regions found interventions that did not make a difference due to scale and investments issues, typically projects which created ‘islands of development’ that did not have an impact on society. The same was found for different interventions funded under the thematic programmes, which predominantly use the project approach. At the same time, the positive results found – particularly in the geographic programmes – were often related to the fact that strong and sector-wide investments have been made, which seems to confirm the validity of the EC’s efforts to rationalise its cooperation and use programme-based approaches for aid delivery. The Relevance and Spread of the Results The evaluation results can also be sorted according to the sector to which they were related by the members of the study team. The below overview shows how the evaluation results achieved in different sectors can be linked to the DCI expected results.

Table 8: DCI related results per sector

Sector Number of unique evaluation results related to the DCI

1 Education 19 2 Economic sector and trade 15 3 Governance and decentralisation 14 4 Environment and natural resource management 14 5 Multiple issues/sectors 11 6 Civil society 8 7 Rural development 8 8 Health 8 9 Food security 7 10 Regional integration 6 11 Water & sanitation 5 12 Energy and nuclear safety 4 13 Social sector 4 14 Poverty reduction 4 15 Gender 4 16 Capacity building 4 17 Developpement alternative 4 18 Democracy & Human Rights 2 19 Planning, Monitoring and Evaluation 2 20 Transport & communication 1 21 Peace and security 1 22 Statistical programmes and projects 1 23 General Budget Support 1 24 Policy Coherence for Development 1 25 Culture 1 26 Fisheries 1 27 Ownership 1 28 Reducing disparities between member states 1 29 Refugees 1

Total 153



The high amount of results in relation to the first sectors listed in the Table above can partly be explained by the fact that these have been covered by specific thematic evaluations in addition to country and regional evaluations (education, civil society, trade related assistance, rural development, health). The distribution does show that during the current negotiations on revising and/or reforming the legal instruments, there is not a strong evaluation evidence base for some sectors. Out of the 147 unique evaluation results that are linked to expected Results of the DCI intervention logic, 44 results are from evaluations in EDF countries or regions. Those results can only be related to the expected Results of the DCI thematic programmes.

Table 9: Results from EDF countries and their links to DCI Result areas

Result areas DCI Related results from EDF countries/region

R6. Thematic, subsidiary to R1-R5– 8

R7. Thematic: Investing in people 4

R8. Thematic: Environment and natural resources 9

R9. Thematic: Non-State Actors and local authorities 10

R10. Thematic: Food Security 8

R12. Thematic: ACP sugar protocol 1

Relating the Evaluation Results to the Expected Impacts of the DCI As was found in the previous two phases of the study, the DCI regulation points out that the expected results of the geographic and thematic programmes are to contribute to the expected Impacts, but does not describe which objective is expected to contribute to which of the expected (global) impacts. For analysing the ex-post evaluation results in relation to this instrument, it is nevertheless important to attempt to find out which of the results can be considered as being “relatable” to which of the expected Impacts. The following diagram represents an excerpt from the faithful intervention logic, with the codes of the various expected results mentioned after the expected impact to which (based on an analysis of the DCI regulation) these objectives can be considered as having the most logical link.



Figure 2: Excerpt from the faithful DCI intervention logic

ImpactsResults

I1: General (Art 2): R1, R1.5, R2, R3, R6, R7, R8, R9, R10, R11, R12

I1.1. Support to democracy, the rule of law, human rights and fundamental freedoms, good governance, gender equality and related international law: R1.1, R1.3, R2.1, R2.2, R7, R9, R11

I1.2. Foster sustainable development [4 sub-aspects] of countries and regions, particularly the most disadvantaged; R1.1, R1.2, R1.3, R1.4, R1.5, R2.1, R2.2, R2.4, R2.5, R3.1, R3.2, R7, R8, R10, R11

I1.3. Encourage smooth and gradual integration into the world economy; R1.2, R1.5, R2.4, R3.2

I1.4. Promote sustainablemanagement of global natural resources for sustainable development [3 sub-aspects] ; R1.5, R2.2, R2.4, R8

I1.5. Strengthen relation between the EC and Countries/ regions.No direct links

Latin America: (Art 6)R1.1 Social cohesion promotedR1.2 Greater regional integration encouragedR1.3 Reinforcement of good governance and public institutions, protection of human rights supportedR1.4 Common EU-Latin American higher education area supported R1.5 Sustainable development promoted

Asia: (Art 7)R2.1 MDGs (Health/education) through policy dialogue promoted R2.2 Governance in particular in fragile states addressed R2.3 Contribute to control of epidemicsR2.4 Sustainable development promoted R2.5 Production, use and drugs trafficking combated

Central Asia: (Art 8)R3.1 constitutional reforms and administrative/regulatory approximation promotedR3.2 Development market economy and integrating in WTO (addressing social aspects) promotedR3.3 Efficient border management and cross-border cooperation supportedR3.4 Production, use and drugs trafficking combatedR3.5 HIV/Aids pandemic addressed R3.6 Regional cooperation, dialogue & integration promoted

Middle East (Art 9): R4.1 Social cohesion encouragedR4.2 Economic diversification, market economy, integration in WTO promotedR4.3 Regional cooperation, dialogue & integration promotedR4.4 Conclusion of international agreements and effective implementation of international law, including UN resolutions R4.5 Governance issues in particular in fragile states addressed

South-Africa:(Art 10):R5.1 Democratic society, good governance, rule of law, regional/continental stability + integration consolidated R5.2 Adjustment in context of TDCA and other regional trade arrangements supported R5.3 Fight against poverty, inequality and exclusion supported R5.4 HIV/Aids pandemic addressed

Art 11: “A thematic programme shall be subsidiary to programmesreferred to in Art 5-10.”“actions undertaken through thematic programmes shall add value to and be additional to, and coherent with, actions funded under geographic programmes.”

Specific objectives “Investing in people” (Art 12)Specific objectives “Environment and sustainable management of natural resources including energy” (Art 13)Specific objectives “Non-State Actors and local authorities in development” (Art 14)

Specific objectives “Food Security” (Art 15)

Specific objectives “Migration and Asylum” (Art 16)

Specific objectives “ACP Sugar Protocol” (Art 17)

R1

R2

R3

R4

R5

R6

R7R8R9

R10R11R12

The below table gives an overview of the number of references to each of the expected impacts of the DCI intervention logic.

Table 10: Evaluation results as related to the DCI expected impact

Expected Impact in DCI intervention logic: Number of unique

results related to this Impact

I1: Article 2: general provisions in relation to DCI impact 75

I1.1: Support to democracy, the rule of law, human rights and fundamental freedoms, good governance, gender equality and related international law 19

I1.2: Foster sustainable development of countries and regions, particularly the most disadvantaged 45

I1.3: Encourage smooth and gradual integration into the world economy 11 I1.4: Promote sustainable management of global natural resources for sustainable development 21

I1.5: Strengthen relation between the EC and countries/regions 0



The following graph displays the distribution of results in relation to the Impacts as a percentage of the total number of unique results identified (171):

Figure 3: Relative distribution of evaluation results to impact DCI (% of total results)

Based on this overview, it can be concluded that there is generally sufficient evaluation evidence available which can be related to the expected Impacts, while the evidence in relation to the third specific Impact (integration in the world economy) is relatively narrow, and fully absent for the last specific Impact (strengthened relations). It might be that this last Impact is more seen as a ‘by-product’ or transversal impact, but it can be observed here that available ex-post evaluation results cannot be directly put to use to find out whether relations between the EC and countries/regions have been strengthened as a result of interventions financed under the DCI. Conclusions to be Drawn from the Evidence The following conclusions emerge from the description of the evidence found: 1. Ample ex-post resul ts from EC evaluations can be related to the expected Results identified in

the geographic and thematic instruments of the DCI. The distribution of these results between the 11 different lists of expected results as defined by the regulation is uneven. While suf f i c i ent resul ts can be found for most , little can be said about the achievement of results identified for the Middle East (R4), South-Africa (R5), Migration and Asylum (R11), and the ACP Sugar protocol (R12).

2. While most of the 147 results related to the DCI regulation results can be judged as positive, a large group of results is mixed in the sense that they were not found to effectively contribute to the achievement of the expected Results as identified in the DCI. Roughly one third of these results were not ‘failures’ per se in the sense of being ineffective, but were for instance too insignificant in size or scope achieved an a scale that would not lead to noticeable achievement of the result at a country or regional level (i.e. so-called ‘islands of development’). Another similar chunk of findings pointed to results that were still too ‘young’ with their subsequent impact and sustainability not yet being guaranteed or questioned by the evaluators.

3. Looking at the linkages of these results with the Impacts level, it is shown that the first four of the five areas are well covered by the various Result areas, but that few results contribute directly



to the fifth Impact of ‘strengthened relations between the EC and country/regions’ (I1.5). Moreover, the thematic migration and sugar protocol instruments are not clearly linked with the Impact level, and hence their relevance in relation to the global impact objective of the DCI may be questioned.

4. One other conclusion we can draw confirms an observation made earlier in this study, namely that the DCI can be described as a not fully logical match between its high number of expected results and instruments and its relatively tight vision as to the Global Impact (namely poverty eradication in the context of sustainable development).18

5. As far as Global Impact is concerned, the wide majority of evaluations reviewed in this study that relate to the DCI do not succeed in finding evidence on whether the results and impacts achieved in fact contribute to the global impact goal of poverty reduction in the context of sustainable development, which is an important ‘evaluation gap’ that remains.

1.4.3 Response to SQ6: On the basis o f resul ts ident i f i ed ( in Q5) are there any overal l t rends or l essons to l earn to be ident i f i ed from the evaluat ions and that might be pert inent to the instruments?

Identification of Any Overall Trends or Lessons Learnt Given the continuity in focus between the DCI and most of the old budget lines that were merged in it, much can be learnt of the effectiveness of EC cooperation in the regions and thematic areas covered by the regulation from recently completed ex-post evaluations. Although most results in fact relate to interventions of the predecessor(s) of the DCI (especially in the case of the thematic programmes), one overall lesson is that the Result areas as mentioned in the intervention logic are sufficiently realistic and concern areas in which it would seem that EC cooperation can make a difference.19 The evaluations also point that beyond the Results level, the higher Impact level (and particularly the Global Impact) is much less within reach, as shown by the much more tentative language found in the reports. A related lesson which emerges from the material is the unclear linkage between the Impacts level on the one hand and the Global Impacts level on the other.20 Whereas the DCI regulation is clear on the aims for both levels in the intervention logic, it cannot be automatically assumed that Impacts achieved contribute to the Global Impact goal of poverty reduction in the context of sustainable development. Most EC evaluations do not provide empirical evidence on the contribution made to poverty reduction but often remain limited to assessing the poverty focus of Country or Regional Strategy Papers.21 The analysis therefore often remains restricted to the measurement of results, which are subsequently related to the impact level (most often in terms of the prospects for sustainable contribution to achieving the impact as formulated). In the case of the DCI, the impact which is achieved in relation to ‘smooth and gradual integration into the world economy’ (I1.3) is implicitly assumed as contributing to poverty eradication, whereas this might just as well actually exacerbate poverty levels in the short-term. Given the Lisbon Treaty’s legal requirement for EC development cooperation to contribute to poverty eradication, one lesson might be that a successor

18 In earlier phases of this study, it was noted that different actors see different ‘logics’ in what a regulation should or shouldn’t do,

and hence either value the flexibility offered or greatly emphasise on the global impact level as a way of disciplining and bringing order into the complex of development interventions that are designed and implemented.

19 With exception of those areas where there is a lack of evaluation evidence (R4, R5, R11, R12) for which this is unknown. 20 The same can be said for the links between the results and impacts levels and is mostly due to the format/approach taken in the

formulation of the regulation: the faithful intervention logic supposes that all result areas contribute jointly to the achievement of the impact level, but the regulation does not specify which result area is expected to contribute to what impact.

21 This was also recently observed in the Feasibility Study for the evaluation of the EU Consensus for Development, which included an inventory of evaluation results.



regulation to the DCI (or an adaptation thereof) includes more targeted review clauses to ensure a more thorough and targeted ex-post evaluation at the level of the Global Impact. Looking at the evaluation results, another lesson is that most positive results have been realised when the objectives of the interventions coincided or were compatible with the partner government or region’s own priorities. Mixed results are particularly found when the objectives of an EC intervention, regardless of the aid delivery method used, are inconsistent with government policies (e.g. a project to promote sound and environmental friendly management of natural resources while the government prioritises short-term economic development). While the ownership principle is widely accepted and acknowledged in the DCI regulation, the rich diversity of areas including more normative ones (e.g. human rights and democracy) however shows that EC development cooperation is not just guided by this principle alone. As a result, some EC interventions are challenged by having to ‘swim against the current’ of government priorities. In those cases, mixed results have to be accepted as a result of the choice made to take higher risks. Related to the previous point, it was noted in the second phase of this study that even though the DCI allows for 13 different means of financing the cooperation activities (see Art. 25), the 2009 EC Mid Term Review noted that the thematic programmes have mostly if in some cases not exclusively relied on calls for proposals/tender procedures. Given the many procedural delays encountered in the spending of the budgets, as well as the large time investments required for particularly the ‘softer’ areas of intervention (non-state actors, environmental sustainability, investing in people), one lesson to be drawn is that there should be further analysis into how more diversified use of available aid modalities may be promoted. Conclusions Based on the analysis in this third phase, it can be concluded that the EC is roughly on track but not yet close to fully realising the expected results as defined for the geographic and thematic programmes under the DCI. Linked to this finding, the results observed differ in whether and to what extent they contribute to the Impacts as defined, while the contribution and realisation of the DCI’s global objective remains nebulous and requires for more research investments for a variety of reasons including financial accountability. Notwithstanding the young age of the DCI regulation (five years as this report is in preparation), it can be expected that a successor regulation will probably differ quite strongly, for instance through new policy trends looking into blending grants and loans, as well as possibly a bigger role for the European Investment Bank in promoting international development. Looking into synergies between ODA and non-ODA sources hence could be a means to a focus on poverty reduction (as required by Lisbon) as well as the possibility for non-development financing in developing regions (stated as desired in the 2009 Mid-Term Review). A third and last conclusion is that a new regulation would benefit from much clearer linkages (and in fact less of a gap) between the Global Impact, Impacts, and the very detailed and disaggregated expected Results. Specific evaluation results or longer-term research investments could be reserved under a successor regulation to commission specific empirical studies at the EC’s contribution to global impact (individually or in consort with other donors).



1.5 The European Neighbourhood Policy Instrument (ENPI)

1.5.1 Response to SQ5: Overal l resul ts and e f f e c t s per t inent to the ENPI instrument ident i f i ed in the evaluat ions

a) Descriptive / Factual Evidence of Results and Effects Identified The analysis of the evaluation reports highlighted a number of observations.

Table 11: Overview of unique results linked to the ENPI

Number Proportion

TOTAL Results relevant to ENPI 74 20%

Type of source

Country evaluations 40 54%

Regional evaluations 13 18%

Thematic evaluations 21 28%

Types of results

Positive 39 53%

Negative 0 0%

Mixed 35 47%

Sectoral focus

Economic and private sector 14 19%

Trade 8 11%

Social sector and civil society 9 12%

Energy and nuclear safety 7 9%

Other sectors 36 49%

Regional distribution

ENPI South 17 23%

ENPI East 36 49%

Worldwide 21 28%

Comparison with the IL

R1 48 65% R2 23 31% NA 3 4%

Description of the table i) Type of Sources:

Results pertinent to the ENPI instrument have been identified in only 15 evaluations out of the 57 evaluations analysed. In these 15 evaluations, which were mostly country evaluations, 74 results have been identified in total (of which 3 have only been linked to the Impact level of the ENPI regulation).

ii) Type of Results:

Half of the results are positive, none are negative, but there is still 47% of results that have been categorised as mixed. Among the positive results (i.e. results that can be seen as effectively contributing to the achievement of the specific results as reflected in the faithful intervention logic), 13 results were associated with EC interventions in the economic, private and trade sectors, and 6 results were associated with EC interventions in civil society and social sector. The remaining results were mixed (i.e. results judged by the evaluators as unsuccessful or ineffective in taking these expected results



forward). None of the results identified is negative, in the sense that what was achieved was counterproductive to the results as described in the regulation. iii) Sectoral Focus

A significant majority of results identified are related to the EC’s interventions in the economic, private and trade sectors (in total 30% of the results). The graph below shows the sectoral distribution of the number of results that have been identified ex-post in relation to those in the intervention logic:

Figure 4: Sectoral distribution of evaluation results for the ENPI

iv) Regional Distribution Half of the results concerned the region ENPI East, while only 23% of the results identified in the evaluations were related to Southern countries of the ENPI region. The rest of the results (28%) have been identified in thematic worldwide evaluations, and therefore only partly refer to the ENPI region. v) Comparison with the Intervention Logic

When comparing with the intervention logic (IL), it appears that overall the distribution of identified results across the IL Results is quite balanced. Most of the results identified in the evaluation reports (48 results out of 74) were associated with the Result number one in the IL (R1: “partner countries efforts aimed at promoting good governance and equitable social and economic development are supported”). 31% of the results identified were associated with the Result number two (R2), pertaining to the implementation of partnership cooperation agreements and other existing and future agreements. The following diagram is an extract from the faithful intervention logic focusing on outputs, expected results and impacts, that have been based on an analysis of the ENPI regulation.



Figure 5: Extract from the ENPI intervention logic

Based on the ENPI regulation, the intervention logic identifies six Outputs. The first one of these Outputs is “promotion of trade and economic integration”, and it is also the Output to which most of the results identified are associated (EC’s interventions in the economic, private and trade sectors). However, it is not possible to associate a significant number of identified results to the other Outputs listed in the intervention logic. However, when looking at the distribution of positive results among Result areas (R1 and R2), it appears that this distribution is quite uneven. As shown in the table, 30 results out of the 39 positive results identified are related to the Result number 1 (R1) from the IL, while only 9 positive results are linked to R2. The following table shows the distribution of types of results between results areas and sectors:



Table 12: Evaluation results for each of the ENPI intervention logic Results

Result areas Sectors

Number of

positive results

by sector

Result numbers

(according to the

database)

Sectors

Number of

mixed results

by sector

Result numbers

(according to the

database)

R1: Partner countries

efforts aimed at promoting

good governance

and equitable

social and economic

development are

supported

Civil society 2 161, 308 Cross-cutting issues 1 120

Cross-cutting issues 1 162 Democracy &

Human Rights 2 164, 309

Democracy & Human Rights 1 20 Economic sector 1 159

Economic sector 4 129, 160, 163, 261 Education 2 15, 16

Energy 2 7, 17 Energy 2 18, 350

Food security 1 14 Environment 2 173, 349 Institutional legal and administrative reform

1 265 Governance 1 170

Peace & Security 1 85 Private sector 2 10, 131

Poverty reduction 2 119, 132 Social sector 2 19, 346

Private sector 4 9, 12, 260, 262 Trade 1 177

Rural development 3 11, 13, 172 Various sector 1 84

Social sector 3 263, 264, 369 Water & sanitation 1 305

Trade 2 24, 176

Various sectors 1 118

Water & sanitation 2 133, 134

Total number of positive results 30 Total number of

mixed results 18

R2: Partnership

and cooperation agreements

other existing or

future agreements

are implemented

Regional Integration 1 175 Energy 1 114

Economic sector 1 347 Cross-cutting issues 1 306

Institutional legal and administrative reform

1 351 Drug-trafficking 1 174

Trade 2 22, 219 Economic sector 2 310, 311

Various sectors 2 98, 296 Food security 1 222

Water & sanitation 1 304 Regional Integration 1 21

Social sector 1 220 Rural development 1 218

Social sector 1 312

Trade 3 23, 130, 348

Various sectors 2 99, 297

Total number of positive results 7

Total number of mixed results 16



Analysis of Positive Results When looking at the positive results identified that are pertinent to the ENPI instrument, it appears that EC interventions have made a positive contribution to the development or trade relationships between the EU and the ENPI partner countries, and in supporting the development of the private sector. More specifically, the following trends have been identified: • The EC appears to have made a positive contribution to the development of foreign trade in

ENPI countries [results 22, 24, 261]. EC interventions (TACIS) have contributed significantly to enhancing the capacity of government institutions, which in some cases has manifested itself in progress towards reaching international agreements such as the WTO and to enable governments to address key areas of PCA implementation. For example, the EC provided TA to the government of Armenia in the negotiations for its accession to the WTO and in implementing post-accession commitments.

• EC interventions (TACIS and MEDA) interventions appear to have contributed to establishing

an institutional and legal framework conducive to the development of the private sector in a number of countries [results 9, 10, 163]. EC interventions (MEDA II) interventions achieved positive results in terms of institutional strengthening and introduction of good management practices with benefits for economic operators and users of public services. They faced major difficulties in contributing to adoption and enforcement of sensitive core reforms.

• EC interventions achieved positive results in strengthening the capacities of small and medium

enterprises as in Armenia, as well as in improving the general environment for SME development as in Moldova [results 9, 176]. It has been noted that EC support (TACIS) seems to have been more successful in relation to policy advice or large scale capacity building programmes, rather than SME development initiatives that require long-term commitments [result 261]. There is no mention of the need or lack of revolving funds in the ENPI region, although this was considered as a potential constraint in the analysis of programming and subsequent documents.

This forms the bulk of results that can be identified clearly from the evaluation. Besides these, the following contributions have also been described: Some positive results related to civil society have also been identified as well as in the water and social sectors. It has been noted that EC interventions (MEDA) resulted in improved professionalism of Civil Society Organisations [result 308]. • In the water sector, EC interventions (MEDA) also contributed to clear improvements in the

management of the institutions responsible for the distribution and quality of water, but their effect on the sector policies remains limited. In particular, the EC was successful in Jordan in promoting sustainable water management as well as in contributing to a forward-looking policy reform targeted at waste water reuse [results 161, 133].

• In the area of training and of institutional development, EC interventions (TACIS) have

been effective in capacity development of institutions that will facilitate replication of efforts [result 264].



• As for health care and the promotion of democracy through social NGOs, EC interventions (TACIS) have proved to act as a catalyst in policy reform, but the project modality was adopted for the main legal and sector reforms. In this regard, the collaboration with International Financial Institutions has been effective in preparing the ground for introducing necessary reforms in the sector [result 263].

No result related to emergency aid has been identified, which can be explained by the fact that our analysis did not cover evaluation reports for Georgia and the occupied Palestinian territory. Analysis of Mixed Results In most cases, mixed results correspond to cases where EC interventions were unsustainable [results 15, 84, 112, 177, 164, 312], in particular because of the scarcity of financial resources available [results 20, 15]. Ex post results have also been judged by evaluators as ineffective or with a limited sustainability when they were focused (geographically for example [results 218, 219, 220]) or too specific. The lack of policy framework (as in the education sector in Armenia, [result 15]) or an overall unstructured approach of the EC (e.g. in the governance sector in Moldova [result 170]), was another reason for limited ex post results. In quite a few cases too, ex post results could not clearly be linked by the evaluators to EC interventions [e.g. result 222, 349] or EC support in one sector could not be linked to any improvement in this sector [result 346, 347]. b) Conclusions Drawn from this Evidence 1. Overall, we have found that half of the results identified pertinent to the ENPI instrument are

positive. These positive results appear to be concentrated in the economic, private, trade and social sectors which are part of R1 in the intervention logic. However, from the evidence collected from the evaluation reports there is no indication that the other Outputs (and Results) of the IL have all been achieved, in particular as concerns the facilitation of mobility, management of migration and people to people exchanges. Their achievement can indeed not be tracked through the evaluations. Furthermore, the evaluation did not show much evidence on the achievement of R2. As a consequence, based on the IL, it is more likely the results contribute indirectly to Impact number one of the IL (I1: “Commitments to EU values as liberty, democracy, Human Rights, fundamental freedoms and rule of law are promoted”) rather than to Impact I2 (“Enhanced cooperation and progressive economic integration is promoted”).

2. While most of the 74 results identified can be judged as positive, a large group of results is

mixed in the sense that they were found to be ineffective in contributing to the achievement of expected results as identified in the intervention logic. In many cases, these results were too small scale (in particular financially or geographically) and would not lead to noticeable achievement of the result at a country or regional level, or their sustainability was not guaranteed.

3. Looking at the linkages of these results with the Impacts level, few results contribute directly to

the first area (I1) of “Commitments to EU values as liberty, democracy, Human Rights, fundamental freedoms and rule of law are promoted (Art. 1)”. Moreover, Outputs related to migration, mobility and people to people exchanges are not clearly linked with the level of expected Results in the IL.



4. Finally, our approach is based on the analysis of ex post results, which may make it more

difficult to identify whether specific outputs that do not appear as such in the Results on the IL (such as the facilitation of mobility, management of migration and people to people exchanges), have been achieved.

1.5.2 Response to SQ6: Overal l trends or l essons to l earn ident i f i ed from the evaluat ions that are per t inent to the ENPI instrument

a) Identification of Any Overall Trends or Lessons Learnt Even though the ENPI instrument has only been in existence for a short period of time, it has already achieved many expected results of the intervention logic, although it should be noted that many results in fact relate to interventions of the predecessors of the ENPI (TACIS, MEDA). However, the Impact level, and particularly the intermediate Impact 1, of the IL is less likely to be reached. A related lesson which emerges is the unclear linkage between the Outputs, expected Results and Impacts levels in the IL. b) Conclusions 1. On sectoral results: From the analysis above, a theme where positive interventions have been

identified emerges: trade and private sector development. This is also the theme at the top of the priorities as described in the IL. This indicates convergence between programming and implementation for this result. However this coherence cannot be demonstrated for the other priorities (outputs) in the Intervention Logic. As regards the support to civil society, evaluation reports show some discrepancies between ENP-East and ENP-South. In the Eastern part of the ENP region, the predecessor of the ENPI instrument has provided significant support to Civil Society in a broad range of areas such as the social and environmental sectors, in accordance with the Regulation which recognises the importance of the development of Civil Society in ensuring the long-term sustainability of economic reform [Evaluation report TACIS 2006, paragraph 270 p. 77]. On the contrary, the good results achieved in the economic and social sectors in the South have not been accompanied by appropriate reinforcement of human rights and democratisation and a strengthened participation of Civil Society in the development process [Evaluation report MEDA II 2009, C5 p. 96].

2. On the programming process: The evaluations do not always enable the tracking of the results in

such a way that they can easily be associated with a specific Impact or with a specific Output from the IL. This could indicate a lack of coherence in the programming process, a theoretical bias in the methodology of the evaluations we have reviewed, or in the approach adopted in this evaluation. Indeed, no evidence was found that all the priorities expressed in the CS were subsequently carried throughout the programming chain. And ultimately turned into specific actions from which results could be traced. It is difficult to see whether it is due to gaps in the intervention logic, to the specific approach adopted for the present survey, or to the way evaluations are conducted and present results. However, the complexity of the document flow linked to the programming (CS, AP, NIP, IP) multiplies the risk that bias in the logic appear and are not detected (and make instrument-level monitoring and evaluation rather difficult). Whereas one can conclude that coherence exists between the IL, the strategic intentions and the implementation with respect to R1, no conclusion can be made for R2. Our methodology has been focused on checking whether the expected Results have been achieved ex-post, but does not allow us to analyse the level of coherence between the design of ENPI programmes and the objectives of the intervention logic.



3. On the thematic dimension of ENPI: One of the main themes emerging from the desk study

was the important divergence between the original proposal of the Commission and the regulation adopted in fine, with respect to the thematic dimension of the ENPI. As thematic evaluations are worldwide, it is difficult to draw conclusions relevant for ENPI.

4. On aid delivery modalities: according to the review of evaluations, it appears that Budget

support is bringing good results to support comprehensive structural/ sector reforms, avoiding the gaps and “stop-go” effect of Project approach. However it requires a strong and reliable Public Finances System s. Other aid delivery mechanisms such as PA or twinning remain very effective in targeted reforms, possibly in combination with BS. The positive aspects of the introduction of BS under MEDA II have been highlighted in the evaluation report. It aimed at supporting comprehensive structural and sector reforms rather than specific sets of activities. It contributed to the formulation, implementation and acceleration of reforms that were nationally owned and considered as important by the EC. [Evaluation report MEDA II 2009, C9 p. 100]. On the other hand, the TACIS project-by-project approach presented the risk of losing momentum and beneficiary commitment during the time between one project and its successor. In addition, the stop-go nature of project-based support to the sector has in some cases resulted in unrealistic goals being set [Evaluation report TACIS 2006, paragraph 270 p. 66]

5. Whilst delivery through BS requires a strengthened Public Financial Management [Evaluation report Moldova, p.10 section 1.3, conclusion c], the support through budget support does also contribute, when successful, to strengthen improvement in PFM [Evaluation report Egypt 2010, EQ7 p. 55]. It appears to be efficient when combined with Policy dialogue and capacity building [Evaluation report Jordan, p.20 section 4.3.3].

6. On ownership: The evaluation report on TACIS underlined weak ownership as being one

consequence of project-based approach. In addition, it noted that partner States considered dialogue not to be conducted as envisaged in the Regulation. In some cases this has resulted in limited ownership of CSP and NIP. Under MEDA II, the ownership has been considered as generally strong, particularly for interventions targeted on reforms that are part of the governmental priorities. The evaluation report underlined the fact that “BS interventions were generally characterised by a higher level and greater depth of preparation and, therefore, benefited from a high degree of ownership. This modality contributed to improving the quality and continuity of the policy dialogue with the partners and to focus it on essential reforms.”

7. On cross-cutting issues: in both regions, evaluation reports [Evaluation reports MEDA II and TACIS] noted that cross-cutting issues, in particular gender and environment, have not been generally addressed and systematically integrated into EC interventions, outside the interventions targeted on them.

1.6 Instrument for Stability (IfS)

1.6.1 Response to SQ5: What are the overal l resul ts and e f f e c t s per t inent to the instruments ident i f i ed in the evaluat ions?

Descriptive / Factual Evidence on Results and Effects Identified At the level of Results only 5 observations related to the Instrument for Stability’s (IfS) Faithful Intervention Logic (FIL) were identified in the analysis of the evaluation reports. Two of these were found in the thematic evaluation on civil society [84, 85] and a further one each in the country evaluations of Namibia [191], Tanzania [273], and Liberia [334]. At the Impact level (to provide



further evidence) there were additional outcomes not specifically related to Results noted above in the Regional Evaluation of the Pacific [217] and Education [353]. The results from the civil society evaluation [84] related to R2 ”Threats to law and order, to security and safety of individuals, critical infrastructure and to public health addressed” and R4 [85] “Strengthened capacity of international, regional and sub-regional organisations, state and non-state actors [to respond to conflict and stability issues]”. Also related to R4 are the results from the Liberia [332] and Namibia [191] evaluations. Although these results and the one for Tanzania [273] also link to the impact level of – “Contribution to help preserve, establish, or re-establish conditions [for]... Community’s development and cooperation policies (In situations of crisis or emerging crisis, to contribute to stability)” which is linked to R1 on the IfS intervention logic. However, in the IfS intervention logic R1 is left blank as it was not clear what result was intended when constructing the intervention logic.22 Conclusion The findings emphasise the importance of flexibility and rapid response, in achieving results in the broad field of conflict prevention [84] or the more specific fields of demobilisation [334] or recovery [385]. The findings also emphasise the achievement of results and utility of engaging civil society for results in conflict prevention and resolution [84,85] and policy dialogue towards this ends [273].

1.6.2 Response to SQ6: On the basis o f resul ts ident i f i ed ( in Q5) are there any overal l t rends or l essons to l earn to be ident i f i ed from the evaluat ions and that might be pert inent to the instruments?

Identification of Any Overall Trends or Lessons Learnt The number of results identified in the evaluation reports is very small, which means it is not possible to see them as a firm evidence base for trends or lessons learnt. Yet it is important that the few results seem to be positive, and also very clearly align with the rationale and modus operandi of the IfS in particular its rapid, flexible response, focus on conflict resolution and its support for civil society. Conclusions No overall conclusion can be drawn on the effect of the IfS due to lack of evidence in the evaluation reports available and analysed. This lack of evidence is probably explainable by, amongst other factors, the lack of resources channeled through the IfS, compared to other financial instruments and the fact that the majority of resources are, by the very nature of the IfS, supposed to be a bridging or interim funds to be used in conjunction with other external action instruments. Yet the lack of results identified in the evaluation report should be seen against the backdrop of two on-going developments that are likely to fill this gap. Firstly that there is an specific evaluation of the Instrument of Stability that was launched from DG RELEX in December 2010, and secondly there is a thematic evaluation of conflict prevention and peacebuilding from the JEU due to report in June 2011. It seems highly probable that both these will provide a much richer source of information and insight on the impact and results of the IfS. It is therefore not appropriate to draw the conclusion from the restricted evidence above that the IfS as an instrument has limited worth or results.

22 It is often the case in constructing intervention logics that it is not apparent what specific result was desired even if other parts of

the intervention logic are complete.



1.7 Industrialised Countries Instrument (ICI)

1.7.1 Introduct ion: The analysis conducted in phase 2 of this study highlighted that the Instrument for Cooperation with Industrialised Countries and High Income Countries (ICI) takes a unique place in this group of six legal instruments in the sense that it focuses exclusively on economic, financial and technical cooperation with industrialised countries. In this brief sub-section, available evaluation results which can be related to this instrument are analysed in more detail in order to define lessons for the future. Given the interaction with the DCI instrument, from which it was ‘separated’ during the negotiations, additional lessons on what can be done in the future may be drawn in Phase 4 of this study by comparing this analysis with the one made for the DCI.

1.7.2 Response to SQ5 : What are the overal l resul ts and e f f e c t s per t inent to the instruments ident i f i ed in the evaluat ions?

Descriptive / Factual Evidence on Results and Effects Identified The analysis of the evaluation reports resulted in only four results which could be related to the ICI. These four results [numbers 25, 26, 27, 28] were all found in one single regional evaluation which assessed the EC’s cooperation with the Association of Southeast Asian Countries (ASEAN). This regional evaluation was finalised in 2009 and was conducted in parallel with country level evaluations covering the EC’s country level cooperation strategies in Thailand, Malaysia, Lao PDR, and Vietnam. Given the low number of results identified in the 57 ex-post evaluations that were relevant to the ICI instruments, the space available for analysing these results allows to display the summaries of these results in the following table:

Table 13: Ex-post evaluation results that can be related to the ICI instrument

# Sector Result description IL

link

25 Regional Integration

EC assistance to ASEAN during the assessment period has resulted in high-level outputs with regard to the support of regional economic integration and trade initiatives, and there has been tangible progress towards regional trade and economic integration. Progress towards the implementation of the ASEAN Free Trade Area, customs harmonization, standards harmonization, IP legislation, registration and awareness etc. would have been reduced without the EC support. [15]

R3 & I2

26 Regional Integration

EC support for the secretariat has increased ASEC’s institutional capacity with regard to the facilitation of the regional integration process. However, limiting factors - that EC interventions have not able to ease - are a lack of institutional memory due to high staff turnover particularly at the level of technical officers and lack of project databases. Moreover, the ultimate impact of the EC’s successful support to the ASEC must be placed in the context of the overall integration process, in which sluggish implementation in ASEAN Member States, not weak capacity at ASEC, are the main constraint to more rapid progress. [16]

R3 & I1

27 Trade

EU-ASEAN trade and investment relations generally improved during the assessment period, even if the data do not reveal a clear and persistent trend. Looked at from the ASEAN point of view, there has been a noticeable increase in the importance of the EU as a trade partner. EU FDI in ASEAN has been strong, although investment promotion was never an explicit focus of EC-ASEAN cooperation, and trends are driven more by the business cycle and portfolio diversification. We are unable to answer “to what extent” the good state of EU- ASEAN trade and investment relations is attributable to the EC-ASEAN cooperation programme. However, interviews across the spectrum of actors suggest that it played some encouraging role. [17]

No R link I1



28 EU Visibility

At the ASEC itself, support has resulted in high visibility of the EC / EU and a high regard for Europe as a source of experience to be shared. The EC’s concentration on its area of comparative advantage, trade integration, has supported this, although there is now an expressed desire to benefit from EU experience in a broader range of areas. At the country level, knowledge of the EU is limited and does not seem to extend far beyond elites and direct project beneficiaries. [18]

R4 I2

As is shown in the fourth column of the table, two of these four results [25 and 26] are related to Result area 3 in the ICI faithful intervention logic, that is Safeguards Links strengthened at the regional level. One result [28] can be related to Result area 4 in the intervention logic, namely Awareness about and understanding of the EU and its visibility in partner countries enhanced. Finally, one of the four results [27] could not be logically linked at the level of expected results, but was found to relate to the Impacts level in the faithful intervention logic. None of the four results relate to the intervention logic Results 1 (links strengthened at bilateral level) and 2 (multilateral level), and it was already emphasised that the results only relate to the ASEAN members and not to other ICI-eligible countries. At the impact level, results 25 and 28 relate to the second Impact area (I2: contribution to creating an environment conductive to pursuing and developing its relations with those countries and territories), result 27 relates to the first Impact (I1: contribution to achieving economies of scale, synergies greater effectiveness and visibility for Community action at bilateral level). Result 26, finally, could not be clearly linked to either of the two Impacts. In terms of the nature of the results, it can be concluded on the basis of the descriptions in the above table that results 25, 27 and 28 are of a positive nature, in the sense that they have effectively contributed to realising the expected results as shown in the faithful intervention logic. Result 26 on the other hand, is judged as mixed given that the training interventions has not led to good results for reasons related to relevance (other factors are more important in determining progress in regional integration) and effectiveness (high turnover reduces the contribution made through training). The result is mixed in the sense that it does not make an effective contribution to realising the expected Impact as shown (as mentioned, a clear link at the result level could not be found). Conclusions to be Drawn from this Evidence One first and obvious conclusion to draw is that the collection of evaluations analysed has not resulted in sufficient evidence to be able to draw meaningful conclusions on the extent to which the expected Results and Impacts as identified in the intervention logic for the ICI have been taken further. The meager amount of results can be related to both areas of impacts, but only covers half of the Result areas identified in the intervention logic. Three of the four results do highlight positive results achieved through the EC’s cooperation with the ASEAN, and it may be carefully concluded that the EC’s cooperation – which was not financed through ODA – has made a good contribution to the relations between Europe and this Asian group of countries.



1.7.3 Response to SQ6 : On the basis o f resul ts ident i f i ed ( in SQ5) are there any overal l t rends or l essons to l earn to be ident i f i ed from the evaluat ions and that might be pert inent to the instruments?

Identification of Any Overall Trends or Lessons Learnt The number of results identified in the evaluation reports is small which means it is not possible to read much into these observations and no real conclusion can therefore really be reached on the value of the ICI as an instrument. Given that the DCECI instrument, as desired by the EC, did not come about in the end, and in view of the remarks on the relations between the ICI and DCI, it would be relevant to undertake further examinations on how interventions financed from both instruments in the same country relate. Conclusions Although no overall conclusion can be drawn on the effects of the ICI due to lack of evidence, the limited evidence that is available is mostly of a positive nature. While it can be argued that a contribution has been made to the expected Impacts of the ICI instrument in the ASEAN region, the exclusive focus of the available results in that region means that no conclusion can be drawn at the Global Impacts level (i.e. the overall purpose and mission of the instruments in the EU’s external action). The meager amount of results identified in the evaluation report clearly suggests that there is a case for strengthening the evaluation of EU work financed through the ICI. If the ICI would continue as a separate instrument, then it could be considered to adopt specific procedures to make sure that evaluations which will be required at the level of individual interventions financed through the ICI are made available for and covered in independent evaluations commissioned by the DEVCO evaluation unit and the EEAS, should its mandate also cover evaluations.

1.8 European Instrument on Democracy and Human Rights (EIDHR)

1.8.1 Response to SQ5: What are the overal l resul ts and e f f e c t s per t inent to the instruments ident i f i ed in the evaluat ions?

The analysis of the evaluation reports highlighted 29 results relevant to the EIDHR. Of these, four unique results were identified in thematic evaluations (civil society and good governance), two in regional evaluations (pacific and TACIS) and twenty two in country evaluations (Armenia, Angola, China, Nigeria, Chad, Jordan, Laos, Malaysia, Maldova, Mozambique, Tanzania, Thailand, Vietnam and Egypt). However, only 28 results can be viewed as ‘unique’ results. Table 17 summarises the results (both unique and not) on the basis of four categories, namely: • Type of evaluation: country, regional, thematic • Type of results: positive, mixed, negative23 • Number of results linked to the intervention logic: link with the different results as presented in

the interventions logic under Phase I of this study • Sector focus: the type of sector under which the result falls

23 Here positive results are defined as results that can be seen as effectively contributing to the achievement of the specific Results as reflected in the faithful intervention logic. Mixed results are results that were judged by the evaluators as unsuccessful or ineffective in taking these expected results forward. Negative results are referred to when the identified result was counterproductive to the results as described in the regulation.



Table 14: Evaluation results related to the expected Results of the EIDHR Intervention Logic

Expected result EIDHR Intervention Logic Number of unique results

Number of results

Number of Results 28 29

Type of evaluation

Country Evaluation 22 22 Regional Evaluation 2 2 Thematic Evaluation 4 5

Type of Results

Positive 21 21 Mixed 7 8 Negative 0 0

Link to the IL Results

R1 – blank result – 3 3 R2 – human rights, democracy & civil society 22 22 R3 – international HR frameworks, Democracy and rule of law 2 2

R4 – electoral processes 2 2

Link per sector

Civil Society 12 12 Governance 4 4 Human Rights and Democracy 9 9 Cross-Cutting 3 4

Results per Type of Evaluation The table above shows that 75% of the results have been identified in country evaluations. This would be consistent with the fact that in recent years most of the EIDHR resources have been disbursed through country rather than regional programmes. The evaluations reviewed also do not include a thematic evaluation on human rights though the JEU does have one currently in progress. Type of Results 75% of the identified results were judged as positive. Only 8 results (or 7 unique results) were mixed and there were no negative results. The identified mixed results were in situations were partner governments were probably not open to political dialogue nor to a stronger role for civil society. Link with the Intervention Logic The information collected from the evaluations indicates that the results were largely related to Result 2 (Respect for and observance of human rights and fundamental freedoms enhanced, Democracy and democratic reform in third countries consolidated, Civil society active in the field of human rights and democracy promotion strengthened) in the intervention logic pertaining to respect for human rights and the rule of law, democratic reforms in third countries and support to civil society organisations (see also Figure 6 below). This raises the question of the balance between the different results and whether or not a good balance in the areas supported and results achieved is important to the achievement of the intended impact.



Figure 6: Link between the evaluation results and the intervention logic results

Sector Focus A closer look at the evaluation results identified this time by sector also shows a strong preponderance of results linked to civil society (see also Figure 7 below). This would tend to reinforce the earlier conclusion that the evaluation results linked to the intervention logic Result 2 are indeed heavily focussed on only one (civil society) of this R2’s three aspects (support to civil society, democracy and human rights). This suggests once again that there was an imbalance in terms of the areas supported.

Figure 7: Results per sector

Analysis of the Evidence

The section above highlighted a number of interrelated conclusions that may indicate a cascading effect as a result of some choices and limitations on the field. One of the main points that came up in the analysis is a strong focus on the delivery of Result 2 with a specific focus on support to civil society. Such support has been successful in most cases and easier to provide since there is no need to use the government channels. But the evaluations also noted that the EC has had limited impact



on the overall environment in some countries (eg. Egypt and Malaysia) with respect to human rights and democracy promotion. This was accompanied by difficult or even non-existent political dialogue with partner governments, which explains why the EC might have used the route of civil society support through the EIDHR. The evaluations that were considered in this study also showed that there appear to be very few results achieved with respect to the other Results envisaged in the intervention logic and particularly a weak support to international human rights institutions (eg. International Criminal Court (ICC)), as well as support to electoral processes. On the latter however, it can be recalled that during the negotiations regarding the establishment of the EIDHR, the EP indicated its wish to see a reduction in support to elections observers’ missions, which consumed a large part of the budget of the predecessor of the EIDHR. Although other potentially-relevant evaluations of other areas of support that may fall under the EIDHR (i.e. Human Rights Evaluation) were not available to be considered in this study, the analysis of the multiple evaluations that were conducted over the last 3 years have highlight an apparent imbalance in how the EIDHR is being used to achieve its intended Impact does however raise questions on the logic of the EC’s intervention in this area. One of the tentative conclusions that can be drawn from the data above is that it would seem that civil society is largely seen as the catalyst for democracy and human rights promotion and only a limited number of other types of action are evident through the evaluations. At the same time it is not clear how this apparent imbalance will influence the achievement of the overall Impact of the instrument.

1.8.2 Response to SQ6: On the basis o f resul ts ident i f i ed ( in SQ5) are there any overal l t rends or l essons to l earn to be ident i f i ed from the evaluat ions and that might be pert inent to the instruments?

• Four key overall trends could be identified from the comparison of the grids and the intervention logic. The first relates to the impact on poverty reduction. The evaluations studied covered by this study did not make reference to the impact of EC’s intervention under the EIDHR on poverty reduction directly or indirectly.

• A close look at the mixed results which indicate areas where the EC could not make impact were all linked to a weak political dialogue with the partner countries in situations where democratic rights are not excepted or where the government of the partner country does not deem necessary the role of the EC in this area (mainly Malaysia).

• Based on the analysis in this third phase, and taking into account the available evaluations, it is difficult to assess to what extent the EC has been successful in promoting democratic governance and human rights through the EIDHR. The evaluations covered by this study showed that the EC has achieved positive results as shown on Table 17 especially with respect to civil society support but it is not clear how this support will contribute to the intended overall impact. The question that this raises is does the lack of a more balanced mix in the type of interventions (i.e. civil society as a focus and less focus on elections, support to parliaments, etc.) have an impact on the achievement of the global impact as outlined in the intervention logic? However, the Study Team recognises that other future evaluations on relevant topics (i.e. Human Rights Evaluation) may uncover more evidence on EU intervention in areas covered by the EIDHR.



1.9 Instrument for Nuclear Safety Cooperation (INSC)

1.9.1 Response to SQ5 : What are the overal l resul ts and e f f e c t s per t inent to the instruments ident i f i ed in the evaluat ions?

Descriptive / Factual Evidence on Results and Effects Identified Only 13 pieces of evidence related to the Instrument for Nuclear Safety Cooperation were identified in the analysis of the evaluation reports. Of these 5 were found in the Energy Thematic Evaluation [110, 112, 116, 307, 377] and all except two of these related to multiple Results (10 in total) in the INSC Intervention Logic. This therefore leaves only 4 ‘unique results’, two in the Energy evaluation and one each in the country evaluations for Armenia [7] and Ukraine [350]. Result 3 in the INSC faithful intervention logic, that is Safeguards for accounting and control of fissile material are established was supported by most evidence from the evaluations with four ‘hits’ [112, 116, 307, 350], three of these were found in the Energy evaluation and the fourth in the Ukraine evaluation. Result 1 in the intervention logic, that is Effective nuclear safety culture exists, was supported by three pieces of evidence from the Energy evaluation [112b, 116, 307]. Result 2, that is EU citizens are protected against radiation and Result 4, Nuclear accidents are prevented, were each supported by two pieces of evidence. In the first case these were both from the Energy evaluation [110, 112c] and for R4 one from the Energy evaluation [307] and one from the Armenia evaluation [7]. Result 5, The Consequences of nuclear accidents are mitigated [110], and Result 6, The EU cooperates with the IAEA [377], were each supported by one piece of evidence both from the Energy evaluation. While most of these results were positive, it should also be noted that three of the pieces of evidence, one relating to R1 [116] and two to R3 [112d, 350] were in fact mixed results that suggests that the actions in question is not yet assured. In the INSC intervention logic only one Impact24 was identified to which all the Results contribute. This was confirmed by the same group of fourteen results identified in the evaluation reports coming up in relation to the Impact in the INSC as shown in the intervention logic. The fourteenth piece of evidence could in fact only be linked to the Impact level and not to anything at the Result level of the INSC intervention logic. This is also a negative piece of evidence in that it indicates that “EU Member States have very different sensitivities regarding nuclear power production. This has an impact on the ability of the Commission to support nuclear safety in the CIS countries” (Energy Evaluation, 2008, p.69)[378]. Conclusions to be Drawn from this Evidence From the identified results it is apparent that the evaluation reports recognize that the EU has made useful contributions to promoting nuclear safety principally in ENPI countries and that it has in fact a record of being one of the largest foreign contributors. The EU’s contribution was particularly noteworthy in addressing the aftermath of the Chernobyl disaster where it acted effectively in an emergency situation and at an appropriate scale by providing up to 50% of the international funding required. In doing so it achieved a number of the results set out in the INSC legal instrument, not least protecting EU citizens against radiation, mitigating the consequences of the accident and probably also preventing further accidents. At the same time, while the evaluations do note this EU strong contribution to positive results and overall impact they also recognize that these cannot be purely attributed to the EU’s input nor is there much certainty as to the longer-term sustainability of the impact. One of the results [112] also specifically mentions that ‘Overall progress in nuclear safety is very hard to evaluate, as no risk assessment has been conducted recently’. One specific difficulty faced by the

24 The one Impact in the INSC intervention logic is: Contribution to support high level of nuclear safety, radiation protection and

application of efficient and effective safeguards of nuclear material in third countries (this can be found in Preamble 1 & Art.1 of the INSC Regulation)



EU and noted by evaluators is that over time, as the immediate need to tackle the crisis retreats into the past, it has become more difficult to maintain a consensus among EU Member States on nuclear issues. Nuclear power is a controversial subject in Europe and the European Commission has therefore found it increasingly difficult to build consensus in the Council on priorities for nuclear safety.

1.9.2 Response to SQ6: On the basis o f resul ts ident i f i ed ( in SQ5) are there any overal l t rends or l essons to l earn to be ident i f i ed from the evaluat ions and that might be pert inent to the instruments?

Identification of Any Overall Trends or Lessons Learnt The number of elements of evidence identified in the evaluation reports is small which means it is not possible to read much into these observations and no strong conclusion can therefore really be reached on the value of the INSC as an instrument. What results there are however, are positive with the clear suggestion that the EU, via the INSC, is making an important contribution in this field. It is also apparent from the Energy Thematic Evaluation (2008) that the European Commission’s Nuclear Safety Unit has built up considerable expertise and credibility in the sector. The evaluators also noted the EC had built up a good network of collaborating institutions both within the EU at national level and internationally under the auspices of the IAEA. There is therefore a good basis on which to build further EU work in this area. At the same they also noted a number of points which are important to stress for the future: 1. First, it was noted that over time without the crystallizing effect of a major nuclear disaster, such

as Chernobyl, it is becoming increasingly difficult for the European Commission to build consensus around the policies it proposes to Council.

2. Second, there is clearly still considerable work to be done to achieve the full “Effective nuclear safety culture” [R1] that the INSC instrument talks about as “…partner governments seem less convinced that the EU of the necessity of promoting the independence and the technical and institutional strengthening of the Regulatory Authorities.”

Conclusions Although no strong overall conclusion can be drawn on the effect of the INSC due to the limited evidence emerging from the evaluations, what limited evidence there is is positive. The EU is now an experienced and credible actor in the field. Yet there are clearly also continuing issues to be tackled and not all of them will be straightforward. On the basis of the evidence available therefore it appears to be important to maintain an EU budget in this area. The case for maintaining a separate legal instrument is however not easy to argue conclusively with the limited evidence available. That said the intervention logic of the INSC is clearer than several of the other instruments and this study shows that it is relatively straightforward to identify evidence (in the couple of evaluations that tackle the subject) that links up with the Results and Impact specified in the intervention logic. One assumes this makes it easier to manage the instrument and focus more effectively on the task in hand. The nature of the nuclear safety task is also very specific which suggests that integrating it into another larger instrument might simply add unnecessary complications. Finally we should note that no evidence whatsoever emerged that suggested that changing the scope of the instrument would have advantages. It would seem therefore advisable to retain the INSC in its present form as a separate instrument.



Having argued that the INSC should probably be maintained as a separate instrument does not mean however, that no improvements could be made to the instrument. In particular the paucity of results identified in the evaluation reports does suggest that there is a case for strengthening the evaluation of EU work financed from the INSC. The clear reference to the need for an overall risk assessment of nuclear safety would also support such a conclusion.



2 SYNTHESIS OF THE PHASE 3 EVIDENCE ON THE SIX LEGAL INSTRUMENTS

2.1 Response to SQ5: What are the overall results and effects pertinent to the instruments identified in the evaluations?

The overall picture that emerges from the analysis of evaluation reports in Phase 3 is that the exercise is only really useful for three out of the six instruments, that is the DCI, ENPI and EIDHR. There are simply too few results that can be linked to the other three instruments, the IfS, ICI and INSC, to be able to conclude anything particularly solid. That said those results that can be linked to these three other instruments are broadly positive suggesting that more targeted evaluation work on these instruments might be expected to endorse their value as instruments. The virtual absence of evidence for these three instruments also means that the case for conducting such evaluations is clearly strong and it is positive to note that at least in some cases (eg. IfS) steps are being taken to remedy this gap. Even among the three instruments for which the exercise does produce some meaningful conclusions there are considerable variations in the number of results linked to each of them: 147 for the DCI, 74 for the ENPI and only 28 for EIDHR. The relatively high number for the DCI can be explained by its thematic programmes that cover all developing countries, which means that relevant findings from evaluations covering ACP countries could be included. This variation needs to be kept in mind in assessing the strength of the trends identified in each case. A quick reminder of some methodological limitations is also useful at this stage. The exercise of identifying results in the evaluation reports and their linking to the intervention logics is subject to a number of limitations which may affect the overall number of results found and the scope for making the desired links. Thus the evaluation reports available may first be inappropriate for the instruments. This is probably the main problem with the three instruments for which too few results were found. Second the focus of the evaluations may be inappropriate for this exercise and therefore not push evaluators to identify results that can be used here. Third, the exercise of extracting results from the evaluation reports may have been inadequate in some way, even though considerable care was put into designing the data collection tools and on quality control of the work done. Finally of course, the analysis made here is preliminary and will be further sharpened and where possible expanded in the fourth and final phase of this study. These introductory comments covered, a number of points on the overall outcome of the exercise for the three principal instruments (DCI, ENPI and EIDHR) can be made with some confidence: 1. The first point is that for all three instruments the results identified are in the majority positive,

though there is also a sizeable proportion of mixed results in each case. For the DCI the there is a 57/40 split with the small remainder (3%) negative, whereas for the ENPI the proportion is 50/50 with no negative results. In the case of the EIDHR the results are overwhelmingly positive. This suggests that overall the evaluation reports by and large indicate that the work done under these instruments is achieving progress towards the objectives of the instruments. At the same this positive conclusion needs to be tempered with the fact that there is also a sizeable proportion of mixed results, which suggest that the objectives set for the interventions were not always effectively pursued. This therefore indicates that there is also scope for considerable improvement.



2. When making the link between the results found in the evaluation reports and the Results and Impact of the intervention logics the overall finding is that some 70% of the results can be linked to specific points in the six intervention logics. However, it is important to note that of the remaining 147 that could not be linked, the big majority (112) occurred in EDF countries and might therefore have been attributable to an EDF intervention logic if it had been practically possible to construct one25. In other words if the results potentially associated with the EDF are discounted there appears to be a very good match between the results identified in the evaluation reports and the Results and Impacts identified in the instruments through the intervention logic exercise.

3. However, taking the analysis one step further down it quickly becomes apparent that there is a

pattern of uneven distribution of the evaluation results across the different intervention logics Results and Impacts. This applies in the cases of all three instruments but is probably particularly noteworthy in the EIDHR, where it may in fact suggest some imbalance in the way the instrument has been implemented. At the level of intervention logic Results we may note the following key trends:

a. For the DCI, two regions in particular, Asia and Latin America, account for the bulk of

the results associated with regional Results in the intervention logic and on the thematic side it is particularly the three programmes Investing in People, Environment and NSAs/Local Authorities that show strong levels of linkage. Very few to nil results were identified for the Central-Asia, the Middle East, South-Africa and the thematic programmes on the ACP sugar protocol and migration & asylum.

b. In the case of ENPI the intervention logic has two main Results (R1 and R2) and 75% of the linkages with the evaluation reports are with R1 (Support to partner country efforts to promote good governance and equitable and social economic development) while only 25% can be linked to R2 (Partnership and cooperation agreements). Given the very different nature of these two Results this distribution is probably not of great concern in this case.

c. Finally in the case of the EIDHR the concentration of linkages is very heavily (75% or

12 out of 16) on the second of the four Results identified in the intervention logic (R2 – human rights, democracy and civil society), whereas another of the Results (R4 – electoral processes) in particular could also be expected to achieve a good number of results. Very few results could have been related to the other two Result areas, which means that little can be learned of what was achieved there.

4. A sectoral criterion was also used for analysing the results from the evaluation reports. For all

three instruments this produced an uneven spread with certain sectors scoring more than others. With the DCI this brought out an emphasis on four main sectors: education, economic sector and trade, governance and decentralisation and finally environment and natural resource management. For the ENPI this analysis suggested economic sectors were achieving the greatest number of results. Finally, in the case of the EIDHR, this showed a heavy emphasis on support to civil society, which confirms the conclusion reached above on links with intervention logic Results.

25 The possibility of constructing an EDF intervention logic was discussed with the JEU and the Reference Group during the

inception phase, but no document equivalent to the regulation of the six instruments under study exists for the EDF. The Cotonou Agreement is of course a very different sort of document. As a result comparisons with the six intervention logics would have been extremely tenuous.



5. In terms of evaluation report results that can be linked to the Impact level in the intervention logics it is harder to identify any overall pattern or trend as different pictures emerge for each of the three instruments, partly dependent on the number and type of Impact in each case. One other general observation is that, while findings are relatively precise at the Results level, they are much more tentative and ambiguous and hence less solid at the Impact level.

a. For the DCI the results that can be linked to the Impact level are distributed across most of the half a dozen listed Impacts, though inevitably the generally worded Impact (I1) scores higher than the more specific ones and the last one that relates more to foreign policy than development policy (I1.5 – strengthen relations between EU and country/region) has a zero attribution. One particular limitation is that what has been achieved in terms of Impact through the geographic programmes is based on sufficient evidence from only two of the five regions identified in the DCI regulation.

b. In the case of the ENPI 73 results can be linked to the intervention logic Impacts with 64% linking to Impact I1 (commitment to EU values) and 29% to Impact I2 (enhanced cooperation and integration). The remainder relate to I3 on cofinancing. The rather different nature of these Impacts would seem to provide an adequate explanation for this uneven distribution.

c. For the EIDHR there is only one Impact identified in the intervention logic and all the results can be related to this Impact.

6. Finally in all three cases the contribution of the results from the evaluation reports to the Global

Impacts of the interventions logics remains somewhat nebulous. This is partly an issue relating to the instruments themselves and their lack of detail on certain aspects of the logic chain of thought that is apparent in them; and partly a question that routinely surfaces in evaluation reports. This implies that little can be learned from this source as to what contribution the EC has really made through its external action programmes to advancing the global objectives and ultimate purpose of the legal instruments (e.g. reducing poverty, strengthening relations, ensuring nuclear safety). As political debates in the Council and Parliament typically focus on the global impact level when discussing these instruments, this points to an information gap that would need to be better addressed in the future.

One last general observation that is particularly striking in the distribution of the evaluation reports examined by the study, over the external actions financed from these six instruments is that it is those that involve ODA funding, along with the EDF (also ODA), that are being evaluated. The two instruments involving non-ODA funding (ICI and INSC) do not have any specific evaluations relating to them at all. It would seem important to correct this in the programming of future evaluations as this situation is clearly not satisfactory even though ODA does constitute the bulk of EU external actions funds. The study only looked at JEU commissioned evaluations, of course, so there may well be others commissioned by other EC services. But then for transversal studies such as the present one, it would be important to ensure these other evaluations can also be examined and that they are conducted to similar standards to those of the JEU so as to allow for comparisons to be made.



2.2 Response to SQ6: On the basis of results identified (in SQ5) are there any overall trends or lessons to learn to be identified from the evaluations and that might be pertinent to the instruments?

Overall then in all three legal instruments for which sufficient evaluation evidence is available, it would seem that there is a fair degree of convergence between what the regulations of the instruments set out as the objectives to be achieved and what is actually achieved through the work recorded in the evaluation reports. There is a good level of results that can be identified in evaluation reports and then convincingly linked to the Results identified in the three intervention logics. Such linkages can also be made at the Impacts level but here they are more uncertain and at the level of Global Impacts they are generally weak. While the majority of these results are also positive there remains, at the same time, a good number that are mixed. Thus the results identified in the evaluations reports also demonstrate that performance can be improved and a higher level of positive results achieved. The exercise of linking the evaluation report results to the intervention logics also demonstrates that in practice major variations occur in the emphasis put on pursuing the different objectives of the instruments. Some intervention logic Results and Impacts thus attract a lot of results from the evaluation reports, others far less. This may be due to a wide variety of reasons, not necessarily negative, although in some cases (eg. the EIDHR) these imbalances do seem to suggest that some aspects of the instruments have been neglected and that this may in turn affect negatively the overall impact of the instrument. In terms of useful lessons to learn, it is clear that the link between instruments and evaluations could be made a lot stronger. This should have benefits on both sides. For the regulations of the next generation of legal instruments to specify what the expectations were in terms of evaluation would probably push the drafters to be more specific about results and impacts to be achieved. For evaluators to be able to link their work more clearly to the instruments would provide a stronger framework within which to work effectively. However, it is open to question how easy it would be in practice to be much more specific in the instruments themselves. It may well be that an intermediary document of some form would be more appropriate for this, perhaps a strategy document drafted in conjunction with the instrument regulation that spells out how the instrument is to be used and evaluated in more detail. Another overall comment that can be made is that the evaluation reports studied under Phase 3 have brought up very little evidence regarding the extent to which synergies occur between the use of several instruments or indeed of the blending of what ODA and non-ODA funds may occur. This was a relatively important issue in the legal instruments. In all of them the importance of synergies between instruments is highlighted and the discussion on blending ODA and non-ODA funds overshadowed the negotiations on the instruments and indeed resurfaced in the Mid-Term Review of the instruments published in 2009. Moreover, given the likely importance of these issues in the upcoming discussion on the next EU Multi-annual Financial Framework it is a shame that such evidence has not emerged so far and this may require a bit more though in the final phase of this study. A further overall comment is that the instruments vary hugely in scope and scale. The smaller instruments could be linked to fewer results in the evaluation reports but their intervention logics are also more specific and clear in what they set out to achieve. One can imagine that if appropriate evaluations were available for these instruments the linking of any results identified with the intervention logic would be fairly straightforward and specific. This is evident to some extent with the EIDHR where enough results were identified and where the links with the intervention logic



were generally very clear and specific. For the two largest instruments, the DCI and the ENPI, the intervention logics suffered from long lists of items at each level and a general lack of clarity about how specific items at one level linked up to achieve the item at the next level. With such an intervention logic there is a lot of leeway for interpretation, and while the results from the evaluation reports do give an impression of positive forward movement the conclusions remain at a certain level of generality that may not be that useful in terms of learning very specific lessons. While the previous reform of legal instruments was evidently driven by a need to reduce the past proliferation of instruments, it should now be considered whether some of the resulting current instruments are now not so wide in scope and coverage as to actually discourage effective evaluation, policy learning and accountability.



3 REVIEW OF EVIDENCE ON IMPLEMENTATION ISSUES

3.1 Introduction

This chapter identifies a number of trends concerning implementation issues that emerge from the evaluation reports examined. These cover issues of management, approach, modalities and performance. The section is based on the systematic analysis of all the extracts of the third sheet of the grids for the extraction of results from the evaluation reports (Question V). This table includes over 230 extracts from more than 50 different evaluations of different types: country, region, and thematic/channel26 evaluations. A first phase of the analysis aimed at identifying 8 key words to filter the information of the table into clusters. Based on the analysis of the content of each cluster, four headings were identified in order to group messages around a few key development issues, namely: 1. Effectiveness and the 3Cs27; 2. Programme cycle management, 3. Actors dimension, and 4. Modalities. These themes are then further sub-divided into a number of sub-themes. These sub-divisions provide the structure of this chapter. Each message, or trend, identified in the text originates from the content of the evaluations and is weighted according to the number of extracts it is mentioned in. In the chapter below, the reference of the extracts used to identify each point is indicated with its reference in the table (i.e. the line it appears on). An annex28 with all extracts used for each theme is also available to trace the sources of each message. The annex also gives a more detailed account of the evaluations used for each of the sub-theme. All messages and argumentations are therefore perfectly traceable.

3.2 Global Count of Evaluations of Extracts Used for the Analysis

The analysis is based over 230 extracts from more than 57 different evaluations. An overall indication of what type of evaluation these extracts originate from is provided in the two tables below. The majority (60%) of the 57 evaluation reports examined were country evaluations and a further 25% were thematic and channel evaluations. As the Table 18 below the distribution of extracts on implementation issues is broadly consistent with this overall pattern

Table 15: Number of evaluations and extracts used for each theme

Theme Number of Evaluations from which extracts originate

Number of Extracts

Total Country Regional Thematic 1. Effectiveness & 3 Cs 41 24 7 11 73 2. PCM 31 17 6 8 58 3. Actors based dimension 26 16 4 5 43 4. Modalities 34 24 5 5 55

26 I.e. aid channel such as Civil Society, Development Banks, International Finance Institutions (IFIs), UN. 27 3C’s: Coherence, Complementarity and Coordination 28 See Annex 9 in Volume 2 for a more detailed account of the sources.



Table 16: Number of evaluations and extracts used for each sub-theme

Sub-Theme Number of Evaluations Number of Extracts 1.1 Alignment 11 14 1.2 Coherence 7 7 1.3 Complementarity 11 15 1.4 Coordination 26 31 1.5 Synergy 21 24 2.1 Programming 13 16 2.2 Delays 17 21 2.3 M&E 19 23 2.4 Learning 10 12 3.1 Participatory Approach 15 20 3.2 Partner Capacity 17 25 4.1 Project Approach 14 17 4.2 Budget Support 19 27 4.3 Modality Mix 21 24

3.3 Effectiveness and the 3Cs

This part is based on 73 extracts from 41 different evaluations29. Most of the comments concerning the 3Cs are made separately30 for each of the individual Cs, that is coherence, complementarity and coordination, so they are reviewed separately. A few additional points on alignment and synergy contribute to inform the reflection on EC aid effectiveness.

3.3.1 Alignment31

A good number of evaluation reports suggest that Country Strategy Paper (CSP) priorities were aligned to government priorities (and respective development plans, such as Poverty Reduction Strategy Papers (PRSPs)) during programming and remained aligned during the implementation phase. This situation contributed de facto to improve coordination with partners and the donor community, and EU internal coherence to respond to its partners’ development challenges [62, 69, 99, 129, 134, 143, 160, 204, 218, 230]. Alignment is also seen to impact positively on a number of elements, such as the quality of (sector) policy dialogue, domestic ownership, speed of implementation, aid fragmentation, effectiveness of government programme and pro-poor development outcomes [99, 100, 134, 218]. A few examples also show that the EC cooperation responded better to partners wish regarding aid delivery modalities, especially with regards to new aid modality and demand-driven instruments (i.e budget support) [110, 129, 134, 230]. Specific alignment on procedures is mentioned only once [100], and there is little evidence that alignment meets the capacity development needs of partner governments [148]. In addition, one case suggests that procedures and pressure on disbursement could end in reducing alignment and predictability [233].

29 See Annex 9 for a more detailed account of the sources. 30 The evaluation for Central Africa is the only one indicating that the appreciation of how the 3Cs are taken into account is not

satisfying [20]. 31 Analysis based on 14 comments, from 11 different evaluations.



3.3.2 Coherence32

The 57 evaluation reports analysed use two different definitions of coherence interchangeably: a) Coherence defined as internal coherence: ‘the extent to which the intervention logic is not contradictory/the intervention does not contradict other intervention with similar objectives’ (Source: TOR for this study) b) Coherence defines as policy coherence for development: ‘The Union shall take account of the objectives of development cooperation in the policies that it implements which are likely to affect developing countries (Lisbon Art. 188D).’ With regard to the first definition, the coherence between EC general development policy and country or region strategies, and between instruments, is generally satisfying, even though a genuine poverty reduction focus is difficult to trace [19, 44, 118]. One example (Egypt) indicates on the contrary that the EC internal coherence was not ‘foreseen at design stage and occurred rarely during implementation’ [163]. With regard to the latter, PCD is analysed in detail in a number of evaluation reports, most notably in the context of the Joint-Evaluation of Maastricht 3Cs (http://www.three-cs.net). In addition to these studies, the Study Team picked up findings here are positive in two cases (Chad, Pacific), but inconclusive in another two cases (including Rwanda) [53, 93, 118, 185].

3.3.3 Complementari ty33

Similarly to coherence, there are also different understandings of complementarity which are reflected in the evaluations, which can be analysed between donors working within the same country (also known as ‘in-country division of labour’) or different countries (‘cross-country division of labour’), but also within the different interventions of a donor (e.g. a technical assistance project in the Ministry of Finance which complements General Budget Support interventions by the same donor). With regard to complementarity within the EC’s strategy, the evaluations often analyse this in a rather theoretical manner (i.e. between NIP and RIP, EC instruments) [20, 22, 44, 141, 162, 208]. Constraints still seem to outnumber incentives to increase complementarity and the CSP is not seen as a conducive tool to do so, unless it contributes to the logic of more flexible and/or joint planning and strategies [55, 56, 102]. Concrete progress is reported on more concretely in the area of in-country division of labour, for instance by exiting from some sectors, by better analysing EC added value compared to other development partners or by developing specific geographic and thematic expertise where most needed [102, 110, 118, 221]. In addition, channelling funds through multi-donor trust funds is seen in itself as a contribution to better in-country division of labour [41]

3.3.4 Coordinat ion34

Out of the 3Cs, coordination is the issue mostly reported on. Coordination varies for different reasons from case to case, but it generally remains rather informal and limited, and does not seem to benefit partner countries and regions particularly [20, 23, 44, 46, 55, 115, 118, 178, 181, 197, 215, 234].

32 Analysis based on 7 comments, from 7 different evaluations. 33 Analysis based on 15 comments, from 11 different evaluations. 34 Analysis based on 31 comments, from 26 different evaluations.



When information sharing between the EC and the Member States or other donors is taking place, it can lead to avoid overlaps and to more successfully exploit comparative advantages but it is not deemed sufficient to enhance coordination and division of labour among donors [8, 53, 104, 156, 198, 199, 220]. In a number of case, progress is reported and mechanisms are put in place, preventing duplication of activities at the operational level and sometimes reducing the administrative burden, but the reductions of transaction costs due to a better coordination are not always clear [14, 141, 162, 185, 205, 216]. Coordination is particularly helpful to foster sector policy reforms and progress-related approaches (i.e. variable tranche) for they require common analysis of the pool of donors on the partners’ performance vis-à-vis the conditionality attached to such support [53, 64, 199, 220]. However, sectors where competition or desire for visibility is particularly high are not conducive for enhanced donor coordination [48, 53].

3.3.5 Synergy35

Synergies between different projects and instruments that could bring benefit to a common sector are not optimized. It results in a weaker potential of the overall strategy to contribute to global objectives and scatters funds despite the choice of areas of concentration [109, 113, 114, 155, 165, 181, 231]. Articulation at the different levels (local, national, regional) does not score very well. For instance, some issues are analysed in the reduced scope of national context, whereas their regional or decentralised dimension is key to ensure conducive development. A few cases also indicate that too little synergy exists between regional and national (including bilateral) strategies and programmes, especially in the context of weak regional institutions [7, 9, 24, 37, 52, 129, 189]. Other constraints are mentioned marginally. The fact that the principle of concentration still leaves some room of manoeuvre to commit funds in non-focal interventions tends to strain resources of the EC and its counterparts and ultimately risks reducing aid efficiency [154]. Besides, the incompatibility between the EC and Member States management, financial and operational rules can prevent synergies of interventions, by reducing the possibilities of joint projects [209]. A few cases of valuable synergies are accounted for. Channelling funds through multilateral organisations and international finance institutions (IFIs) has developed over the years and is accounted for bringing added value to the EC aid delivery, allowing for a critical mass of resources and expertise that could not have been mobilised by EC instruments alone [38, 183]. Other channels have not been used to their full potential (i.e. partnership with international NGOs or autonomous local CSOs) [124]. Linking Relief Rehabilitation and Development Strategy and ensuring a continuum of actions initiated by ECHO seem to add value to EC interventions in post war or fragile contexts [2]. Unfortunately, such linkages do not always happen [52].

3.4 Programme Management Cycle

This part is based on 58 extracts from 31 different evaluations36. Rather than providing a systematic look into each phase of the project cycle, this part aims at extracting a few trends that come out of the evaluations in terms of management, from programming to evaluation.

35 Analysis based on 24 comments, from 21 different evaluations.



3.4.1 Programming37

Programming results in a complete scope of activities and modalities that usually are grounded in and correspond to the strategy documents, but it sometimes lacks a prioritisation in each sector or theme which can result in a basket of activities with few links between them so that potential synergies and complementarities with other Commission policies and instruments have not been optimized [34, 39, 113, 114]. The quality and quantity of analysis available to inform the programming and the policy dialogue is sometimes insufficient. It is partly due to the limited EC resources available to mobilise specific support and expertise that would support the process [53, 71, 80, 81, 167, 218]. A number of comments highlight the limits of the programming process to foresee and address future challenges and the fact that it looses its strategic interest once the funds are programmed [61, 70, 72, 180]. It risks to impact negatively on impact and sustainability. Another shortfall of programming is the limited genuine joint ownership of the process, whereby most the arbitrage governing the content of the strategy is either done at an early stage in Brussels, or in a rather unilateral way. [135, 159, 180] In rare cases, decisions to channel funds were neither explicitly motivated, nor based on formal guidance criteria, although relying on donor assessments (e.g. funds channelled through multilateral organisations) [39, 224].

3.4.2 Delays 38

Delays in contracting and implementation are often reported. They are seen to increase the transaction costs of aid and affect its efficiency [6, 19, 23, 24, 40, 45, 53, 62, 168, 180, 191, 217]. The ambition of the programmes, the inadequacy of the procedures and/or the modality, and the organisations of management responsibility and task division between stakeholders are among the three main reasons mentioned to explain such delays [23, 24, 25, 40, 66, 82, 107, 161, 168, 180, 191, 217]. The time required to mobilise technical assistance or providers and improper distribution of management responsibilities are also quoted as risks for a smooth, efficient programme implementation and [24, 25, 53, 106]. When decisions come from headquarters level [45, 66] and the weak capacity of the government can also cause delays in contracting or implementing aid programmes [40, 161, see also 3.2]. The effects of conditionality is only referred to as a specific cause for delay in one case [40]. It is interesting to note that one evaluation indicates that such important delays were fully understood by the partner government as a necessary constraint in order to ensure sound utilisation of aid and public finance [106]. In a couple of cases it is seen as regrettable that lessons learnt from this trend are not duly analysed and taken into consideration [152], especially when it prevents the provision of support in the context of fragility [168].

36 See the Annex 9 for a more detailed account of the sources. 37 Analysis based on 16 comments, from 13 different evaluations. 38 Analysis based on 21 comments, from 17 different evaluations.



3.4.3 M&E / Exit Strategy / Sustainabi l i ty39

In a good number of cases, the weakness of monitoring systems and baseline data and the lack of proper indicators making it difficult to identify and assess impact are highlighted [2, 24, 36, 77, 86, 90, 152, 192, 196, 212]. Efforts are still required to measure results and to enhance their traceability and visibility, especially by correcting insufficient design of verifiable indicators in logical frameworks, and by systematic reference to the original situation and needs for which the interventions have been conceived [12, 77, 86, 87, 106, 152, 180, 196, 202, 217]. Lack of sufficient resources - including fully dedicated staff - to ensure an adequate follow up of implementation is one of the factors that is identified in a few cases as hampering consolidated monitoring [87, 90, 107]. Limited access to and exchange of information between the different services dealing with cooperation instruments is another constraint [36], and the issue of attribution remains complex [157]. Besides, in addition to insufficient monitoring, evaluation reports and recommendations can be poorly taken into consideration [152], but reinforcing the European Union Delegation (EUD) human resources can help in addressing the situation [219]. A number of comments suggested that the consolidation of results is often hampered by the fact that most interventions do not benefit from any follow-up funding mechanisms or proper exit strategy, whereas the continuity of their benefits should be taken more seriously into account in policy dialogue [2, 77, 174, 193]. The priority given in M&E to disbursement, operational issues and activity results rather than impact, process approaches, contextual analysis and changing policy environment is another reason that impacts negatively on sustainability [107, 152, 157, 180]. Project cycles and their duration are unlikely to match a country owned pace of structural reform but there is usually little coherent medium/long term approach agreed with partners on the sustainability of interventions, and how their results fit into a wider impact [180]. So even though continuity in interventions proved a success factor, the absence of an exit strategy affects sustainability [45, 66].

3.4.4 Learning40

Another regularly made point is that mainstreaming and systematic dissemination of best practices does sometimes take place, but is not yet a common practice, although it would enhance the impact of the cooperation [2, 19, 66, 97, 108, 155, 157, 182, 232]. Knowledge management is not formally prioritised, and both the EU and its partner lack resources to sustain dedicated work to capitalise and share their experience in implementing aid programmes [66, 108, 232]. The process of transferring knowledge is complex and not fully mastered [97], even if some progress is noted [175, 180, 232]. Developing such learning opportunities would not only contribute to a better-informed policy dialogue [66, 111], but could also have a spill over effect, for instance by using the capacities developed at the institutional level to inform knowledge based economic development [203]. Mutual learning could be enhanced at different levels: • Between all stakeholders [19, 66, 108]; • Between different EU external relations instruments [133, 155]; • Between different types of modality [97, 133]; • Through specific procedures and tools [232].

39 Analysis based on 23 comments, from 19 different evaluations. 40 Analysis based on 12 comments, from 10 different evaluations.



3.5 Actor Based Dimension

This part is based on 43 extracts from 26 different evaluations41. Two main trends appear in the evaluations: a gradual intensification of the participatory approach, especially of the role of civil society in development, and continuing problems with the way partner governments’ capacity affects cooperation and aid performance.

3.5.1 Part i c ipatory Approach42

Various evaluation report suggest that the evolution of EC strategy over time shows a clear tendency towards intensified participation of non-governmental stakeholders, in policies and strategic dialogue, as well as in the channelling funds, with a growing number of good practises [10, 26, 27, 28, 51, 140, 173, 206]. The importance of the role of CSOs at national and regional level is consensual, but their actual participation varies, and their potential seems to be under used and not fully supported so far [24, 27, 31]. A number of evaluations illustrate the fact that beyond general principles, there is no clear vision of the added value of the different categories of CSOs, according to the context they operate in, the level of their capacity and the operational aspects of their participation, especially in new aid modalities [26, 27, 28, 30, 109]. Besides, mobilisation of CSOs relies partly on how many concrete initiatives they are associated with, or benefit from: if too limited, there is a risk of marginalisation of this group of actors [24, 103]. The difficulty for CSOs to apply for and manage EC funding, and the effort needed at management level to boost and facilitate participation are amply illustrated [27, 30, 103, 155, 188]. CSOs effective partnership with the EC would benefit from more flexible and diversified approaches, including longer term, strategic support and quick, flexible access to funding, in order to give them the opportunity to be pro active and autonomous [26, 172, 188]. A shift in the prevailing EC administrative culture, focusing primarily on spending aid and ensuring financial accountability, towards a more result and impact oriented approach is needed to be able to use CSO channels in a more strategic way [30, 45]. In addition, coordination between different financial instruments in support of civil society (thematic, geographic, horizontal), and their managers (EUD and headquarters in Brussels) is rather limited, thus inhibiting the achievement of a higher degree of concerted action and knowledge sharing (especially beyond the output level) [155]. The relevance of EC interventions focused on specific (vulnerable) target groups varies but is confirmed in most cases as being positive, especially when using CSO channels [10, 51, 137, 212]. Decentralised cooperation and Micro Projects programmes have fostered participatory approaches, even though the capacity development dimension was not sufficiently addressed [32].

3.5.2 Partners Capaci ty43

Partners’ often limited capacity to cope with EC management systems and to implement and monitor EC funded programmes is often quoted as a factor that impacts negatively on aid effectiveness and efficiency [19, 61, 209]. Weak institutional frameworks within the governments often impede effective policy dialogue, even when the EUD can assist partners with developing sector strategies and sector budget support [57]. Strong technical capacity of the national or regional counterparts are key to optimising the quality of interventions, especially when EUDs are not well

41 See the Annex 9 for a more detailed account of the sources. 42 Analysis based on 20 comments, from 15 different evaluations. 43 Analysis based on 25 comments, from 17 different evaluations.



equipped to manage sector related interventions that require a high degree of specific technical expertise, or to engage in multi donor dialogue on these subjects [45]. So the choice of the appropriate implementation mechanisms according to the prevailing institutional context and the assessment of the capacities of the beneficiaries are key to achieving cooperation objectives [176]. Indeed, a few examples illustrate that structural public management weaknesses or absence of enabling environment have constrained the EC effort to contribute to capacity development, which in turn, caused delays in implementation and disbursement, and affected negatively the development results obtained [82, 87, 106, 213]. Some cases illustrate that the limits of partners’ capacity - and the risk that it affects the implementation of joint development strategies - is not well taken into account in the programme identification phase. Besides, monitoring indicators do not always include the right incentive for capacity development [73, 111]. Various ways of supporting capacity development are identified in the evaluations. For instance, multi-donor capacity building projects to lay the foundation for sector budget support can contribute to make some sector administration more effective, but improvement relies on quick success of the support [136, 149, 203]. A more strategic use of technical assistance within domestic institutions is seen as a way to contribute to reduce partners’ capacity constraints, even though there is little evidence in the evaluations of sustainable capacity development delivered by years of technical assistance [101, 106]. The introduction of the Contribution Agreements, which allow for the use of the recipient institution procedures, has significantly improved aid delivery efficiency, but the issue of regional organisations capacity to be eligible have not been adequately addressed so far [62, 186, 188]. Externalisation of demanding programmes in terms of human resources to international organisations can foster project efficiency, but it does not contribute to addressing capacity shortages in partner administrations [128]. Overall, a number of solutions have been tried out, but they do not really address structural deficiencies and the EC has had difficulties in finding the adequate support role in helping its partners to develop sustained organisational capacity [202].

3.6 Modality

This part is based on 55 extracts from 34 different evaluations44. Out of the modalities available to channel aid, projects and budget support are the ones the most reported on. The evidence also illustrates that a balanced mix of modalities is key to effective development.

3.6.1 Projec t Approach45

Evidence from the evaluations shows that the project approach is still very much in use, at national and regional level and is seen as capable of delivering results with respect to strategy implementation, in a wide range of sectors [i.e. 19, 154, 186, 201, 219]. In situations of fragility or instability, or in areas that are complicated by the nature of the subject, it allows for a step-by-step approach. Small project facilities have proved to be more successful in involving all relevant stakeholders and building necessary understanding and commitments, for closer monitoring and control and reducing the risk of leakage [9, 66, 181]. It is also seen as being

44 See the Annex 9 for a more detailed account of the sources. 45 Based on 17 comments from 14 different evaluations.



conducive for innovative approach, for testing new ideas or new way of doing things [58], as a first step towards a sector-based approach [58] and to build NSA capacities [58]. The procedure used for project selection - i.e. the call for proposal system – regularly comes in for criticism. It is felt to not contribute to building a sector policy dialogue with the partners; it is difficult to handle for CSOs; and it only gives limited autonomy to the EUD [45, 58, 159, 213]. The main weaknesses of projects mentioned include: • The cost of Programme Management Units (PMUs), which (indirectly) weaken governmental

structures. Besides, they do not always have the capacity to reorient the course of a project in the event of a changing environment (political, financial, situation of a target group), which in turns impacts negatively on the possibility to achieve the expected results [19, 108, 175];

• The fact that yet many projects remain too ambitious, and may only attract limited human resources away from strategic priorities [19, 58];

• The inadequacy of procedures that do not match the specific requirements of some sectors or actors, thereby slowing down the pace of implementation [23, 154];

• The difficulty to see the complementarity between all projects [9]; • The fact that their impact eventually depends on wider policy reform and/or local integrated

development [58].

3.6.2 Budget Support46

What emerges from the evaluation reports is that there seems to be a consensus between partner governments and donors on developing budget support at the expenses of more traditional modalities such as projects [58, 68, 105, 125, 202, 228]. Although in some situations, improvement due to the use of budget support (BS) cannot be demonstrated [9, 66, 117, 158], its growth is generally perceived as being positive, especially in enhancing intervention efficiency [11,12, 16, 51, 60, 142]. On the other hand it is difficult to attribute positive results to the EC alone [202]. Various evaluations suggest budget support tends to speed up implementation processes, especially when accompanied by targeted short-term technical assistance; directly contribute to alignment; and allows for a better identification of intervention costs [11, 13, 51, 194]. Other virtues mentioned in the evaluations include the improvement of predictability and the promotion of government ownership and democratic accountability [58, 60, 66]. In some specific situations of instability budget support can have a political stabilising role in contributing to ensure public servants income, therefore decreasing the level of social claims, and increasing the acceptance of governmental reforms [16]. It can support high levels of macro-economic growth and stability, good results in strengthening the planning, budgeting and public finance management systems, including by permitting a significant increase in the proportion of the public expenditure subject to the national budget [105]. The “variable tranche” mechanism offers a good compromise between the need to increase financial aid and the principle of progress-related support. It has allowed the EC to combine overall budget support with its commitment to focus on the social sector and the most vulnerable populations [58, 121, 164]. General budget support has been the only channel through which large scale financial support has been disbursed in time, while maintaining a connection with policy reforms through the variable tranche mechanism, and attempting to address long-term structural issues and institutional weaknesses through policy dialogue [59, 105].

46 Based on 27 comments from 19 different evaluations.



In a couple of cases, general budget support is said to have an equal or even better potential to contribute to poverty reduction, and to help the EC to establish itself as one of the more important development partners [58, 148]. However, various comments point out that budget support is not a panacea and it would gain from having a number of its shortages addressed. The transaction costs remain quite high due to the considerable coordination and monitoring efforts required, even though these costs are seen as ‘useful’ costs47 [66, 92, 105]. General budget support is also seen as the modality involving the highest leakage risks [66, 92, 165]. In addition, the predictability of disbursements has been undermined because the EC has established an all-or-nothing link between financial support and sector policy reforms [57, 59]. Sometimes, the use of BS is premature, and risks becoming counter-productive by putting too much responsibility in the hands of a government, which capacities are not enough developed to handle the task [149, 170]. Last but not least, being based on a number of conditionality, it relies on a common analysis and will to proceed from the donor community, which can be a weakening factor when their analysis of the situation varies [9].

3.6.3 Modali ty Mix48

The EC has demonstrated its capacity to make use of the different financial instruments available in a complementary manner, ensuring a better synergy between them and resulting in a higher level of commitment and disbursement. Improvements are feasible and this effort should be pursued [7, 116, 125, 175, 194, 207, 219]. The key element in adopting an adequate mix of modalities is ensuring a continuum of complementary approaches, ranging from short term, ad hoc projects, to general budget support. In other words, matching of the modalities to the phasing of development processes in a timely articulation is the best way to ensure efficiency [9, 12, 96, 98]. Or in the words of one evaluation team: ‘Real-life choices are not that much a matter of selecting one best modality, but rather of setting up an adequate mix and/or sequence of funding modalities’ [58]. Combining EC aid instruments with other horizontal instruments (for instance military interventions, humanitarian aid), channelling funding through multilateral donors (i.e. UN agencies) or through development banks can prove very helpful in addressing situations of fragility, crises or post-conflict situations[2,18, 126, 193]. More flexibility in using funding modalities and in strengthening local management skills is also a success factor in these contexts [217]. The choice of modality gains in being carefully adapted to the actor dimension of the intervention, and the context they live in. Proximity of the intervention to the target group, demand driven technical assistance based on local capacity levels, risks of financial leakages or politico-administrative culture are some of the guiding elements to be taken into consideration when choosing the best suited modality [60, 68, 73]. Some of the limits to such an approach are illustrated in a number of cases: when the financial volumes mobilised under the CSPs are too limited to allow for a strategic mix of various instruments, approaches or modalities [127]; when new aid modalities are used prematurely and end

47 Useful in the sense that they prompt policy dialogue, ownership, alignment and donors coordination, therefore they enhance the

quality of the strategy and the effectiveness of its implementation, contrary to the costs of bilateral discussion between donors and partner governments on individual projects, with their own specific procedures, and their disconnection from wider policy reform.

48 Based on 24 comments from 21 different evaluations.



up not being able to address needs [138]; or when the objectives are overambitious and their implementation requirements are underestimated [161].

3.7 Conclusion49

It is difficult to assess the poverty reduction focus out of the extracts available since it is not adopted as an angle to report on in the selection of evaluation report extracts. The few mentions50 of poverty reduction convey a mixed message: either it has been considered in a ‘superficial way’, or on the contrary, it has been taken into account according to the specificity of the country context and strategy. Mainstreaming is not specifically accounted for either in the evaluations reviewed. However, budget support, capacity development, good governance and participatory approaches are regular trends reported on. Little is said on environment and gender is mentioned only once. A few references51 to the devolution process are worth mentioning because they all indicate that it has contributed to improve Delegations performance, organizational and management capacities, even though not fully solving their work overload.

49 This conclusion is based on a global assessment of all extracts available but it takes into specific consideration 8 extracts from 7

different evaluations on the issue of devolution and poverty reduction. 50 See extracts [62, 69, 201, 218]. 51 See extracts [57, 142, 186, 219].



ANNEX 1: GRID FOR ANALYSING EVALUATION REPORTS

I : des cr ip t ive in format ion Title Focus (national, regional, or thematic with further specification) Legal instruments covered by the evaluation II : Ident i f i ca t ion o f r e su l t s Result with page number between [brackets] Sector/a

rea52 Results level in IL to which result can be related53

Impact level in IL to which result can be related54

III : inventory o f r e l evant r esu l t s in re la t ion to ins truments Based on the results featured under II, please tick the boxes below with an X for those instruments for which the evaluation includes at least one relevant result DCI ENPI IfS ICI EIDHR INSC EDF or other IV: Are some o f the r esu l t s ident i f i ed under III , f inanced through ODA and/or non-ODA sources , o f mutual bene f i t to the EU and the partners concerned (as opposed to a ‘deve lopment ob j e c t ive ’ which seeks the exc lus ive in teres t o f the partners concerned) V: Comments made in the r epor t in re la t ion to the e f f e c t iv eness o f the modal i t i e s used , and any o ther r e l evant non -resu l t in format ion that can be used for construc t ing the opt ions in phase 4

52 Where possible, please indicate at the end in brackets to which focal sector(s) or specific area(s) of work the result relates. 53 Here, please indicate the number or letter of the expected results of the legal instruments, as shown in the third column of the Intervention Logic to which you can relate the result. For example, if a

result is specified in the area of primary education for a DCI country, then add “DCI: 1” in the column. E.g. if a result in a DCI country (e.g. Bolivia) relates to higher education, add “DCI: LAM 4”. If a result relates to the promotion of governance in an ENPI country, add “ENPI: 1” (unless it is specifically mentioned in the report that ENPI or DCI thematic was used, of course).

54 Here, please indicate the number or letter of the respective impacts of the legal instruments, as shown in the fourth column of the Intervention Logic to which you can relate the result. For example, if a result is specified in the area of primary education for a DCI country, then add “DCI: 1” in the column. E.g. if a result in an ENPI country relates to economic cooperation, add “ENPI: 2”. It is possible for this column to refer to multiple intervention logics when it is not clear from the report which instrument it can be linked to (most of all): e.g. governance results in DCI countries can be linked to both ENPI and DCI regulations, governance results in ENPI countries can be linked to both ENPI and EIDHR. In those less-clear cases, please mention all references you find. The column will remain empty for EDF countries in case of ‘geographic instrument results’, but in case results relate to other instruments covering these regions in a thematic sense (i.e. DCI or EIDHR), then this can be mentioned. In those cases (ACP countries/regions), this can only be done if the report specifically refers to the other instruments.



FULL LISTING OF EVALUATION REPORTS

Year # Country # Regional # Thematic

Year: 2006 1 Armenia 35 TACIS 44 trade capacity building phase 3

2 Tanzania 36 Central Africa 45 Joint donor evaluation of General Budget Support (joint evaluation)*

3 Mali 46 EC Support to Good Governance

4 Comoros 47 Water and sanitation

5 Mauritius 48 Micro projects in ACP countries*

6 Seychelles

7 Rwanda Year: 2007 8 China 37 Central America 49 Statistical Support in ACP countries

9 Jordan 38 Pacific 50 Rural Development

10 Bolivia 39 SADC 51 Synthesis 3Cs (joint evaluation)

11 India

12 Moldova

13 Mozambique

Year: 2008 14 Guyana 40 West Africa 52 Energy

41 Eastern Africa 53 United Nations*

54 Development Banks and EIB*

55 Civil Society*

Year: 2009 15 Central African Republic

42 MEDA II 56 Conflict Prevention and Peace Building (Preliminary study)

16 Tchad 43 ASEAN

17 Laos 18 Malaysia 19 Thailand 20 Uganda 21 Angola 22 Vietnam 23 Nicaragua 24 Namibia

25 Botswana

Year: 2010 26 Burkina Faso

27 Nigeria

28 El Salvador

29 Egypt

30 Niger

31 Liberia

32 Maldives

Year: 2011

33 Senegal 57 Education

34 Ukraine



ANNEX 2: TRENDS IN MANAGEMENT, MODALITY, APPROACH AND PERFORMANCE

Evidence identified in evaluation report grids (Database table 3 on V – non result information)

This annex is an intermediary product that groups the extracts from the evaluation reports collected in the datase and organises them according to a series clusters of topics for use in the Desk Report, Part 2 Chapter 3: Review of Evidence on Implementation Issues Clusters used to group key messages emerging from the evaluation reports

1. Effectiveness & 3Cs Alignment Coherence Coordination Complementarity Synergy

2. Programme management cycle

Programming Delays M&E / exit strategy / sustainability Learning

3. Actor based dimension Participatory approach Partners capacity

4. Modality

Projects approach Budget Support Modality Mix

For the conclusion

Poverty reduction focus Devolution + remarks on mainstreaming (no specific

extract) Sources, traceability, count of extracts Global count of evaluations of extracts used for the analysis The analysis is based over 230 extracts from more than 50 different evaluations. All extracts from each theme and sub theme are available in the next pages, together with a more detailed identification of their matching evaluations.

Number of evaluations and extracts used for each theme

Theme Number of Evaluations from which extracts originate

Number of Extracts

Total Country Regional Thematic 1. Effectiveness & 3Cs 41 24 7 11 73 2. PCM 31 17 6 8 58 3. Actors based dimension 26 16 4 5 43 4. Modalities 34 24 5 5 55

Number of evaluations and extracts used for each sub-theme Sub-Theme Number of Evaluations Number of Extracts 1.1 Alignment 11 14 1.2 Coherence 7 7 1.3 Complementarity 11 15 1.4 Coordination 26 31 1.5 Synergy 21 24



2.1 Programming 13 16 2.2 Delays 17 21 2.3 M&E 19 23 2.4 Learning 10 12 3.1 Participatory Approach 15 20 3.2 Partner Capacity 17 25 4.1 Project Approach 14 17 4.2 Budget Support 19 27 4.3 Modality Mix 21 24

1. Effectiveness & 3Cs (73 extracts, from 41 different evaluations) 1.1 Alignment (14 extracts, from 11 different evaluations)55

1. CSP priority sectors/areas were aligned by design on PRSs, and remained aligned de

facto [62] 2. convergentes avec les priorités du gouvernement pendant les 7 et 8ème FED et y ont

correspondu encore plus étroitement pendant le 9ème FED [69] 3. The EC has been innovative and pace-setting in aligning its development assistance

with that of GoI. The shift has also improved EC-India co-ordination and the internal coherence of the EC’s response to India’s development challenges. The overall general trend towards alignment has been strong and allowed for qualitative deepening of the ongoing dialogue between the EC, GoI and other DPs in the two main sectors of support, health and education. [99]

4. There has been an early and increasing reliance on and alignment to domestic procedures, policies and plans, which have increased domestic ownership and commitment and reduced the aid fragmentation with its negative impact on capacity development. The stronger alignment to government structures has also allowed for comparatively speedy implementation, as only one set of procedures had to be followed. However, it is important to realise that increased alignment and relevance does not in itself guarantee more pro-poor development outcomes, as that depends on the effectiveness of the government’s programmes. [100]

5. EC support is coherent (in itself), it is complementary to other donors’ support as a result of coordination arising from EC membership of PAP and thus is closely aligned to GoM needs and priorities as expressed in PARPA. It responds to GoM wishes regarding aid delivery modalities (budget support). [110]

6. Flexible, demand-driven instruments have enabled the Commission to meet perceived national environmental needs and priorities. Reliance on bottom-up, regional instruments precluded rational policy sequencing of programmes. [129]

7. “The evaluation has shown that the EC programmes and cooperation strategies in Uganda are aligned with the GoU’s priorities as stated most clearly in the PEAP. The EC has also clearly moved away from a dependence on the project approach modality toward PBAs, implying that to a large extent, EC funds are now channelled through GoU systems, thus creating a greater harmonisation of processes, or at least a drop in duplication [134]

55 For some reason, this list starts on #2, but there is indeed 14 extracts.



8. The EC has made important advances in aligning its support to GRN structures and in setting up the institutional prerequisites for harmonising its support with that of other donors. [143]

9. EC still needs to improve the alignment of its capacity development support with capacity development plans and initiatives of the Government of Namibia [148]

10. Commission’s cooperation strategy with Egypt was found to be highly relevant to the realization of the EU-Egypt Agreements’ objective. EU policy objectives, and specifically the promotion of the objectives of the EU-Egypt association agreement, shaped the Commission’s cooperation strategy with Egypt. also became more explicitly relevant to the needs and priorities of the Government of Egypt. when the GoE adopted a reform minded policy in 2004. [160]

11. EC seeks to align its support to national and regional (governance) priorities of partner countries [204]

12. Les documents de stratégie pays de la CE sont en cohérence avec les stratégies nationales de réduction de la pauvreté (DSRP) des pays partenaires, [218]

13. major advancements have been made in terms of aligning EC support to education with partner countries’ national priorities and policies over the course of the period evaluated.. This increase in alignment occurred in conjunction with the greater use of modalities such as SBS and GBS, which, by their inherent characteristics, entail a high level of coherence with, and alignment to national policies. In cases where no education sector policies or national frameworks existed, the EC has made an effort to support their development. [230]

14. Alignment of EC funding to partner country’s fiscal has thus been frequently undermined by failure to comply with indicators essential for disbursements to be released, thus leading ultimately to a reduced alignment to the financing cycle and reduced predictability of financing. [233]

1.2 Coherence (7 extracts, from 7 different evaluations)

1. La pertinence de la stratégie régionale de la CE en Afrique centrale est satisfaisante. La

stratégie correspond dans ses grandes lignes aux objectifs globaux des organisations régionales et de la CE, mais la prise en compte des aspects de réduction de la pauvreté n’est pas évidente. choix des secteurs est bien justifié, mais au niveau des activités l’approche projet dominante ne permet pas de répondre à tous les problèmes [19]

2. Regional and national programmes are substantially coherent and provide a strong basis for complementarity that is however not supported by explicit linkages between the regional and national levels. [44]

3. Sharing views and information alone has proven to be insufficient for moving towards greater complementarity between the Member States and the Commission, as well as to improve PCD. [53]

4. il y a un effort de mise en cohérence entre la coopération de la CE et sa politique du commerce extérieur, mais cet effort repose sur une hypothèse qui n’est pas encore vérifiée en ce qui concerne le Rwanda [93]

5. . La cohérence d’ensemble des instruments et politiques de la Commission européenne semble assurée, et la démarche LRRD mise en place. Il n’y a pas de contradictions entre la stratégie pays et d’autres politiques communautaires, notamment celles concernant le commerce, l’environnement, la sécurité et le transport [118]

6. As regards the internal coherence of the Commission’s cooperation strategy, synergies between interventions (within the NIP, between the bilateral programmes and EIDHR



projects, and between regional and bilateral programmes) supporting the same objectives so as to maximise their contribution to overall objectives were generally not foreseen at design stage and occurred rarely during implementation. [163]

7. Commission policies relating to fisheries, the environment and the Overseas Countries and Territories (OCTs) have generally been fully coherent with EC development cooperation interventions, and have been drawn up to take specific account of EC policies towards developing countries and the ACP States in particular. The Commission’s participation in international conventions and its support for their ratification and enforcement by the Pacific Countries has been beneficial in stimulating coherence in views and policies on these global challenges and in adjusting the Commission’s interventions accordingly. [185]

1.3 Complementarity (15 extracts, from 11 different evaluations) 1. La complémentarité entre PIR et PIN est surtout théorique et reste à améliorer dans la

pratique. [20] 2. l’avantage théorique de la complémentarité des instruments de la Commission n’a pas

été suffisamment traduit en réalité. Le premier problème vient des faibles liens entre d’une part l’objectif global du PIR, à savoir l’insertion harmonieuse de la région dans l’économie mondiale, et d’autre part l’objectif global des PIN, c’est-à-dire la lutte contre la pauvreté. Différents projets sont financés à travers les différents instruments sans que les liens soient clairs à quelques exceptions près, notamment dans les secteurs de transport et des ressources naturelles. Le manque de complémentarité réelle est en grande partie dû à la faible coordination interne et aux faibles capacités de la Commission pour mener la stratégie régionale. [22]

3. Channelling funds through the WB and in particular through its multi-donor trust funds was in itself a form of coordination and complementarity. [41]

4. Regional and national programmes are substantially coherent and provide a strong basis for complementarity that is however not supported by explicit linkages between the regional and national levels. [44]

5. Sharing views and information alone has proven to be insufficient for moving towards greater complementarity between the Member States and the Commission, as well as to improve PCD. [53]

6. Lack of coordination, complementarity and coherence in EU humanitarian aid has a negative impact on its efficiency and diminishes the impact of the EU in global humanitarian aid policy and operations. national priorities and the preference for national visibility are more important issues in international humanitarian aid than the effective and coordinated assistance of the EU as a whole. [54]

7. So far, constraints seem to outnumber incentives in the quest for more EU coordination and complementarity at partner country and local levels (e.g. centralised procedures, unclear mandates, quality and timing of information exchange). Coordination mechanisms often lack an explicit, formal and operationally focused mandate and do not lead to joint assessment and decision-making that is binding on all the parties involved. [55]

8. Country Strategy Papers (CSPs) do not stimulate and may even obstruct progress on the 2Cs, since donor priorities can only be formally changed when new strategy papers are drawn up. This situation is improving now that different Member States and the Commission are involved in developing more flexible planning and joint-planning arrangements. [56]



9. The EC has progressively enhanced its efforts to promote complementarity during the period under evaluation. The EC has deliberately exited from the rural development sector, based on careful analysis of how it could best complement the efforts of other development partners (including GoI) and provide value added. Similarly in the sector programmes, the EC has developed capacities within specific areas that complement those of other partners and also ensure very close co-ordination and ownership by GoI. internal complementarity between interventions funded by thematic budget lines could have been improved. for all development partners, but more efforts are needed to jointly formulate strategies aimed at reducing the governance failures that significantly reduce the impact of the EC’s support for health and education [102]

10. EC support is coherent (in itself), it is complementary to other donors’ support as a result of coordination arising from EC membership of PAP and thus is closely aligned to GoM needs and priorities as expressed in PARPA. It responds to GoM wishes regarding aid delivery modalities (budget support). [110]

11. les interventions de la Commission européenne sont relativement complémentaires de celles des autres bailleurs, sur le plan géographique comme sur le plan thématique. [118]

12. Good performance in coordination and design, but not enough to really make complementarity happen [141]

13. Complementarities foreseen between the sample interventions examined and other donors’ interventions have generally not materialized [162]

14. EC support at country level is generally in harmony with EC policies but that coordination could be strengthened, in particular between Delegations and ECHO operations. Further, there are undoubtedly considerable efforts being made by Delegations, and some Member States to align their respective projects and programmes. They recognise that overlap and conflict is counterproductive and divisive, but there is a long way to go before coordination and complementarity reach acceptable levels. [208]

15. La complémentarité des interventions s’exprime principalement en termes de répartition géographique. En règle générale, le degré de spécialisation est relativement faible en matière de développement rural. [221]

1.4 Coordination (31 extracts, from 26 different evaluations)56 1. The EC has engaged in basic coordination to the extent of informal discussions,

information sharing, etc. It has attempted to avoid overlap and has successfully exploited comparative advantage (despite the need for better coordination between ASEAN and bilateral-level programmes noted elsewhere). However, there is no evidence that the EC and other major donors have coordinated policy positions regarding ASEAN, not have they engaged in joint programming. Information-sharing has not resulted inaformal division of labour in support to the ASEC. There is little evidence of coordination with the IFIs. [8]

2. Comme il y a meilleure coordination des interventions de l’ensemble des bailleurs de fonds, les coûts de transaction sont réputés plus faibles, même si la perception de cette réduction n’est pas partagée par l’ensemble des acteurs (l’absence de mesures quantifiées systématique des coûts de transaction est à souligner [14]

56 For some reason, this list starts on #2, but there is indeed 31 extracts.



3. La coordination des activités pose des problèmes, et la coordination interne, notamment, démontre des faiblesses. De plus, la coordination entre les bailleurs de fonds de l’Union Européenne et autres est surtout informelle et ne correspond ni aux normes prévues dans la déclaration de Maastricht sur les trois C ni à la Déclaration de Paris [20]

4. The level of coordination between the EC and Member States is rather low. There are sporadic meetings to request MS for inputs in the elaboration of strategy documents but these are normally late and with no impact. Although there many other international donors and a lack of operative coordination, the programs are apparently well complemented. The unique case where programs are potentially overlapped would be for the support to an institutionalized regional integration. [23]

5. Channelling funds through the WB and in particular through its multi-donor trust funds was in itself a form of coordination and complementarity. [41]

6. Actual coordination remains weak. [44] 7. Energy interventions are provided at bilateral and regional levels; the degree of coherence

and complementarity between these levels varies. In some cases the absence of a regional institution impacts on ownership and on co-ordination with bilateral actors. The technical capacity of the regional counterpart was also a key to optimising the regional approach. [46]

8. The Commission is a major actor in nuclear safety, where it has extensive co-operation with the other active donors. Member States can be in competition for security of supply, which affects dialogue and co-ordination in the energy sector as a whole. Energy has not been a focal sector of successive EDFs. This discouraged the EC Delegations from participating in sector dialogue or co-ordinating with the partner and other donors. [48]

9. The report concludes that sector policy reforms have been successfully fostered when all development partners acted consistently, and vice-versa. Coordination is even more necessary in the instance of flexible progress-related approaches like the variable tranche, which is intrinsically weaker than traditional approaches in terms of incentives. [53]

10. Lack of coordination, complementarity and coherence in EU humanitarian aid has a negative impact on its efficiency and diminishes the impact of the EU in global humanitarian aid policy and operations. national priorities and the preference for national visibility are more important issues in international humanitarian aid than the effective and coordinated assistance of the EU as a whole. [54]

11. So far, constraints seem to outnumber incentives in the quest for more EU coordination and complementarity at partner country and local levels (e.g. centralised procedures, unclear mandates, quality and timing of information exchange). Coordination mechanisms often lack an explicit, formal and operationally focused mandate and do not lead to joint assessment and decision-making that is binding on all the parties involved. [55]

12. The EC has contributed to a large extent towards harmonised dialogue with the Tanzanian Government and all development partners. The lead that it has, and is continuing to take in the various working groups and other mechanisms for dialogue with the Government, shows a strong commitment to supporting the Tanzanian approach to coordination. EC contributions in the harmonised dialogue have been directed towards sector strategies and sector budget support. [64]

13. The EC has entered into all the right commitments and attended all the right meetings in the area of inter-donor communication. However, with the notable exception of the FSP, the EC has not devoted substantial resources to building capacity for coordination. [104]

14. There is no ascertained evidence that the Commission has built its strategy in close coordination with the GoG and other donors [115]



15. Le niveau de coordination entre gouvernement et bailleurs et entre bailleurs-mêmes varie selon les domaines. Tant que, par manque de capacité, l’appropriation de la coordination par le partenaire national n’est pas institutionnalisée et effective, la coordination entre bailleurs reste informelle et insuffisante [118]

16. There is no evidence that the EC exploited its role as a supranational organization to address issues that bilateral donors, including European MSs found too sensitive [140]

17. Good performance in coordination and design, but not enough to really make complementarity happen [141]

18. There appears to have been no overall complementarity with the support from other cooperation partners, although complementarity was evident in relation to DFID. Coordination mechanisms with the other cooperation partners existed, this being a priority for the ECD. But according to those partners, the ECD was not very active in the coordination efforts. In contrast with the ECD, local representatives of the EU Member States interviewed by the Team found only limited added value from the EC cooperation. There was no consensus among the Member States about the adequacy of EC support of a political nature [156]

19. Several donor coordination mechanisms have been established at strategic and implementation levels between the Commission and EU MS as well as with other donors. Involvement of EU MS in the CSP/NIP formulation has progressively improved while coordination among the donor community within the Development Partners Group (DPG) was variable across sectors. Coordination with other donor-funded interventions has not been the general rule except for BS interventions. [162]

20. Donor coordination is mainly restricted to exchange of information and has not prevented overlaps between projects- [178]

21. poor levels of cooperation between participating States. Moreover, in the areas bordering the EU problems have arisen in coordinating the different instruments applicable in the border States in a manner detrimental to both efficiency and effectiveness. [181]

22. The Commission also plays an active role in the informal aid coordination processes in the region. These processes are of considerable importance in preventing duplication of activities at operational level. Commission regional interventions have generally been complementary to other donor support. . [185]

23. “EU co-ordination mechanisms for TCB at HQ level are limited to exchanging information on EU donor programmes and activities in TCB. Co-ordination goes little further than this. Reticence is evident on the MS side on the adoption of common EU guidelines for TCB: despite the Commission’s efforts and the Council’s Communication on Co-ordination, these Commission-designed guidelines are not taken into consideration by the MS. But even at the most basic level of co-ordination, namely information sharing, weaknesses exist: the information flow is essentially a one-way flow, from the Commission to the MS, with low involvement of most MS. There is little information on what is achieved in the field. There is no continuous or structured link between HQ-based co-ordination mechanisms for TCB and corresponding in-country mechanisms. In the programming of TCB, co-ordination is not designed around a specific EU framework, but around multilateral initiatives. Mechanisms such as the IF process, the WTO Trust Funds and the WTO accession plans facilitate a participative and complementary approach between all participants. EU donors tend to act autonomously [197]

24. At field level, evidence suggests that EU co-ordination has not succeeded in offering a more consistent and coherent framework for TCB interventions [198]



25. “The Commission is seldom perceived as the relevant level for co-ordination of TCB activities. In practice evidence that these potential gains from coordination have materialised is scarce. To the extent that there have been gains it is mainly from the co-ordination process involving all donors (IF; preparations for WTO accession), rather than from any specific EU co-ordination [199]

26. The EC has purposefully designed the steering of the activities it carries out in governance in close interaction with other agencies, donors and partner country governments [205]

27. Overall, coordination with other donors had been very limited or even non-existent when implementing diagnostic analysis, although other donors were active in the field and had contributed to formulation of national statistical strategies. The establishment of appropriate linkages between regional and national statistical programmes was achieved in regions where both types of programme were being implemented. [215]

28. The EC participates in donor co-ordination, and its support is well co-ordinated with the interventions of member states and other donors. Also, implementing EC funds in co-operation with the World Bank and UN agencies strengthens the co-ordinated spending of funds and reduces administrative procedures… The EC took a lead function in co-ordinating the democratic transition. [216]

29. En pratique, des progrès significatifs ont été réalisés en matière de coordination entre bailleurs. Ces améliorations se manifestent au sein des groupes de bailleurs de fonds où la CE joue souvent un rôle important. Toutefois, les actions se limitent généralement à des échanges d’informations et il y a peu d’interventions cofinancées. Ces groupes sont parfois impliqués dans le dialogue politique avec le Gouvernement. L’absence de consensus sur le champ du développement rural entre les bailleurs eux-mêmes, et entre les bailleurs et le gouvernement, l’absence de stratégie de développement rural opérationnelle du gouvernement, ainsi que le manque de coordination interministérielle expliquent les limites de la coordination dans le domaine du développement rural. Il est à noter que l’absence de coordination est source de difficultés pour la mise en œuvre de certaines opérations et entraine des impacts négatifs dus à des approches contradictoires entre bailleurs [220]

30. There is no evidence that coordination between the Commission and the EU Member States is improved when the Commission channels its funds via UN bodies. However, there is evidence that channelling often enables the Commission to take a leading role in donor coordination and that in practice it does so [226]

31. While EC support to FTI has increased towards the end of the period under evaluation, co-ordination, at the country level, between support provided by the FTI and EC support must be described as very limited [234]

1.5 Synergy57 (24 extracts, from 21 different evaluations) 1. The adoption of the Linking Relief Rehabilitation and Development Strategy as a

guiding strategy in all sectors proved a major contribution to the timeliness and effectiveness of EC interventions and was clearly the most important feature of the EC intervention in Angola. The fact that it was initiated by ECHO interventions and then continued through budget line and special interventions (e.g. PEAPP) is highly relevant; it generated positive lessons learned in all sectors of intervention and ensured that immediate postwar needs were satisfied [2]

57 The 4 comments originally under articulation national / regional levels have been integrated in this part on synergy.



2. better coordination between country-level and ASEC-level ASEAN programmes, and between ASEAN-level cooperation and bilateral cooperation, would have resulted in synergies and improved the overall effectiveness of the EC cooperation programme. local Commission staff have hardly any information on projects still managed from the Brussels Headquarters. [7]

3. le domaine d’application de la politique budgétaire du gouvernement central et celui de la mise en oeuvre des politiques appuyées ne sont pas identiques suite à la décentralisation et la régionalisation. Dans ce cas, l’instrument ne sera pas efficace sauf si la matrice de conditions et résultats à atteindre incorpore le niveau décentralisé. Les interventions sont peu coordonnées et articulées entre elles. Les différents instruments de coopération et même les différentes interventions ne sont pas utilisés en vue de possibles complémentarités et synergies, mais sont gérés de façon indépendante. l’instrument ne sera pas efficace sauf si la matrice de conditions et résultats à atteindre incorpore le niveau décentralisé. Si le GdB a la volonté et la capacité de le faire, il est tout à fait possible d’incorporer les municipalités et/ou les gouvernements régionaux dans les discussions de politique publique et de respecter le rôle de chacun des acteurs institutionnels. [9]

4. issues related to good governance like combating corruption, transparency or accountability of the institutions were only analysed at national level despite the important role of credibility in the process of integration. [24]

5. joint approach of national and international partners and the higher level of funding, the intended effects are likely to be achieved. [25]

6. Synergies between the bilateral EC programme, the regional programmes and other EU policies or EIB are not fully developed [37]

7. The Commission’s channelling of funds through the WB and EIB has over recent years represented a growing and significant part of the Commission’s aid delivery. It brought added value to the different parties involved, notably by allowing the Commission to intervene in certain sectors and countries with a critical mass of resources and expertise [38]

8. Energy Support has mainly been channelled through specific projects which were seldom integrated into a sectoral approach with strategic objectives. provided at bilateral and regional levels; the degree of coherence and complementarity between these levels varies. In some cases the absence of a regional institution impacts on ownership and on co-ordination with bilateral actors [45]

9. There were few linkages between EC regional and country strategy papers, between regional and national interventions, and between AIDCO and ECHO activities in post-Mitch reconstruction. Coordination with the government and the international community was marked by little national ownership and a weak donor framework, with consequently limited results. Overall EC coordination with EU MS has been weak over the last decade, owing notably to the limited EC presence in the country [52]

10. coordination between the support provided to NSAs and the support to the health SWAP needs to be improved [109]

11. Indeed, it is not shown how different interventions are inserted in the overall strategy and contribute to precise global objectives. resulted in a basket of activities with few links between them so that potential synergies and complementarities with other Commission policies and instruments have not been recognised and optimized [113]

12. Non-programmable instruments have even been mobilised independently of the 8th NIP and 9th CSP strategic orientations. As a result, dispersion of funding is significant in Guyana in spite of the definition of areas of concentration with the aim of avoiding such scattering [114]



13. international NGOs act as partners to national organisations. potential of such partnerships to foster the development of a truly autonomous civil society has been limited [124]

14. Flexible, demand-driven instruments have enabled the Commission to meet perceived national environmental needs and priorities but the lack of horizontal/ thematic coordination at national level has prevented the realization of environmental projects tailored to Malaysia’s specific stage of development. Reliance on bottom-up, regional instruments precluded rational policy sequencing of programmes. Instead, environmental projects emerged from existing networks and activities [129]

15. no evidence that the EC exploited its role as a supranational organization to address issues that bilateral donors, including European MSs found too sensitive [140]

16. Subsequent addition of several non-focal interventions has strained the management resources of both the ECD and its counterparts and led to an overall reduction of efficiency in terms of aid delivery and programme implementation. [154]

17. In support to civil society, there appears to have been little or no coordination of decisions in support of interventions originating from EDF funds and those from Brussels-based thematic budget lines. Such coordination would allow a higher degree of concerted action and as well as adherence to the same MER procedures, making possible increased and much needed knowledge of the results of the budget line projects beyond the output level [155]

18. La mise en cohérence des interventions sur budget et sur FED présente parfois des difficultés. Certaines initiatives financées sur budget CE sont gérées à Bruxelles (exemple : Facilités ACP-UE Eau ou Energie). Dans ces cas, la Délégation a un rôle d’avis qui n’engage pas le siège. Comme évoqué lors du focus group relatif aux instruments d’aide des DPE, ces initiatives décidées par le siège peuvent étendre le champ d’intervention sectoriel de la CE au-delà des domaines de concentration définis par le CSP ce qui peut potentiellement affecter la cohérence de la stratégie communautaire dans le pays. [165]

19. EU problems have arisen in coordinating the different instruments applicable in the border States in a manner detrimental to both efficiency and effectiveness. [181]

20. In the sector of environmental protection and management of natural Resources, TACIS support has tended to be more effective when working in collaboration with IFIs, most notably the EBRD, [183]

21. CAs allowed improved definition of responsibilities and ownership by project implementers, as well as exploration of synergy effects arising from mandating of Regional Organisations. [186]

22. enabling environment for trade and regional economic integration are about to start. However, the most critical problem identified is the lack of domestification – national level interests continue to take precedence over the regional level, and there is no guarantee that decisions taken at the regional level will be implemented nationally. By the same token, the CSP/NIPs also do not take into account the regional dimension. [189]

23. Incompatibility between the EC’s management, financial and operational rules and Member States: [209]

24. EC support to education is generally coherent in terms of the various DGs involved, the various financing modalities employed, and the various levels at which it was targeted. However, room for improvement remains, particularly in terms of coherence between the geographic and thematic instruments in support of education. [231]



Theme 1: EFFECTIVENESS & 3Cs

1.1: Alignment 1.2: Coherence

1.3: Complementarity 1.4: Coordination

1.5: Synergy

Number of extracts analysed

Reference of the extract

(see excel table) Name of evaluations

Number of

evaluations

Theme

1.1: Alignment 1.2: Coherence

1.3: Complementarity 1.4: Coordination

1.5: Synergy

Total

1 2 Armenia 1 1.5 2 7 ASEAN 2 1.5 3 8 = 1.4 4 9 Bolivia 3 1.5 5 14 Burkina Faso 4 1.4 6 19 Central Africa 5 1.2 7 20 = 1.3 1.4 8 22 = 1.3 9 23 Central America 6 1.4 10 24 = 1.5 11 25 China 7 1.5 12 37 CPPB 8 1.5 13 38 Development Banks 9 1.5 14 41 = 1.3 1.4 15 44 East. & South. Africa 10 1.2 1.3 1.4 16 45 Energy 11 1.5 17 46 = 1.4 18 48 = 1.4 19 52 El Salvador 12 1.5 20 53 Synthesis 3Cs 13 1.2 1.3 1.4 21 54 = 1.3 1.4 22 55 = 1.3 1.4 23 56 = 1.3 24 62 Tanzania 14 1.1 25 64 = 1.4 26 69 Mauritius 15 1.1 27 93 Rwanda 16 1.2 28 99 India 17 1.1 29 100 = 1.1 30 102 = 1.3 31 104 Moldova 18 1.4 32 109 Mozambique 19 1.5 33 110 = 1.1 1.3 34 113 Guyana 20 1.5 35 114 = 1.5 36 115 = 1.4 37 118 Chad 21 1.2 1.3 1.4 38 124 Laos 22 1.5 39 129 Malaysia 23 1.1 1.5 40 134 Uganda 24 1.1 41 140 Vietnam 25 1.4 1.5 42 141 Nicaragua 26 1.3 1.4



43 143 Namibia 27 1.1 44 148 = 1.1 45 154 Nigeria 28 1.5 46 155 = 1.5 47 156 = 1.4 48 160 Egypt 29 1.1 49 162 = 1.3 1.4 163 = 1.2 50 165 Niger 30 1.5 51 178 Ukraine 31 1.4 52 181 TACIS 32 1.4 1.5 53 183 = 1.5 54 185 Pacific 33 1.4 55 186 = 1.2 1.5 56 189 SADC 34 1.5 57 197 Trade & Cap. Dev. 35 1.4 58 198 = 1.4 59 199 = 1.4 60 204 Good Governance 36 1.1 61 205 = 1.4 62 208 Water & Sanitation 1.3 63 209 = 1.5 64 215 Statistics 37 1.4 65 216 Maldives 38 1.4 66 218 Rural Development 39 1.1 67 220 = 1.4 68 221 = 1.3 69 226 United Nations 40 1.4 70 230 Education 41 1.1 71 231 1.5 72 233 1.1 73 234 1.4 73 different extracts for theme 1

14 extracts 1.1 7 extracts 1.2 15 extracts 1.3 31 extracts 1.4 24 extracts 1.5

41 evaluations for 1 11 evaluations for 1.1 7 evaluations for 1.2 11 evaluations for 1.3 26 evaluations for 1.4 21 evaluations for 1.5

Total



2. Programme management cycle (58 extracts, from 31 different evaluations) 2.1 Programming (16 extracts, from 13 different evaluations) 1. gamme des activités est complète. Par contre, il manque une priorisation des activités

par domaine, de manière à éviter le saupoudrage. [34] 2. Les modalités de mise en œuvre des interventions de la CE donnent la priorité à la garantie

de bonne utilisation des fonds, ce qui retarde et complique de manière non proportionnée la réalisation des objectifs des interventions, particulièrement en termes d’impact et de durabilité. Ces limitations sont liées pour une part au pourcentage insuffisant de l’aide mis à disposition du dispositif de programmation, de mise en œuvre et de suivi pour augmenter ses capacités, notamment par l’externalisation. Cela entraîne des retards et faiblesses dans la programmation et la préparation des interventions, sans apporter des garanties de bonne fin et de partenariat proportionnelles. Les délais inhérents à la sélection des prestataires ou à l’achat des équipements ne sont pas anticipés et mettent sous pression les équipes dès lors qu’elles sont en place, les incitant à faciliter les décaissements plutôt que les dynamiques participatives ou de dialogue de politique. Les ressources financières et physiques sont généralement suffisantes. [35]

3. Decisions to channel through the WB were sound, but were neither always explicitly motivated, nor based on formal guidance criteria, decisions were taken on a case-by-case basis and were not grounded in specific strategy documents, these decisions were generally based on sound analyses relying on joint donor assessments. channelling through the EIB, decisions were explicitly motivated and based on specific criteria at the level of the financing “instruments” was grounded in long-term strategic EU cooperation agreements and included a prior analysis of needs. [39]

4. doubts about the capacity of the EC strategy-making process to foresee and to address future challenges. ritual exercise which has lost its interest once funds were programmed. This has not prevented the EC from successfully addressing cutting-edge issues like in the area of trade, something which was not anticipated in the strategy paper. [61]

5. Les processus et les produits de l’analyse stratégique et de la programmation ont primé sur le sens et les résultats. Le dialogue de politique a eu une qualité très inégale, et est parfois resté largement formel, du moins au niveau de la stratégie-pays. [70]

6. La quantité comme la qualité des éléments d’information et d’analyse des DSP élaborés spécifiquement pour définir la stratégie de réponse ont été insuffisantes pour fonder de manière crédible une vision communautaire des politiques et des bonnes pratiques donc un dialogue constructif avec le gouvernement comme avec les autres parties prenantes. La délégation comme le siège n’ont pas eu le volant de ressources financières nécessaire pour mobiliser des appuis spécifiques en expertise et en savoir-faire pour éviter une certaine marginalisation de la démarche stratégique [71]

7. Il a manqué dans la stratégie-pays une gestion par résultats plutôt que par séquence de programmation ; la détermination d’un changement de secteur de concentration n’est pas argumentée sur la base des résultats acquis (ou de l’impossibilité d’atteindre les résultats attendus), mais plutôt sur la pertinence d’un nouveau secteur de concentration. L’entrée dans un secteur de concentration est rarement anticipée et préparée dans la période de programmation précédente. Le désengagement progressif d’un secteur de concentration n’est pas envisagé dans la période de programmation suivante. Les volumes des PIN des petits pays insulaires n’autorisant qu’un secteur de concentration, le phénomène est amplifie [72]



8. Les processus et les produits de l’analyse stratégque et de la programmation ont primé sur le sens et les résultats. Le dialogue de politique a eu une qualité inégale, et est parfois resté largement formel, [80]

9. The analysis of the Commission strategy has shown that its different elements are relatively independent. Indeed, it is not shown how different interventions are inserted in the overall strategy and contribute to precise global objectives. This is particularly the case for sea defences for which social, economical and environmental impacts are not sufficiently reflected at strategy level. (p. 69) The choices of focal areas and interventions under the 8th and 9th EDF have resulted in a basket of activities with few links between them so that potential synergies and complementarities with other Commission policies and instruments have not been recognised and optimized [113]

10. Non-programmable instruments have even been mobilised independently of the 8th NIP and 9th CSP strategic orientations. As a result, dispersion of funding is significant in Guyana in spite of the definition of areas of concentration with the aim of avoiding such scattering [114]

11. Low adherence to ownership principles (page 41): “the EC could have been more pro-active in putting in place a genuine joint-ownership and “ownership transfer” process when it was putting in place the pillars for the 10th EDF; it was not conducted within what could be called “joint” programming but was seen by officials and civil society as being essentially carried out unilaterally by the EC in Brussels. [135]

12. financing modalities and amounts were decided by the Commission’s headquarters at a very early stage of the programming exercise and the process by which they were arrived at was not based on an explicit argumentation; [159]

13. The European Commission’s procedures and mechanisms for design, planning and implementation did not adequately take into account the difficult conditions, vagaries, risks and operational constraints of Liberia’s post-conflict environment. [167]

14. A combination of limited contextual analysis and dialogue with beneficiaries has led to projects being of poor design in terms of achievable results, a slow rate of contracting, and poor coordination between projects in the same sector. These constraints and a lack of a coherent medium-long term approach agreed with partners puts at risk both the wider impact of the good results achieved- in terms of institutional development (training) and development of specific models- and sustainability since the project duration and cycle unlikely matches a country owned pace of structural reform. [180]

15. causes de la pauvreté et de sa dynamique ainsi que les besoins essentiels des bénéficiaires ne sont pas toujours clairement identifiés. Et si cette identification est correcte, les solutions proposées pour y répondre restent peu opérationnelles. [218]

16. Little information could be found on any analysis of possible alternatives prior to the decision, but often channelling through the UN appeared to be the only option. The decisions to channel through the UN were furthermore not based on explicit guidelines. Systematic guidance on the conditions under which channelling through the UN or through a specific UN body is recommended is exceptional. [224]

2.2 Delays (21 extracts, from 17 different evaluations) 1. some delays in implementation were incurred at the ASEAN level as a result of

financing delays. [6] 2. De grands délais dans la formulation et la mise en oeuvre des projets et programmes

ont été constatés. Ces délais créent des surcoûts [19]



3. Concernant les aides projets, il est à noter les retards dans la mise en oeuvre induisant des retards de décaissements et des reports budgétaires. Cependant, cet état de fait est surtout révélateur d’une trop grande ambition des programmes ou d’une inadéquation des procédures et modes de faire aux spécificités des secteurs. [23]

4. delays in the implementation of programs and the lack of social organizations participating in the program cycle of operations [24]

5. L’efficacité du programme santé (8ème FED) a été fortement affaiblie par des pertes matérielles (pillages, vols liés aux troubles) et des retards liés à des conflits de personnes, eux-mêmes liés à une gestion irrationnelle des postes au sein du projet et à des renouvellements continuels, compréhensibles dans le contexte d’agressions. La responsabilité de la gestion des procédures n’a pas été attribuée de manière cohérente à l’intérieur de l’équipe d’assistance technique, induisant de nouveaux retards. [25]

6. les délais de mobilisation de l’assistance technique comme les profils effectivement mobilisés constituent déjà un facteur de risque pour l’efficacité du programme. [34]

7. Delays in contracting or implementation were reported in about half of the cases. channelling: (i) difficulties in reaching WB-Commission agreements; (ii) inadequate reporting by the WB; (iii) administration of the TF (time-consuming coordination and donor constraints, WB management); (iv) weak capacity of governments; and (v) heavy conditionalities (HIPC). Funding EIB-managed instruments proved efficient in cost terms, but delays were frequent for interest rate subsidies [40]

8. Delays and lack of transparency were reported when decisions come from headquarters level. [45] 9. Country Strategy Papers (CSPs) do not stimulate and may even obstruct progress on

the 2Cs, since donor priorities can only be formally changed when new strategy papers are drawn up. This situation is improving now that different Member States and the Commission are involved in developing more flexible planning and joint-planning arrangements. [56]

10. The Tanzanian innovative approach to harmonised policy dialogue is unquestionably effective, but it is still too early to qualify it as a success story. In this framework, the EC Delegation has made important contributions. It has played a key role in assisting the Government with developing sector strategies and sector budget support for agriculture, education, roads and trade. In these sectors and in several other areas, the EC Delegation shows a strong commitment in supporting the Tanzanian approach to co-ordination. These achievements could not have taken place without the recent de-concentration process and the associated growth in the Delegation’s staff. The movement towards sector policy reforms has, however, been generally slow, which delayed the introduction of the sector budget support. Effective policy dialogue has been impeded by weak institutional frameworks within the Government. Although the EC has wisely connected substantial financial support to progress towards policy reforms, its commitment to the JAS process has not always succeeded in ensuring consistency in development partners' actions. This inconsistency has resulted in under-effective dialogue in areas like education and agriculture. Due to slow progress in policy dialogue, a part of the support has been delayed. This stems from the fact that financial support and policy reform have often been related in an “all-or-nothing” way1. Such an approach has been successful in promoting a major policy reform in the road sector in the late 90’s, but the success has been paid a high price in terms of delayed disbursement, and the pace of reforms has not been maintained during the following years. In several instances, the financial support has been interrupted, although significant poverty reduction effects could have been achieved in the absence of the awaited policy reform. In addition the “all-or-nothing” approach collides with the willingness of development partners, and EC in particular, to increase their financial support significantly. The more flexible approach of “variable tranche”, as applied to



general budget support over the last two years, is likely to be an effective alternative. Its actual effectiveness in encouraging policy reforms has, however, not yet been proved, as shown in the area of fight against corruption [57]

11. EC strategy-making process has failed to ensure a timely and coherent implementation. [62]

12. Implementation of interventions is constrained by rigidity of procedures. Delays and lack of transparency were reported when decisions come from headquarters level. [66]

13. La programmation s’est avérée trop ambitieuse par rapport au volume des PIN. La prise en compte de coûts fixes de mise en œuvre et de suivi-évaluation des projets et des délais inhérents aux procédures n’a pas été suffisamment considérée pendant l’élaboration de la programmation. [82]

14. Les délais d’instruction et de démarrage de l’exécution des projets ont été très importants, mais ils sont acceptés comme une contrainte nécessaire par le gouvernement et, dans une moindre mesure, les bénéficiaires (déficit de communication). Il en va de même pour les durées exigées par la plupart des procédures d’appel d’offres (hormis pour les fournitures, pour lesquelles il y a inadéquation entre la lourdeur des procédures et le volume des achats correspondants). L’effectivité de la contrepartie en termes de sécurisation de l’emploi des fonds publics est reconnue et appréciée. [106]

15. Initial delays in implementation and management capacity constraints of the pilot project in justice sector were due to lack of clear definition of the role of the involved institutions and the complex nature of the sector [107]

16. Implementation delays are not analysed and corrective actions not taken in due time. [152]

17. Lead time from project identification and formulation to implementation; § Overambitious objectives of some programmes; § Complexity of procedures; and § Weakness of human capacities and institutions. [161]

18. Procedural delays prevented the EC from providing timely support in crucial post-war / post election years, affecting primarily education, but also infrastructure (electricity, water) and county development. [168]


20. La lourdeur des procédures et les difficultés inhérentes à la coopération dans un cadre régional ont pour conséquence une exécution très lente des programmes. [191]

21. implementation delays, and some from under-budgeting or overambitious planning. [217]

2.3 M&E / exit strategy / sustainability (23 extracts, from 19 different evaluations) 1. the weakness of monitoring systems and baseline data has made it difficult to gauge

impact and sometimes even to identify it properly. also attributable to a lack of continuity of Commission interventions, which addressed several issues (employment, health, social security) with ambitious projects which, however, were neither allocated



adequate resources nor followed up by subsequent projects that might have consolidated their results. [2]

2. Des efforts sont encore nécessaires pour mieux mesurer les résultats et améliorer la visibilité et la transparence de la chaîne des résultats. De même une meilleure appropriation des mécanismes de mise en oeuvre de l’ABG et de l’ABS devrait conduire à une plus grande maîtrise et connaissance des coûts de transaction par les acteurs qui les supportent. [12]

3. evidence of the lack of indicators and monitoring systems. [24] 4. Les institutions qui gèrent les différents instruments de financements disposent de

leurs propres services et procédures d’évaluation. Les modalités d’accès et d’échanges d’information ne sont pas assez clairement définies pour assurer l’accès aux données et documents nécessaires pour répondre de manière precise [36]

5. absence of an exit strategy affects sustainability. Continuity in interventions proved a success factor. [45]

6. In some cases the absence of an exit strategy affects sustainability. Continuity in interventions proved a success factor. [66]

7. Le dispositif de suivi a été une faiblesse majeure de tous les projets sous revue. Les dysfonctionnements se retrouvent : - dans l’absence de système d’alerte en cas de blocage interne, l’incapacité de mesurer les effets du projet, que ce soit par manque de situation de référence ou par insuffisance d’investissement dans la conception et la mise à jour des indicateurs prévus dans le cadre logique. Le dispositif de suivi est nécessairement associé au cadre logique des projets, or le constat de faiblesse conceptuelle et pratique de cet exercice central de programmation et de préparation des projets est récurrent. Cette inadaptation est particulièrement marquée pour la définition des indicateurs objectivement vérifiables qui sont identifiés de manière très rudimentaire. rarement fait par exemple un état de la situation de référence qui est pourtant le fondement même de la validité du dispositif pour le monitoring comme pour l’évaluation [77]

8. Le dispositif de suivi a été une faiblesse majeure de tous les projets sous revue. Les dysfonctionnements se retrouvent: - dans l’absence de système d’alerte en cas de blocage interne (conflits au sein des équipes chargées du projet ou avec l’administration) qu’en cas de d’interférences externes (octroi des prêts sans respecter les critères initialement convenus, pressions pour défendre les intérêts d’un concessionnaire ou d’un prestataire) ; - dans l’incapacité de mesurer les effets du projet, que se soit par manque de situation de référence ou par insuffisant investissement dans la conception et la mise à jour des indicateurs prévus dans le cadre logique. [86]

9. L’absence d’une représentation permanente de la Délégation aux Seychelles a constitué un facteur aggravant pour les dérives des projets, que cela soit celui d’appui aux PME ou d’Anse Royale. Un phénomène de perte de contact s’est opéré qui met en lumière l’inadéquation des dispositifs de suivi des projets et la partie informelle et relationnelle du suivi dans des conditions normales, c'est-à-dire de présence sur place d’un représentant de la CE. Le budget de déplacements et la charge de travail des responsables en Délégation n’a pas permis d’assurer sur la période une présence de plus de 2 ou 3 semaines par an. Le choix du gouvernement de ne pas divertir les ressources limitées du PIN des réalisations concrètes pour disposer d’une cellule d’appui à l’Ordonnateur National a renforcé la distanciation d’avec la Délégation et réduit considérablement le dispositif de suivi. Ce choix est justifié par le volume des ressources disponibles sur PIN (de l’ordre de 4 M€). Le suivi n’a pas été mis en œuvre par l’administration par défaut de capacité, de ressources internes [87]



10. Le dispositif de suivi est déjà établi comme partie intégrante des projets et a déjà donné lieu à de nombreux travaux et documents pédagogiques. Le problème est celui de la motivation des responsables des projets au niveau de la conception et de la réalisation dans la priorisation de leur plan de charge. S’il n’a pas été résolu jusqu’ici, posant un sérieux problème de justification de l’utilité même de l’aide communautaire auprès du public, c’est que le suivi-évaluation doit être pris en charge par d’autres responsables que ceux auxquels il est affecté actuellement. Il faut donc que la fonction soit spécifique en positionnant en Délégation un responsable du suivi évaluation, éventuellement fonctionnellement rattaché à l’Unité d’évaluation au siège de la CE. Les cellules d’appui à l’Ordonnateur National ont démontré (Comores, par exemple) pouvoir apporter une contribution significative à l’amélioration des dispositifs de suivi. Dans les pays comme les Seychelles, un dispositif similaire, même aussi réduit qu’un seul assistant technique, serait utilement mis en place et devrait faire l’objet d’une position forte de la CE dans le dialogue de programmation. Le dispositif de suivi se base pour l’essentiel sur la qualité du cadre logique des projets. Il faut mettre en place, en plus des modalités actuelles de vérification de la qualité, un dispositif de contrôle de la qualité du cadre logique et du caractère viable et cohérent avec la finalité du suivi-évaluation des indicateurs proposes. [90]

11. reconnue et appréciée. Il n’en va pas de même pour la qualité de la conception des projets, particulièrement en termes de dispositifs de mise en œuvre et surtout de suivi-évaluation [106]

12. Les règles de durée de vie des projets et les délais d’instruction et démarrage raccourcissent la période de mise en œuvre d’activités généralement trop ambitieuses. Elles mettent sous pression les unités de gestion des projets qui se focalisent alors sur les décaissements plutôt que sur les processus, la démarche participative, le dialogue, avec un impact négatif sur les résultats et surtout sur la durabilité des interventions de la CE. [107]

13. Monitoring, Evaluation, Reporting Strategic recommendations by monitors and evaluators are largely ignored or are addressed only by employment of additional Technical Assistance. There is generally insufficient monitoring. The focus of both ECD and NPC is more on contracting than on monitoring of results. The changing policy environment is not monitored and programmes lack the option of addressing it. Programmes tend to get stranded (e.g. SRIP, the final evaluation of which describes this phenomenon in detail). No appropriate indicators for State and sector governance are defined or addressed in the intervention logic. Mid-term reviews are geared to operational issues rather than to ensuring emphasis on monitoring of results and impacts. Dissemination is restricted and evaluation of recommendations not thorough enough. Programmes are not designed to include benchmarks and adjustment according to results. [152]

14. results of the various interventions have been difficult to capture due to (i) interventions being mostly focused on the measuring and monitoring of activities rather than of results and impacts and (ii) attribution problems [157]

15. le dialogue sectoriel de ces dernières années n’a pas permis le renouvellement du PST 2 arrivé à échéance en 2004 ce qui crée un vide important pour le cadre stratégique du secteur. Par ailleurs, les arbitrages concernant le financement du secteur, qui favorisent les travaux de réhabilitation au détriment de l’entretien courant, soulèvent le problème de la durabilité du financement du secteur. En dépit de la création récente du FERA qui vise à la sécurisation des ressources dédiées à l’entretien, la CE et le Sénégal n’ont donc peut‐être pas suffisamment investi les questions de gouvernance et de durabilité dans le cadre du dialogue sectoriel mené ces dernières années. [174]




17. De façon générale, la CE n’est pas en mesure de connaître les effets de ses interventions de soutien parce qu’elle n’a pas les systèmes de suivi adéquats en place. [192]

18. when withdrawing from the rural development sector, the EC did not have a clear strategy and approach to ensure that the continuity of benefits of its projects would be assured and the rich expertise and experience in the RD sector safeguarded and shared with other actors. [193]

19. no clear assessment of the outcomes since most interventions have been designed without factoring-in the means of measuring their effects on the final beneficiaries. [196]

20. Related efforts to support line ministries and the NPC were generally not backed up by appropriate results frameworks [202]

21. With respect to monitoring systems, those designed at programme level are generally weak, while standard external monitoring systems are generally comprehensive, albeit with some limitations. [212]

22. Monitoring of programmes was carried out, but was hampered by weak consistency of planning documents. [217]

23. En renforçant les équipes au niveau de la délégation, le suivi régulier des opérations est mieux assuré et les mesures correctives peuvent être prises plus rapidement. [219]

2.4 Learning (12 extracts, from 10 different evaluations) 1. impact could have been greater if examples of best practice had been better

disseminated [2] 2. Un aspect positif est que la Commission et les partenaires tirent régulièrement des leçons

de ces expériences, mais cet apprentissage pourrait être plus systématique. [19] 3. Knowledge management is neither formally structured nor systematised. In ACP

countries Delegations are not always equipped to manage energy-related interventions or to conduct sectoral dialogue with the partner and donors. [66]

4. The shift from project to sector budget support is not yet fully mastered In two instances (support to SMEs and poor municipalities), a part of the accumulated experience and knowledge has been lost during the shift from EC project / programmes to Jordanian institutions. The process of transferring knowledge from EC projects to the Government is therefore to be recognised as complex and not yet fully mastered [97]

5. not enough exchange of experiences and best practices amongst different institutions (including CSOs) involved in justice sector [108]

6. Le degré de maîtrise des connaissances spécifiques relatives aux orientations communautaires en matière de politique sectorielle (ou sous-sectorielle comme la gestion des déchets) est insuffisant. Il ne permet pas d’assumer un leadership dans le dialogue de politique, basé sur autre chose que le poids de la contribution financière [111]

7. An alternative to the Development Cooperation Instrument (DCI) should consider the rationale and implementing modalities of EC support to the EU enlargement process,



intending to share EU experience on trade, investment regulations and an opening up to the world market. This sharing should employ on the one hand targeted technical assistance programmes and, on the other, twinning programmes within a policy dialogue framework based on a jointly monitored Memorandum of Understanding [133]

8. In support to civil society, there appears to have been little or no coordination of decisions in support of interventions originating from EDF funds and those from Brussels-based thematic budget lines. Such coordination would allow a higher degree of concerted action and as well as adherence to the same MER procedures, making possible increased and much needed knowledge of the results of the budget line projects beyond the output level [155]

9. having public health personnel in the Delegation was definitively both instrumental and cost-effective. [180]

10. TACIS and the Commission suffer both from weaknesses in institutional memory and from lack of visibility in terms of demonstrating and disseminating its experiences and results. [182]

11. This challenge is – at least in part – linked to the EC’s difficulties of adequately aligning EC capacity development support to often only nascent capacity development efforts within the Government of Namibia. This affected the EC’s two focal sectors education and rural development individually and also limited the extent to which improvements in educational outcomes could be utilized to support knowledge or skill-based economic growth in Namibia’s rural areas. [203]

12. targets related to learning achievements are often not met where included in SBS and GBS indicators for flexible tranche release. Setting such indicators, which seems increasingly to be happening in ACP countries since the start of EDF 10, is a good step forward. However linked to these indicators countries should be provided with guidelines and tools to adjust and improve the situation. This is especially the case with GBS where, given partner country ownership, the EC is not in the position to steer funding to programmes directly relevant to enhancing learning. [232]



Theme 2: PROJECT CYCLE MANAGEMENT

2.1: Programming 2.2: Delays

2.3: M&E / Exit strategy / Sustainability 2.4: Learning


Reference of the extract (see excel

table)

Name of evaluations Number of evaluations

Theme 2.1: Programming

2.2: Delays 2.3: M&E

2.4: Learning

Total

1 2 Armenia 1 2.3 2.4 2 6 ASEAN 2 2.2 3 12 Burkina Faso 3 2.3 4 19 Central Africa 4 2.2 2.4 5 23 Central America 5 2.2 6 24 = 2.2 2.3 7 25 China 6 2.2 8 34 Comores 7 2.1 2.2 9 35 = 2.1 10 36 = 2.3 11 39 Development Banks 8 2.1 12 40 = 2.2 13 45 Energy 9 2.2 2.3 14 53 Synthesis 3Cs 10 2.2 15 57 Tanzania 11 2.2 16 61 = 2.1 17 62 = 2.2 18 66 = 2.2 2.3 2.4 19 70 Mauritius 12 2.1 20 71 = 2.1 21 72 = 2.1 22 77 = 2.3 23 80 Seychelles 13 2.1 24 82 = 2.2 25 86 = 2.3 26 87 = 2.3 27 90 = 2.3 28 97 Jordan 14 2.4 29 106 Mozambique 15 2.2 2.3 30 107 = 2.2 2.3 31 108 = 2.4 32 111 = 2.4 33 113 Guyana 16 2.1 34 133 Thailand 17 2.4 35 135 Uganda 18 2.1 36 152 Nigeria 19 2.2 2.3 37 155 = 2.4 38 157 Egypt 20 2.3 39 159 = 2.1 40 161 = 2.2 41 167 Liberia 21 2.1 42 168 = 2.2 43 174 Senegal 22 2.3 44 175 = 2.1 45 180 TACIS 23 2.1 2.2 2.3 2.4



46 182 = 2.4 47 191 West Africa 24 2.2 48 192 = 2.3 49 193 MEDA II 25 2.3 50 196 = 2.3 51 202 Good Governance 26 2.3 52 203 = 2.4 53 212 MPP in ACP 27 2.3 54 217 Maldives 28 2.2 2.3 55 218 Rural Development 29 2.1 56 219 = 2.3 57 224 United Nations 30 2.1 58 232 Education 31 2.4 58 different extracts for theme 2

16 extracts 2.1 21 extracts 2.2 23 extracts 2.3 12 extracts 2.4

31 evaluations for 2: 13 evaluations for 2.1 17 evaluations for 2.2 19 evaluations for 2.3 10 evaluations for 2.4

Total



3. Actor based dimension (43 extracts, from 26 different evaluations) 3.1 Participatory approach (20 extracts, from 15 different evaluations) 1. The evolution of EC strategy over time shows a clear tendency towards intensified

participation of non-governmental stakeholders from EDF8 through EDF9 and 10. Recently stakeholder participation has also become a principle embodied in the ongoing adaptation of the strategy (through stakeholder meetings at central and local levels during annual assessments of the indicators of budget support). Major stakeholders, including other donors confirm the relevance of EC interventions to target groups in Botswana [10]

2. the important role of civil society in the process of regional integration. CSOs at regional level are marginalized as a consequence of few concrete initiatives. [24]

3. policy documents reaffirm the principle of participatory development in relevant areas. Beyond general principles, the EC has not defined a clear vision on the added value of different categories of CSOs in various contexts nor systematically addressed other key operational aspects of the CSO channel. ... Despite recognition of the potential value added of CSOs within new aid modalities, official parties have not stepped up efforts to fully integrate them. Overall Chinese CSOs confirm an effective partnership with the EC. more flexible and diversified approach with longer-term planning horizons for the CSOs. Some CSOs perceive the EC as overly concerned with the relationship between CSOs and the government [26]

4. policy documents reaffirm the principle of participatory development in relevant areas. Beyond general principles, the EC has not defined a clear vision on the added value of different categories of CSOs in various contexts nor systematically addressed other key operational aspects of the CSO channel. ... Despite recognition of the potential value added of CSOs within new aid modalities, official parties have not stepped up efforts to fully integrate them. [27]

5. generally limited strategic reflection and dialogue with the various actors during programming on how best to use the CSO channel in a specific country or sector context. This has impacted negatively on the quality of programming. However, the Evaluation could identify a number of good practices reflecting a shift towards a more strategic and participatory programming ... [28]

6. EC still has to make important adjustments to its overall management systems and capacities in order to be able to strategically use the CSO channel in conformity with its new political agenda towards CSOs. ... ... The most pervasive bottleneck lies in the prevailing administrative culture, which focuses primarily on spending aid and ensuring financial accountability. [30]

7. The added value of the CSO channel is not optimally used by the EC. [31] 8. PPMR ont effectivement porté une démarche participative, même si des faiblesses ont

pu être identifiées, notamment en termes de prise en compte de la dimension du renforcement des capacités. Cette dimension a été bien prise en compte dans le programme de coopération décentralisée du 9ème FED. [32]

9. The EC still has to make important adjustments to its overall management systems and capacities in order to be able to strategically use the CSO channel in conformity with its new political agenda towards CSOs. ... ... The most pervasive bottleneck lies in the prevailing administrative culture, which focuses primarily on spending aid and ensuring financial accountability. [45]



10. Les interventions de la CE de mobilisation des acteurs non-étatiques ont eu lieu dans une phase politique extrêmement difficile dans l’histoire des Comores. Elles ont réussi – dans une situation de quasi-absence de l’État – à assurer la continuité des investissements dans le secteur social au profit des populations les plus démunies et à reconstruire des rapports constructifs de coopération entre les associations communautaires, certains services administratifs et le programme. [51]

11. All NGOs interviewed expressed the view that it is difficult to apply for EC funding. Except for some representatives of NGOs active in providing social services, NGO representatives interviewed did not report a significant EC financial contribution or, if they did, identified it as an isolated instance. [103]

12. lack of a clear vision of the role of NSAs in the health sector [109] 13. A number of EC-supported NGO implemented projects on primary education in regions

predominantly inhabited by members of ethnic minority groups have been extremely relevant, effective, and efficient [137]

14. EC support to civil society added value [140] 15. Coordination different financial instruments In support to civil society, there appears to

have been little or no coordination of decisions in support of interventions originating from EDF funds and those from Brussels-based thematic budget lines. Such coordination would allow a higher degree of concerted action and as well as adherence to the same MER procedures, making possible increased and much needed knowledge of the results of the budget line projects beyond the output level [155]

16. En ce qui concerne la durabilité des appuis de renforcement des Acteurs Non Etatiques, il n’existe pas encore de réflexion partagée ni de vision établie sur une stratégie d’autonomisation des ANE. [172]

17. programme d’appui à la planification locale, participative, et de renforcement de la maîtrise d’ouvrage des collectivités se sont avérées novatrices, même si les moyens alloués l’étaient moins. [173]

18. The complex procedures have also affected the capacity of the EC to work with civil society. For civil society to make a difference, it needs to be proactive rather than reactive. The main requirement for this is a certain amount of core funding in order to be able to mobilise quickly, and the EC will need to consider whether their procedures can be modified to accommodate this kind of support. [188]

19. The EC has embraced the principle of participatory development in all regions [206] 20. Local community participation is effective in most countries observed but with varying

quality. Lower levels of contribution have sometimes caused implementation delays, but have not significantly altered the sense of ownership. [212]

3.2 Partners capacity (25 comments, from 17 different evaluations) 1. Les capacités de formulation et de mise en oeuvre limités des partenaires, mais aussi de

la Commission ont aussi un effet négatif sur l'efficience [19] 2. technical capacity of the regional counterpart was also a key to optimising the regional

approach. Delegations are not always equipped to manage energy-related interventions or to conduct sectoral dialogue with the partner and donors [45]

3. Tanzanian innovative approach to harmonised policy dialogue is unquestionably effective. approach to harmonised policy dialogue is unquestionably effective, but it is still too early to qualify it as a success story. In this framework, the EC Delegation has made important contributions. It has played a key role in assisting the Government with



developing sector strategies and sector budget support. Effective policy dialogue has been impeded by weak institutional frameworks within the Government. [57]

4. The shift toward the use of the contribution agreements under the 9th EDF has greatly increased the efficiency of the implementation and the ownership of the beneficiary RIOs. Such shift, however, has emphasised the problem of the institutional capacity of the potential beneficiaries. [61]

5. Aid delivery efficiency has significantly improved with the introduction of contribution agreements (at COMESA level), though the capacities needed by RIOs to enhance their eligibility for CA are not yet adequately addressed [62]

6. Le renforcement des capacités et de l’efficacité de l’agence en charge de l’assainissement a été le principal point faible de l’action conjointe du gouvernement et des bailleurs de fonds. Le tableau de bord des indicateurs s’est avéré n’avoir pas anticipé ce risque et n’a pas joué son rôle de mécanisme d’incitation à des actions décisives de la part du gouvernement. [73]

7. La CE n’a pas pu contribuer à compenser des faiblesses structurelles de la gestion publique qui ont eu des effets négatifs sur les résultats atteints sur la période. [82]

8. There has been a focus on capacity building activities to support a regional organisation (the SADC Secretariat) in a situation where: there are too few posts for the mandate; existing posts are frozen or unfilled; the posts that are filled are often filled through a quota system and not through a competitive procedure; and the Executive is dominated by the SADC heads of state and has little independent manoeuvrability. [87]

9. It would have also helped to anchor capacity within domestic institutions to utilise and engage TA. [101]

10. EC support continues to contribute to management capacity support, particularly at central rather than provincial levels, but the institutional situation continues to constrain sector managers. little evidence of long term, sustainable capacity development actually having been delivered by many years of technical assistance to sector institutions, especially ANE [106]

11. The choice of implementation channels has certainly contributed to facilitating disbursement, but it is not clear to what extent Mozambican capacity constraints have been addressed or the contribution of EC support to overcoming these constraints in the longer term [111]

12. all projects were efficiently implemented due to the call of proposals procedures inherent in regional programmes, the good level of capacity of Malaysia shareholders and externalization of manpower demanding programmes to international organization [128]

13. lack of horizontal/ thematic coordination at national level has prevented the realization of environmental projects tailored to Malaysia’s specific stage of development. [129]

14. All donor support in health is hampered by the lack of an effective health care financing system and lack of capacity at all levels of the health care policy making establishment. In recognition of the lack of capacity, the EC is participating in a multi-donor capacity building project to lay the foundation for sector budget support [136]

15. Considering the central role of capacity issues in the principles of the Paris Declaration, this is one of the most significant challenges the EC needs to solve, before it can establish itself as the leader in promoting the principles of the Paris Declaration [148]

16. 9th EDF181 put the full responsibility of correctly and effectively implementing these funds into the hands of the Namibian government. Considering that based on the Commission’s own assessment the capacity of the education system at all levels was



considered to be “fragile”, this decision meant that the European Commission had to accept a high risk that these early tranches would not be used efficiently. Based on this rationale, a lot depended on quick successes of the efforts to support the development of capacity in sector organisations [149]

17. Badly implemented relocation and shortages of budgetary resources linked to poor public financial management impaired the sustainability of positive impacts, and the EC has responded to both; through intensified policy dialogue for uplands development in the first case and support to the Government’s reform agenda through PRSO budget support and associated policy dialogue in the second [164]

18. implementation mechanisms were appropriate for achieving the cooperation objectives in the prevailing institutional context, taking into account the capacities of the beneficiaries. [176]

19. CAs allowed improved definition of responsibilities and ownership by project implementers, as well as exploration of synergy effects arising from mandating of Regional Organisations. [186]

20. Contribution agreements are an additional opportunity, which will allow for the utilization of the procedures of the recipient institution rather than EC procedures, but this does imply a major challenge to institutionally weak organisations with poorly developed procedures such as SADC [188]

21. The EC has made efforts to ensure that capacity building activities (TA and twinning programmes) respond to genuine needs [201]

22. The EC so far had difficulties in finding and filling an adequate support role for helping the Government of Namibia with the development of sustained organisational capacity [202]

23. This challenge is – at least in part – linked to the EC’s difficulties of adequately aligning EC capacity development support to often only nascent capacity development efforts within the Government of Namibia. This affected the EC’s two focal sectors education and rural development individually and also limited the extent to which improvements in educational outcomes could be utilized to support knowledge or skill-based economic growth in Namibia’s rural areas. [203]

24. Governments receiving aid through a sectoral approach unsure of what this means and how it will manage the process; and, • Project payments being sent back and forth between a beneficiary country and Brussels because the method of payment and receipt were misaligned. [209]

25. Most programmes experienced relatively large slippages in their implementation time schedule, and accordingly delays in the delivery of their outputs. MPP [213]



Theme 3: ACTORS BASED DIMENSION

3.1: Participatory Approach 3.2: Partners Capacity

Number of

extracts analysed

Reference of the extract

(see excel table) Name of evaluations

Number of evaluations Theme Total

1 10 Botswana 1 3.1 2 19 Central Africa 2 3.2 3 24 Central America 3 3.1 4 26 China 4 3.1 5 27 Civil Society 5 3.1 6 28 = 3.1 7 30 = 3.1 8 31 = 3.1 9 32 Comores 6 3.1 10 45 Energy 7 3.1 + 3.2 11 51 El Salvador 8 3.1 12 57 Tanzania 9 3.2 13 61 = 3.2 14 62 = 3.2 15 73 Mauritius 10 3.2 16 82 Seychelles 11 3.2 17 87 = 3.2 18 101 India 12 3.2 19 103 Moldova 13 3.1 20 106 Mozambique 14 3.2 21 109 = 3.1 22 111 = 3.2 23 128 Malaysia 15 3.2 24 129 = 3.2 25 136 Vietnam 16 3.2 26 137 = 3.1 27 140 = 3.1 28 148 Namibia 17 3.2 29 149 = 3.2 30 155 Nigeria 18 3.1 31 164 Niger 19 3.2 32 172 Senegal 20 3.1 33 173 = 3.1 34 176 Ukraine 21 3.2 35 186 Pacific 22 3.2 36 188 SADC 23 3.1 + 3.2 37 201 Good Governance 24 3.2 38 202 = 3.2 39 203 = 3.2 40 206 = 3.1 41 209 Water & Sanitation 25 3.2 42 212 MPP in ACP 26 3.1 43 213 = 3.2 43 different extracts for theme 3.

20 extracts for 3.1 25 extracts for 3.2

26 evaluations for 3. 15 evaluations for 3.1 17 evaluations for 3.2

Total



4. Modality (55 extracts, from 34 different evaluations) 4.1 Project approach (17 extracts, from 14 different evaluations) 1. L’instabilité peut conduire à préférer le court terme (le pilotage à vue) et donc à

maintenir l’approche projet. toutefois difficile de percevoir les relations et complémentarités entre les projets [9]

2. L’approche projet qui est pratiquement la seule modalité d’aide en vigueur pour la mise en oeuvre de la stratégie régionale est caractérisée par beaucoup de problèmes qui ont un effet négatif sur l’efficience tels que les unités de gestion coûteuses [19]

3. Concernant les aides projets, il est à noter les retards dans la mise en oeuvre induisant des retards de décaissements et des reports budgétaires. Cependant, cet état de fait est surtout révélateur d’une trop grande ambition des programmes ou d’une inadéquation des procédures et modes de faire aux spécificités des secteurs. [23]

4. In particular, calls for proposals have not contributed to building a sectoral dialogue with the partner. [45]

5. Stand-alone projects nevertheless have three major limitations: (1) they are disconnected from the harmonised policy dialogue, (2) they may attract limited human resources away from strategic priorities, and (3) they depend on wider policy reforms and/or local integrated development in order to fully achieve their potential impacts. On the contrary, these modalities have been used successfully, for instance in order to test new ideas, to open pathways into new areas of cooperation, to strengthen the capacity of the Government or Non State Actors, or to pave the way towards a wider sector approach [58]

6. In the current Tanzanian context, general budget support is particularly good at improving predictability and country ownership, and promoting democratic accountability. On the contrary, projects and basket funds are better at involving civil society, and at preventing financial leakages. In the framework of general budget support, funds are channelled through the Tanzanian public service and public procurement processes, which suffer from higher than average corruption levels. Funds channelled through general budget support have followed a poverty reduction pattern, which is better aligned on Government’s priorities. [60]

7. Transaction costs are at their lowest with budget lines, but this simply means that there is no Government involvement. General budget support has reduced transaction costs for neither Government nor the development partners because it has involved a considerable co-ordination efforts (5.2.2). This evaluation is therefore unable to rank modalities along this criterion, although the costs have been claimed to be useful in the case of general budget support, as opposed to useless in the case of the design and implementation of hundreds of projects. [66]

8. early policy dialogue with Gov. of India (GoI) on more systemic issues relating to sector-wide reforms, which potentially have far more impact than any projects or programmes. [100]

9. Les unités de gestion des projets n’ont généralement pas la capacité de faire obstacle à une dénaturation du sens du projet ou de ses procédures par l’environnement politique et, dans une moindre mesure, administratif dans lequel il s’inscrit, comme dans le cas de l’utilisation des crédits pour les PME. Le dispositif de suivi et la Délégation sont restés inefficaces pour résoudre ce type de situation. Les projets acquièrent, une fois lancés, une forme d’inertie qui rend extrêmement difficile l’obtention de toute sanction ou suspension, quelle qu’en ait été sa nécessité pour l’atteinte des résultats attendus. [108]

10. the mode of implementation was the project approach, which was able to deliver results in respect of strategy implementation, albeit at a slow pace due to bottlenecks arising from procedural problems and deficient interplay between ECD, NPC and the ministries. [154]



11. The choice of financing modalities has mostly been found appropriate except for the support used to strengthen CSO and NGOs (calls for proposals) [159]

12. structures de gestion parallèle des projets qui affaiblissent les structures gouvernementales ont fait l’objet de critiques [175]

13. Cooperation in the area of border management is quite complicated by the nature of the subject new to many countries and requiring more step to step approach. In this sense, small project facilities have proved to be more successful in involving all relevant stakeholders and build necessary understanding and commitments. [181]

14. The overall efficiency of the organisational and administrative arrangements for Commission support in the Pacific has been satisfactory: the majority of projects delivered inputs by the due dates and within planned cost, and were efficiently managed on a day-to-day basis. [186]

15. project and programme approaches can, under certain conditions, be effective and efficient tools for promoting governance. [201]

16. In comparison to Capacity Building Programmes for Non-State Actors, MPPs’ combined approach creates a learning-by-doing effect and provides more autonomy to Delegation by virtue of not relying on calls for proposals [213]

17. De nombreux instruments financiers sont utilisés pour mettre en oeuvre les interventions de développement rural et agricole de la CE. Chacun de ces instruments a ses propres spécificités : les procédures pour l’aide programmable sont de plus en plus complexes alors que celles des autres instruments (Stabex, ligne budgétaire ONG, ligne budgétaire sécurité alimentaire) sont beaucoup plus souples et flexibles. Cette plus grande facilité d’utilisation a été un facteur de succès de nombreuses interventions, notamment lorsqu’il s’agissait de s’inscrire dans des dynamiques de réforme sectorielle. Si deux types de modalités d’intervention coexistent, l’approche projet/programme et l’approche sectorielle, c’est l’approche projet/programme qui reste la plus largement utilisée en matière de développement rural. L’efficience de l’approche projet/programme est faible et se caractérise par des ajustements à la baisse des objectifs initiaux du fait d'une sous-estimation des coûts unitaires, des délais d’exécution systématiquement dépassés ainsi que des coûts de mise enoeuvre et d’encadrement lourds par rapport au budget total. [219]

4.2 Budget Support (27 extracts, from 19 different evaluations) 1. l’utilisation de l’aide budgétaire. Le premier vient de l’instabilité et de l’incertitude quant à

l’application des politiques sectorielles. Un système d’aide budgétaire, basé sur des conditionnalités, ne peut fonctionner que si les partenaires (GdB et bailleurs) partagent une analyse et une volonté commune. il n’a pas été observé d’amélioration au niveau de l’impact des interventions réalisées à travers l’aide budgétaire. [9]

2. new aid delivery modalities (budget support accompanied by short-term TA) have sped up processes [11]

3. L’évolution de l’utilisation de l’ABG a été globalement positive, améliorant significativement l’efficience des interventions. [12]

4. ’efficience des interventions en général semble positive du moins en ce qui concerne les interventions utilisant l’aide budgétaire : l’alignement a un impact direct sur l’efficience des interventions. En effet, les coûts d’intervention sont mieux cadrés, mieux identifiés. [13]

5. La CE a apporté des appuis budgétaires ciblés pertinents qui se sont révélés efficaces en réduisant de façon très significative l’intensité des revendications portant sur les arriérés de salaires de la fonction publique [16]

6. flexible use of the budget support modality proved to be an efficient way of supporting a national programme, in terms of time and resources at both the EC and the



government sides. benefited from the advantages of a results-oriented modality, relying on national procedures and management, and respecting the principles of the Paris Declaration. [51]

7. This stems from the fact that financial support and policy reform have often been related in an “all-or-nothing” way. In several instances, the financial support has been interrupted, although significant poverty reduction effects could have been achieved in the absence of the awaited policy reform. [57]

8. There is consensus between the Government and the development partners towards developing budget support (either general or sector) at the expense of traditional funding modalities like basket funds3 and projects. In the case of Tanzania, this evaluation confirms that general budget support has been the best funding modality in most respects, and particularly as regards improving predictability, Government ownership, and promoting democratic accountability. General budget support has had an equal or even better potential to contribute to poverty reduction. the “variable tranche” mechanism offer a good compromise between the need to increase financial aid and the principle of progress-related support. [58]

9. the predictability of disbursements has been undermined because the EC has established an all-or-nothing relation between financial support and sector policy reforms. General Budget Support has been the only channel through which large financial support has been disbursed in time, while maintaining a connection with policy reforms through the variable tranche mechanism. [59]

10. general budget support is particularly good at improving predictability and country ownership, and promoting democratic accountability. [60]

11. General budget support has reduced transaction costs for neither Government nor the development partners because it has involved a considerable co-ordination efforts the costs have been claimed to be useful in the case of general budget support, as opposed to useless in the case of the design and implementation of hundreds of projects. General budget support is the only modality which has ensured predictability in Tanzania over the past years. This modality is flexible by nature in the sense that allocated funds are fungible. general budget support is the modality involving the highest leakage risks [66]

12. Le recours à l’instrument d’appui budgétaire a augmenté au cours de la période d’évaluation. Son utilisation est appropriée dans le contexte du Mali. [68]

13. in Tanzania, “most PRBS/PRSC donors do not believe that transaction costs have fallen, neither for the Government nor for the donors. Some even think that transaction costs may have risen in the short term. Most are nevertheless hopeful that they will fall over the medium-term as the various processes surrounding the provision of general budget support improve”. The policy dialogue involved in GBS and the monitoring of the PRS implementation implies the involvement of Government and of Development Partners in a very large number of working groups and forums. transaction costs induced by an intense policy dialogue between the Government and the donor community, and a close monitoring of policy implementation and outcomes, can be put on the same footing as the transactions costs linked to multiple negotiations carried out separately with the various donors or by the utilisation of different procedures by different donors. Frantz suggests differentiating useful transaction costs, which contribute to enhance the quality of the strategy and the effectiveness of its implementation, and useless transaction costs. [92]

14. GBS in Mozambique can be considered a successful case of donor-government collaboration. Without GBS, the Mozambican economy would not have achieved such high levels of macro-economic growth and stability and good results on strengthening the planning and budgeting systems and public finance management in general. Furthermore, GBS has permitted a significant increase in the proportion of the public



expenditure subject to the national budget and has strengthened the governmental systems. It would not have been possible to achieve the same or similar results under a project approach and without donor harmonisation and permanent dialogue with the GoM. Thus it can be concluded that EC’s macro-economic budget support strategy delivered good levels of efficiency, effectiveness and impact compared to other possible support measures. [105]

15. L’appui budgétaire à la santé, intervention ponctuelle prolongée par un appui budgétaire non ciblé, n’a pas, à lui seul, durablement contribué à l’amélioration de l’accès aux soins. La Commission européenne a su sélectionner les modes de financement les plus appropriés pour chaque domaine d’intervention et a fait preuve d’une certaine flexibilité dans le cadre de l’aide budgétaire et de l’aide non programmable. Cependant, l’efficience des projets a été assez faible, dans tous les secteurs, avec déperdition de ressources humaines et financières et de nombreux retards dans la mise en oeuvre. En conséquence, les coûts de transaction de l’aide dans le contexte de fragilité au Tchad sont très élevés. [117]

16. The application of the Variable Tranche in the EC’s support to the PRSO process, i.e. budget support is an effective means of addressing this problem. The Variable Tranche is noteworthy because it has allowed the EC to combine overall budget support with its commitment to focus on the social sector and the most vulnerable populations. [25] Through the Variable Tranche, the EC has ensured that its engagement via budget support is consistent with its commitment to support the social sectors, particularly in the poorest areas of the country. Through PRSO-based policy dialogue, it is attempting to address long-term structural issues and institutional weaknesses. [121]

17. The main strategic shift over the evaluation period, from traditional development projects to support for the government’s own reform agenda through PRSO budget support with a focus on public financial management and the dedication of variable tranche funds to the social sectors, was appropriate. [125]

18. Budget support has helped in efficiency. [142] 19. Moving to policy-led sector budget support in education and in some of the sub-sectors in

rural development has helped to establish the Commission more firmly as one of the most important development partners of the Namibian Government. [148]

20. The early use of sector budget support in education, instead of a higher share of “pooled funding” exposed the EC to risks from capacity gaps in the Namibian education system. put the full responsibility of correctly and effectively implementing these funds into the hands of the Namibian government. Considering that based on the Commission’s own assessment the capacity of the education system at all levels was considered to be “fragile”, this decision meant that the European Commission had to accept a high risk that these early tranches would not be used efficiently. [149]

21. SBS implementation has had no effect upon budgetary processes and budget management, partly because PFM issues have only very recently been integrated into Commission’s SBS programmes in Egypt but mostly because the Egyptian Government used SBS funds like project funds. [158]

22. L’aide budgétaire, toujours non ciblée, qu’elle soit globale ou sectorielle. Le fait de mettre en place une tranche variable liée à l’évolution de certains secteurs (finances publiques, santé, éducation) implique néanmoins le souhait d’utiliser cet instrument pour atteindre des objectifs dans certains secteurs spécifiques. Les fonds communs extrabudgétaires, qui précèdent généralement l’aide budgétaire lorsque certaines conditions ne sont pas remplies. La pratique est récente, puisqu’elle n’était pas autorisée avant le 10ème FED. Les procédures se rapprochent ici des procédures nationales, avec quelques aménagements en matière d’audit notamment. [164]

23. certaines difficultés en termes d’alignement des priorités entre la DG DEV et AIDCO. Ainsi, la DG DEV présente un discours politique favorable à l’aide budgétaire (objectif



politique de 60% d’aide budgétaire sous le 10 FED), tandis qu’AIDCO est plus sensible au risque financier. [165]

24. La montée en puissance de l’aide budgétaire de la CE a sans doute été trop hâtive et aurait bénéficié de passer par une amélioration préalable des mécanismes budgétaires. Ce constat est amplifié par le faible impact des mesures de renforcement institutionnel de l’AT de l’ABSRP du 9ème FED. Les scandales qui ont récemment marqué le pays ont provoqué une surcharge et une « politisation » de l’instrument qui nuisent à sa prévisibilité. [170]

25. Globally, the increased use of budget support can be considered as positive in terms of efficiency. Indeed, BS favoured stronger ownership and involvement than under traditional TA programmes and gave rise to policy dialogue on the reforms supported. [194]

26. The EC has invested heavily in budget support (in terms of conceptual understanding, development of adequate tools, participation in donor fora, training). Solid evidence on the overall impact of budget support on a wide range of possible (region-specific) governance outcomes (including country ownership, capacity development, public financial management, public service delivery, etc.) is not readily available (nor evident to attribute to the EC alone) [202]

27. As for modalities, SBS is the main modality used globally in the education sector, and is also the main modality used in each region, except ENP-TACIS where project support prevails [228]

4.3 Modality Mix (24 extracts, from 21 different evaluations) 1. The adoption of the Linking Relief Rehabilitation and Development Strategy as a

guiding strategy in all sectors proved a major contribution to the timeliness and effectiveness of EC interventions and was clearly the most important feature of the EC intervention in Angola. The fact that it was initiated by ECHO interventions and then continued through budget line and special interventions (e.g. PEAPP) is highly relevant; it generated positive lessons learned in all sectors of intervention and ensured that immediate postwar needs were satisfied [2]

2. The effectiveness of Commission interventions may also be related to Commission managers’ demonstrated capacity to make use, in a complementary manner, of the different financial instruments available to them. [7]

3. ’aide budgétaire, basé sur des conditionnalités, ne peut fonctionner que si les partenaires (GdB et bailleurs) partagent une analyse et une volonté commune. Mais ces conditionnalités deviennent vite intolérables en cas de changement de politique ou d’orientation du gouvernement. L’instabilité peut conduire à préférer le court terme (le pilotage à vue) et donc à maintenir l’approche projet comme un des instruments de la coopération avec la Bolivie, simplement parce que les conditions ne sont pas réunies aujourd’hui pour une aide budgétaire généralisée efficace. développer un continuum d’instruments complémentaires qui vont depuis le projet traditionnel jusqu’à l’aide budgétaire « pure ». toutefois difficile de percevoir les relations et complémentarités entre les projets [9]

4. l’utilisation pertinente de l’aide projet comme précurseur à la mise en place d’ABS a été menée avec succès dans le secteur eau et se prête à être élargie à d’autres secteurs[12]

5. La combinaison d’outils communautaires et d’autres instruments horizontaux d’intervention, et tout particulièrement l’intervention militaire (FOMUC) financée par la Facilité de Paix pour l’Afrique a non seulement accompagné la sortie de crise de la RCA, mais a été opportune et décisive. A un niveau plus détaillé, la mise en oeuvre des différents outils amène à penser que des problèmes de combinaison et de synergie restent à résoudre. Une stratégie communautaire globale reste à construire. [18]



6. Government and the development partners towards developing budget support (either general or sector) at the expense of traditional funding modalities like basket funds3 and projects. real-life choices are not that much a matter of selecting one best modality, but rather of setting up an adequate mix and/or sequence of funding modalities. [58]

7. the current Tanzanian context, general budget support is particularly good at improving predictability and country ownership, and promoting democratic accountability. On the contrary, projects and basket funds are better at involving civil society, and at preventing financial leakages. [60]

8. approche projet ont rencontré plus de difficultés de mise en oeuvre que ceux correspondant à l’approche appui budgétaire avec laquelle elle est toutefois largement complémentaire. L’assistance technique et les actions de proximité sont particulièrement appréciées des beneficiaries [68]

9. L’appui budgétaire sectoriel en faveur de l’assainissement a fortement contribué à l’adoption d’une approche sectorielle. La prévalence des intérêts à court terme dans les pratiques politico-administratives constitue un risque réel dans la perspective de l’aide budgétaire globale [73]

10. The support to SMEs is a success story of an efficient joint policy development EC support to SMEs is a showcase of joint policy development. It has led to the establishment of a private-sector development policy which is now fully managed and owned by the Government, and partly funded by the EC through sector budget support. It took ten years from the initial pilot project to the full transfer of the policy to the Government. timely articulation of the three instruments (pilot project, large-scale programme, and sector budget support), much more than to their intrinsic merits. [96]

11. The decision to phase out most of the stand-alone projects, concentrated within rural development, and to focus instead on a few selected sectors seems well justified and was very visionary and innovative at that time. The shift to sector support also facilitated a qualitative improvement in the ongoing dialogue on development co-operation between the EC and India. [98]

12. à l’exception de quelques projets de cofinancement d’ONG, les interventions prévues dans la stratégie de la Commission européenne sont cohérentes avec les autres instruments de la coopération (DG Echo, BEI, lignes budgétaires) et vice versa. [116]

13. shift over the evaluation period, from traditional development projects to support for the government’s own reform agenda through PRSO budget support with a focus on public financial management and the dedication of variable tranche funds to the social sectors, was appropriate. some areas, namely UXO and civil society development, where the mix of approaches used has failed to meet some needs. [125]

14. EC financing of UNDP governance projects and the emerging use of thematic budget lines such as EIDHR and non-state actors and local authorities to address these issues are other examples of adaptation. [126]

15. The financial volumes mobilised under the CSPs were too limited to allow for a strategic mix of various instruments, approaches or financing modalities. [127]

16. With hindsight, the start of an SPSP, with budget support as the chosen modality, was premature and did not address needs, and the approach has not worked as well as was hoped. [138]

17. the interventions used a good mix of implementation modalities but their objectives were overambitious and their implementation requirements were underestimated. The 1998-2008 period saw cooperation evolve towards projects of lesser financial weight and Sector Budget Support Programmes which often combined different implementation modalities to good effect. [161]

18. introduction de nouveaux instruments et à leur combinaison (aide programme et aide budgétaire) sur un nombre limité de domaines de concentration, une amélioration nette du taux d’engagement [175]



19. CSO-directed activities are designed to complement the work of the numerous other donors working in this area, as well as other EC instruments such as EIDHR. [179]

20. The blending of Commission and EIB instruments, especially risk capital facility operations and interest subsidies, and the use of the twinning modality, reinforced the Community dimension. [193]

21. The Commission used a comprehensive set of instruments to finance various interventions when the conditions associated with budget support and the objective of alignment with the “acquis communautaire” were accepted by the partner. Over the years there has been a growing attempt to ensure that different instruments, such as BS, TA and more recently twinning, are mutually supporting and contributing to the same goals. [194]

22. efforts to strategically combine different instruments to promote governance (e.g. geographic programme and EIDHR) [207]

23. In a context of post-natural disaster, political instability, and institutional change in the Maldives, the EC has made flexible and cost-effective use of aid modalities, delivery channels and existing implementing capacities, thereby contributing to principles of the Paris Declaration and Accra Agenda, and strengthening local project management skills. [217]

24. De nombreux instruments financiers sont utilisés pour mettre en oeuvre les interventions de développement rural et agricole de la CE. Chacun de ces instruments a ses propres spécificités : les procédures pour l’aide programmable sont de plus en plus complexes alors que celles des autres instruments (Stabex, ligne budgétaire ONG, ligne budgétaire sécurité alimentaire) sont beaucoup plus souples et flexibles. Cette plus grande facilité d’utilisation a été un facteur de succès de nombreuses interventions, notamment lorsqu’il s’agissait de s’inscrire dans des dynamiques de réforme sectorielle. Si deux types de modalités d’intervention coexistent, l’approche projet/programme et l’approche sectorielle, c’est l’approche projet/programme qui reste la plus largement utilisée en matière de développement rural. [219]



Theme 4: MODALITIES

4.1: Project Approach 4.2: Budget Support 4.3: Modality Mix


Reference of the extract (see excel

table)

Name of evaluations Number of evaluations

Theme 4.1: Project Approach 4.2: Budget Support 4.3: Modality Mix

Total

1 2 Armenia 1 4.3 2 7 ASEAN 2 4.3 3 9 Bolivia 3 4.1 4.2 4.3 4 11 Botswana 4 4.2 5 12 Burkina Faso 5 4.2 4.3 6 13 = 4.2 7 16 Central African Rep. 6 4.2 8 18 = 4.3 9 19 Central Africa 7 4.1 10 23 = 4.1 11 45 Energy 8 4.1 12 51 El Salvador 9 4.2 13 57 Tanzania 10 4.2 14 58 = 4.1 4.2 4.3 15 59 = 4.2 16 60 = 4.1 4.2 4.3 17 66 = 4.1 4.2 18 68 Mali 11 4.2 4.3 19 73 Mauritius 12 4.3 20 92 Rwanda 13 4.2 21 96 Jordan 14 4.3 22 98 India 15 4.3 23 100 = 4.1 24 105 Mozambique 16 4.2 25 108 = 4.1 26 116 Chad 17 4.3 27 117 = 4.2 28 121 Laos 18 4.2 29 125 = 4.2 4.3 30 126 = 4.3 31 127 Malaysia 19 4.3 32 138 Vietnam 20 4.3 33 142 Nicaragua 21 4.2 34 148 Namibia 22 4.2 35 149 = 4.2 36 154 Nigeria 23 4.1 37 158 Egypt 24 4.2 38 159 = 4.1 39 161 = 4.3 40 164 Niger 25 4.2 41 165 = 4.2 42 170 Senegal 26 4.2 43 175 = 4.1 4.3 44 179 TACIS 27 4.3 45 181 = 4.1 46 186 Pacific 28 4.1 47 193 MEDA II 29 4.3 48 194 = 4.2 4.3 49 201 Good Governance 30 4.1 50 202 = 4.2 51 207 = 4.3



52 213 MPP in ACP 31 4.1 53 217 Maldives 32 4.3 54 219 Rural Development 33 4.1 4.3 55 228 Education 34 42 55 different extracts for theme 4

17 extracts 4.1 27 extracts 4.2 24 extracts 4.3

34 evaluations for 4. 14 evaluations for 4.1 19 evaluations for 4.2 21 evaluations for 4.3

Total



CONCLUSION58 Poverty reduction focus (4 extracts, from 4 different evaluations) 1. contribution of EC strategy to poverty reduction has been considered (concept of

“poverty incidence”), but in a superficial way. [62] 2. Les interventions de la CE répondent aux besoins de la croissance économique du

pays et aux exigences de la lutte contre la pauvreté en tenant compte des spécificités de Maurice et de Rodrigues [69]

3. the EC is acting on its comparative advantage of being able continue to address poverty issues in Namibia, despite the fact that Namibia is officially classified as a middle- income country. While other donors, including those of EU Member States, are discontinuing their assistance, the EC is able to maintain its support to Namibia in crucial areas such as education or rural development that can help to reduce poverty in Namibia. [201]

4. Les documents de stratégie pays de la CE sont en cohérence avec les stratégies nationales de réduction de la pauvreté (DSRP) des pays partenaires, [218]

Devolution (4 extracts, from 4 different evaluations) 1. recent de-concentration process and the associated growth in the Delegation’s staff.

[57] 2. Deconcentration has improved Delegation performance but the high number of projects

overloads the Delegation. Fiduciary risk concerns lead to a control-focus. [142] 3. the organisational and management capacities of the Commission have improved,

notably as a result of devolution [186] 4. La déconcentration a favorisé la production des résultats escomptés. [219]

58 Based on a general assessment and extracts from 7 different evaluations.

study on legal instruments and lessons learned from...

Documents