pcr evaluation note for public sector operations · team-new and pilot facilities constructed,...

1. BASIC INFORMATION a. Basic project data Project title: Support for the Harnessing of Surface Water for Domestic and Agricultural Use in Rural Areas Project code: P-DJ-EAZ-001

Instrument number(s): 56001 55001 001

Project type: Studiesand Works

Sector: Water

Country: Djibouti Environmental categorization (1-3) : 1Processing Milestones Key Events Disbursement and Closing

date Date approved: 29 January 2008

Cancelled amount: - Original disbursement deadline:

Date signed: 4 February 2008

Supplementary financing: - Original closing date: 31December 2010

Date of entry into force :

Restructuring: - Revised disbursement deadline:

Date effective for 1st disbursement: 4 February2008

Extensions (specify dates): from 31/12/11to 31/12/12 and to 31/12/14

Revised closing date: 31December 2014

Date of actual 1st: September 2008b. Financing sources

Financing source/instrument (MUA)

Approved amount(MUA) :

Disbursed amount(MUA) :

Percentage disbursed(%):

AWF EUR 1,937,000 EUR 1,937,000 100%Government EUR 138,600 EUR 138,600 100%Beneficiaries EUR 70,560 EUR 70,560 100%

TOTAL : EUR 2,146,160 EUR 2,146,160 100% Co-financiers and other external partners: Saudi Fund and IFAD (PMU Staff)

Execution and implementation agencies: Ministry of Agriculture, Water, Fisheries, Livestock and Fishery Resources (MAEPE-RH), through the Project Management Unit (PMU).c. Responsible Bank staff

Position At approval At completion Regional Director F.J.M.PERRAULT NEGATU Gabriel Sector Director K. BEDOUMRA Mohamed El AZIZI Sector Manager Akissa Bahri Jean Michel OSSETE Task Manager T. DAMHAUG; Yvan KEDAJ;

OUTAGEROUINE Abderahmane; El Faleh JALEL; Francis Daniel BOUGAIRE

Francis Daniel BOUGAIRE

Alternate Task Manager P. AKARI

PCR EVALUATION NOTE FOR PUBLIC SECTOR OPERATIONS

PCR Team Leader Francis Daniel BOUGAIRE PCR Team Members Jérôme L. THIOMBIANOd. Report data PCR Date : 9/7/2015 PCR Mission Date: From: 16/6/2015 To: 26/6/2015 PCR-EN Date: 25/5/2018 Evaluator/consultant : Daniel VAN ROOIJEN Peer Reviewer/Task Manager:

2. PROJECT DESCRIPTIONSummary from Appraisal Report including addendum/corrigendum or loan agreement, and taking into account any modification that occurred during the implementation phase.

The project is focused on improving water availability for domestic, agricultural and livestock use in Digri in the region Ali Sabieh and Sadai in the Obock area of Djibouti. Much of fresh water potentially available is lost since most of it drains into the ocean when water from high intensity rainfall events turns into flash floods without any structures in place to capture and store this water. It also focuses on capacity strengthening of the Ministry of Agriculture, Livestock and the Sea, in charge of Hydraulic Resources (MAEM-RH) in charge of carrying out the project. The project also sought to improve knowledge and data generation through the carrying out of hydro-geological studies and establishing of a data-base to guide andinform future investments in the water sector.

a. Rationale and expected impacts:

The project stems from the rationale that increasing water availability will directly improve the quality of live and reduce poverty levels among the rural population in the Digri and Obock regions in Djibouti. The associated impact expected from the project was improved socio-economic conditions of the target population including a reduction of the poverty rate. The listed beneficiaries are 2,400 rural people living in the Digri and Obock regions of Djibouti, 17 technical staff working at the Ministry (MAEM-RH), contractors hired to carry out the civil works as well as large institutions involved in the project (IFAD, FAO). No major changes took place except that the project duration was 67 months longer than initially designed (20 months) due to various technical, administrative and financial difficulties faced by the project that led to successive extensions of the project implementation period, bringing the actual total implementation period to 87 months (four times longer).

b. Objectives/Expected Outcomes:

The main project objective was to improve the socio-economic condition of the target population through improved water availability. Expected outcomes are:

In the short-term:- The identification of projects for the harnessing of water through average-capacity facilities

finalized, benefiting the project team- Reports on feasibility studies, preliminary design studies (PDS), final design studies (FDS) and

bidding documents for water harnessing facilities prepared and validated, benefiting the project

team- New and pilot facilities constructed, benefiting the water users at the new and pilot facilities- The capacity of MAEPE-RH services built, benefiting MAEPE-RH staff and the population of

Djibouti- A geophysical survey on 42 potential borehole sites conducted and a resistivity map prepared,

benefiting the project team- The files prepared that are required to prepare an inventory of water resources and launch the

updating of the monitoring network and facility restoration, benefiting the Ministry

In the medium-term the following outcomes were expected:

- Greater water availability in Dikhil in the Ali Sabieh Region and Sadai in the Obock Region, benefiting the people living in those regions

- Greater knowledge on water resources through staff training, benefiting the technical staff at the ministry

- Reduced prevalence of water-borne diseases in the two project areas, benefiting the people living in those areas

- Optimization of outcomes during the construction of boreholes, benefiting MAEPE-RH staff at the Ministry

- Provision of a project bank to the authorities, benefiting the government authorities

The intended beneficiaries are 2,400 rural people living in the Digri and Obock regions of Djibouti, 17 technical staff working at the Ministry (MAEM-RH), contractors hired to carry out the civil works as well as large institutions involved in the project (IFAD, FAO).

c. Outputs and intended beneficiaries:

The project had the following expected outputs:

Component A, Output 1: Working designs for water harnessing facilities prepared, benefiting project staffComponent A, Output 2: Pilot and new facilities constructed, benefiting the target populationComponent B, Output 1: Geophysical prospecting conducted, benefiting project staffComponent C, Output 1: The capacity of MAEM-RH services built, benefiting MAEM-RH and target populationComponent C, Output 2: Experience shared with similar countries, benefiting institutions in these countriesComponent D, Output 1: The inventory and the study on the updating of monitoring networks conducted, benefiting the MinistryComponent D, Output 2: A project bank for surface water harnessing using average-capacity facilities available, benefiting the Ministry and MAEM-RH unitComponent E, Output 1: Project vehicles procured, benefiting the project teamComponent E, Output 2: Project staff recruited, benefiting the project Component E, Output 3: Project financial audit conducted, benefiting the project

d. Principal activities/Components:

The project was composed of 5 major components and activities:

Component A: Construction of Water Facilities- Site selection for implementation of civil works- Implementation studies of works

- Construction of 14 underground tanks with a volume of 200 m3- Construction of 1 impluvium- Construction of 1 structure diversion and spreading flood waters the wadi Sadai,- Construction of 1 catchment reservoir structure

Component B: Geophysical Prospecting- Carrying out of a geophysical survey

Component C: Capacity Building- Technician training of the MAEM-RH and exchange experiences with Similar countries- Training of a borehole driller who will train others

Component D : Water Sector Study- Inventory and balance sheet of current water resources- Study for the update from the rainfall and hydrometric database;- Identification of sites and preparing a bank mobilization project waters by the realization of works of average capacity

Component E: Project Management- recruitment and approval of project staff- acquisition of 2 vehicles- acquisition of office furniture and others supplies- establishment of a participatory approach- the recruitment of a specialized cabinet for financial audit of the project during the 2 exercises concerned (2008 and 2009)

3. PROJECT PERFORMANCE ASSESSMENT

RELEVANCE

a. Relevance of the project development objective:

Djibouti is highly water-stressed country, due to its hot climate including very low and erratic rainfall and much of its water resources not suitable for consumption. Water availability is therefore very critical for securing domestic use, as well as sustaining agricultural and pastoralist livelihoods. The country has been affected by drought events which are likely to repeat in the future. The project, focusing on increasing water availability, is considered highly relevant with regards to sustaining or improving the living conditions of the rural people in Djibouti. The project was designed to respond to the need of the local beneficiary population with an overall objective of improving the living conditions of the rural population by harnessing water for domestic and agricultural use. The project is in line with the Poverty Reduction Strategy Paper (PRSP) adopted in Djibouti in July 2004 and the priority thrusts of AWF’s Operational Programme 2005-2009. It will, therefore, contribute to achieving the Millennium Development Goals (2015) and the African Water Vision for 2025 by supporting the establishment of appropriate and viable drinking and irrigation water services accessible to all the rural populations of the Dikhil and Obock Regions. The review and the evaluator wish to rate this criterion with highly satisfactory(4), which is line with the PCR rating.

b. Relevance of project design (from approval to completion):

The project has a straightforward and logical design, comprising of assessments and studies, civil works, capacity building and project management. The design of these components seem to serve well the envisaged project outcomes. The assessment of risks done in the appraisal report is adequate and realistic. Implementation arrangements were in principle well designed, including PMU being physically based at the Ministry and the regional coordinators presumably permanently located in the regions of intervention.

Information is missing regarding the criteria used for the selection of the target population of approximately 2,400 inhabitants living in of areas of Digri and Obock. Indicators such as remoteness/accessibility have an influence on the costs of operations, so it would have been interesting to understand how these indicators were weighted against socio-economic indicators including poverty levels. The appraisal report and PCR note that environmental and social impact studies were carried out under component A (construction of the hydraulic structures), however results of these studies were missing in theavailable information, nor any corrective measures taken to mitigate negative social or environmental impacts.

A major shortfall is the short time frame in which the project had to be completed, which lead to substantial extension of the project period. The project duration was 67 months longer than initially designed (20 months) due to various technical, administrative and financial difficulties faced by the project that led to successive extensions of the project implementation period (87 months). This could have been avoided through better estimation of time required to carry out the civil works and including some mechanism to ensure that all PMU staff is fully on board from the start. As a consequence there was a high project overhead cost. The review and the PCR observe the absence of the designs and estimated costs of major water facilities (flood spreading dam and catchment pond) during project appraisal. The updating ofthe working designs has led to pre-empt project implementation coming to a standstill due to under-estimation of the costs of constructing the said facilities. The search for supplementary financing, has resulted in a too lengthy extension of the project implementation period. In addition, the PCR note that the pilot nature of this project requires that provision be made, during project design, for the financing of a monitoring and evaluation system in order to learn all relevant lessons. The project appraisal report did notprovide for the financing of a monitoring and evaluation system. Instead, the project appraisal report indicates that monitoring and evaluation system activities will financed by IFAD funding. Therefore the review rates this criterion with unsatisfactory (2), which differs from the PCR rating.

EFFECTIVENESS

c. Effectiveness in delivering outputs :

Based on the information available, it can be concluded that the project delivered well on its intended outputs. All civil works, studies and trainings as part of consulting contracts were carried out within the contractual timeframe. The delays in recruitments of project staff for the PMU delayed the starting up of project activities in the first year, and associated outputs expected under the project management component. Based on the information available to the evaluator, there is some lack of evidence on reported outputs, for example through photos, in order to verify the output reporting in the PCR. The review and the evaluator wish to rate this criterion with satisfactory (3), which is line with the PCR rating.

d. Effectiveness in delivering outcomes:

The project has shown increased technical capacities of staff working at the ministry and staff responsible for O&M of the water intake and storage structures installed. To the knowledge of the evaluator the projectdoes not include public awareness campaigns or sensitisation with regards to water use from these structures. The documentation suggests that institutions are very well engaged in the project. Progress of some of the expected outcomes listed in the appraisal report were not reported in the IPR, for example the

expansion of agro-pastoral area around the new water facilities, the number of days without water and drilling success rate. The review and the evaluator wish to rate this criterion with satisfactory (3), which is line with the PCR rating.

e. Project development outcome:

It is very likely that the project has contributed to the development objective of improving the socio-economic conditions and reducing poverty levels of the target population. However, contributions of this project to development outcomes were not measured in a household survey carried out among the target population. An important project objective, increased water availability, was not well measured or estimated in terms of actual increased availability for the different water users and uses. This criterion is rated with satisfactory (3), which is the same as the PCR rating of satisfactory (3).

f. Beneficiaries:

There seems to be a discrepancy on type of reporting of beneficiaries between appraisal report and PCR. The PCR reports that the capacity of the beneficiary populations of the project area was built by involving them in the construction of water facilities, however without including total numbers, but with a reported gender balance of 40% women. There is insufficient detail on the way communities were involved, in order to evaluate the effectiveness of this involvement in terms of capacity building. The appraisal report includesapproximately 2,400 rural people that are expected to benefit from increased water availability however thenumber of people is not reported in the PCR. Some of the items listed as beneficiaries in the PCR are equipment and should not be included here. IFAD and FAO are listed as beneficiaries, however the evaluator is not sure if this is appropriate. Apart from their mere involvement in the project, there is no clear information confirming that they participated in trainings. Contractors are also reported as beneficiaries which is generally unusual in reporting of development projects. The review rated this criterion related to beneficiaries’ coverage with satisfactory (3).

g. Unanticipated additional outcomes (positive or negative, not taken into consideration in the project logical framework) :

Despite being included in the project justification/rationale, the project lacks discussion on how increased water storage (and availability) may reduce the possible impacts of future droughts. Any considerations on gender outcomes are not reported in the PCR. The PCR mentions the positive impact of water for wildlife and, as a result, possible development of wildlife hunting. Also the possible risk of malaria is mentioned andconsidered negligible. In accordance with the above issues, the review confirms this criterion with satisfactory (3), which is line with the PCR analysis.

EFFICIENCY

h. Timeliness:

The project has faced a number of administrative, technical and financial difficult which caused the projectto be seriously delayed. These difficulties have been duly acknowledged in the PCR. The total implementation period was designed for 20 months, but effectively took 87 months after a series of non-costextensions. Some of the causes of delays are considered beyond the control of the project while other factorscould have been better anticipated during the project design stage. The ratio of planned implementation time (PAR) and actual project implementation time from the date of effectiveness is 0.23 which is very low. The review and the evaluator concur with the PCR score and rates this criterion with highly unsatisfactory (1).

i. Resource use efficiency:

The project has used 100% of the allocated resources in all project components except under the civil workscomponent. In the civil works component, the costs needed to build the catchment pond was seriously underestimated and as a result, additional funding was sought and provided in order to finalise the structure.Given the substantial difference between expected cost and actual cost of construction of the catchment pond, the evaluator was missing further technical details on these cost estimations in the available information. The review and the evaluator rate this criterion with satisfactory (3), in line with the PCR rating.

j. Cost-benefit analysis:

Unable to evaluate due to no information provided in the PCR.

k. Implementation progress:

Despite the key successes a number of issues are worth mentioning, which were discussed in the internal review reports and final audit carried out during project implementation.

- A lack of evidence that environmental and social safeguards were adhered to- Concerns regarding the modalities of procurement and extend of monitoring carried out by the

PMU.- Concerns regarding the compliance with the terms and conditions for the award of contracts,

specifically with regards to following the planned acquisition methods, community works and recruitment of some consultants.

- Concerns regarding the efficiency and performance of the procurement process- Concerns regarding the operating expenditures carried out by the PMU and the overall

procurement capacity of the PMU- Lack of sufficient detail on the apparent amendment to contract- Concerns regarding the quality and completeness of the archiving system for acquisition documents- Lack of a robust monitoring and evaluation system for tracking progress towards project outputs

and outcomes

Based on the above issues, the evaluator rates this criterion with unsatisfactory (2), which is lower than the PCR rating (3).

SUSTAINABILITY

l. Financial sustainability:

There is generally a lack of data regarding the financial sustainability post-project, including cost-benefit analysis and a long term cost-recovery plan. This was not included in the project. There is a lack of provision of details on how the constructed water intake and diversion structures will be maintained and from which budget would be used. This criterion is therefore rated as unsatisfactory (2) which is in line with the PCR rating.

m. Institutional sustainability and strengthening of capacities:

It is apparent from the available information that the Ministry has developed its capacity to manage this project and it will be better prepared to run similar projects in the future based on the lessons learned fromthis project. However, this capacity strengthening was not sufficiently measured and included in an overall

monitoring and evaluation framework, which is considered a missed opportunity. A challenge remains to upscale the physical interventions carried out in this project to other suitable areas in Djibouti, which is rightly acknowledged in the PCR. The review rates this criterion with unsatisfactory (2) which is lower than the evaluating of satisfactory (3) and the PCR rating of highly satisfactory (4).

n. Ownership and sustainability of partnerships:

The project reports demonstrate convincing information that the Ministry has taken ownership of the project including the considerable achievements made as a result of the project. However, there as indicatedearlier, there is no information on the way operation and maintenance of the water structures is secured beyond the project lifeline. The PCR report that the target communities have taken ownership of the water intake structures through their involvement in their construction. However, there is no sufficient details on how this involvement was done and the perspectives of the communities were not recorded. On this basis it is not possible to determine the level of ownership as perceived by the communities. There is no informationon involvement, perspectives of civil society organisations. The review and the evaluator rated this criterionwith satisfactory (3) which is lower than the PCR rating of highly satisfactory (4).

o. Environmental and social sustainability:

The appraisal report and PCR note that environmental and social impact studies were carried out under component A (construction of the hydraulic structures), however results of these studies were missing in theavailable information, nor any corrective measures seem to have been taken to mitigate negative social or environmental impacts. The review confirms the evaluator rating with unsatisfactory (2) which is lower than the PCR rating of satisfactory (3).

4. PERFORMANCE OF STAKEHOLDERS

a. Bank performance:

The Bank has performed well in supervising this project from start to end. A sufficient number of supervisory and review missions were carried out. Protocols and procedures in place at the Bank have beenadhered to. The Bank could have tried to reduce the number of turnovers of the project leader to create more stability in project management, including ways to ensure maximum commitment of the PMU unit. The Bank could have given closer attention to the use of a monitoring and evaluation framework and could perhaps have provided a stronger supervisory role with regards to ensuring social, environmental and financial project sustainability. Given the project’s success despite the challenges during implementation, Bank’s performance is rated as satisfactory (3), in line with the PCR rate. The PCR narrative for this criterion is somewhat general and discusses the Banks performance in very little detail, however it is considered objective. Based on the above issues, the review rates this criterion with satisfactory (3), which isline with the evaluator and the PCR rating (3).

b. Borrower performance:

The borrower has generally performed well during project implementation. Perhaps more could have been done to mitigate extensive delays (see earlier section on timeliness). Gaps exist with the borrower with regards to the operation and maintenance of the water structures constructed during the project, which willendanger the sustainable functioning of these works. Borrowers performance is rated as satisfactory (3), inline with the PCR rate.

c. Performance of other stakeholders:

The contractors and other consulting firms have performed well, producing work up to required standard and within the contractual timeframe. One of the co-financiers, the Saudi Fund provided sound collaboration which is acknowledged in the PCR. Despite a lack of details on community involvement, no issues have been reported with regards to the functioning of the communities and their involvement in constructing the water structures. Performance of other stakeholders is rated as satisfactory (3), in line with the evaluator and the PCR rating.

5. SUMMARY OF OVERALL PROJECT PERFORMANCE

a. Overall assessment:

The project is considered highly relevant, responding to the critical issue of water insecurity in rural Djibouti. The project has performed well despite challenges causing significant delays in project implementation. Au terme du Projet, le pays disposerait actuellement (i) des infrastructures nouvelles et pilotes de mobilisation des eaux, (ii) une connaissance des conditions hydrogéologiques de la zone du projet,(iii) un renforcement des capacités des services techniques du MAEM-RH et (iv) des études thématiques du secteur de l'eau en milieu rural à Djibouti ainsi que les investissements nécessaires y afférents.The associated impact expected from the project was improved socio-economic conditions of the target population including a reduction of the poverty rate. This impact, or even outcomes leading to this impact were not sufficiently measured in order to make credible statements on project attribution to better socio-economic conditions or reduced poverty. Some concerns have been noted earlier with regards to especially the social and financial sustainability of the project.

No major changes took place except that the project duration was 67 months longer than initially designed (20 months) due to various technical, administrative and financial difficulties faced by the project that led to successive extensions of the project implementation period, bringing the actual total implementation period to 87 months (four times longer). Administrative difficulties concerned: (i) the non-availability of PMU staff during the first year of the project; (ii) the late start of the process for the working designs of thefacilities and the procurement of project goods and services (the process effectively started 32 months after the Grant Agreement signature in February 2008); (iii) failure to lay down beforehand the terms and conditions for constructing the facilities under State control; and (iv) the frequent replacement of the project officer(four) at the Bank.

Based on the above issues, the peer reviewer suggests an overall rating of unsatisfactory (2). However, the average of the dimensions is 2.56 which, according to the guidelines, places the project just inside the satisfactory category. The PCREN agrees therefore with the overall PCR rating (3).

b. Design, implementation and utilization of the M&E (appreciation of the evaluator):

The appraisal report mentions the use of a monitoring and evaluation plan to be put in place by the Management Unit of the Project based on the project matrix. In addition, the monitoring and evaluation plan would be integrated into the monitoring and evaluation framework of the water mobilization program, co-financed by IFAD. The available information does not sufficiently demonstrate the generation of monitoring and evaluation data. The PCR mentioned the need to finance a monitoring and evaluation system under lessons learned related to relevance. The above leads to the conclusion that a monitoring and evaluation framework has not been effectively utilized. Based on the above issues, the review and the evaluator rate this criterion with highly unsatisfactory (1), which is in line with the PCR rating (1).

6. EVALUATION OF KEY LESSONS LEARNED AND RECOMMENDATIONS

a. Lessons learned:

The evaluator fully concurs with the four major lessons learned listed in the PCR. They relate to the functioning of the PMU (issue 1), assessment of the implementation schedule / planning (issue 2), assessment of financial resources vs costs of major works (issue 3) and financial viability (4).

The Evaluation’s opinion of each lesson is given in summary below:

1. VALIDATED2. VALIDATED3. VALIDATED4. VALIDATED

b. Recommendations:

The review and the evaluator concur with the two recommendations listed in the PCR. These relate to financing maintenance of the water structures (1) and consolidation of the institutional framework (2). Regarding the latter, the PCR does not provide much detail on how institutional capacities can be maintained or further increased.

The Evaluation’s opinion of each recommendation is given in summary below:

1. VALIDATED2. VALIDATED

The evaluator suggests adding the following recommendation:

In future projects the Bank should consider paying more attention to ensuring sustainability of investmentsmade in water structures beyond project completion.

7. COMMENTS ON PCR QUALITY AND TIMELINESSThe overall PCR rating is based on all or part of the criteria presented in the annexe and other: The quality of the PCR is rated as highly satisfactory (4), satisfactory (3), unsatisfactory (2), and highly unsatisfactory (1). Le timeliness of the PCR is rated as on time(4) or late (1). The participation of the Borrower, co-financier, and the bank’s external office(s) are rated as follows: Very Good (4),

Good (3), Fair (2), Poor (1).

The quality of the PCR is considered as unsatisfactory (2.45). Generally the narrative and numeric scoring are not consistent with each other and insufficient explanation was provided. The PCR lack evidence and data analysis to substantiate the ratings of the various sections.

The extent of quality and completeness of the PCR evidence and analysis to substantiate the ratings of the various sections was considered unsatisfactory. The absence of an efficient system of generation of monitoring and evaluation data. The project appraisal report did not provide for the financing of a monitoring and evaluation system. The PCR lack evidence and data analysis to substantiate the ratings of the various sections. The extent of objectivity of PCR assessment score was considered satisfactory. The extent of internal consistency of PCR assessment ratings; inaccuracies; inconsistencies was considered as satisfactory. The extent of identification and assessment of key factors (internal and exogenous) and unintended effects (positive or negative) affecting design and was considered satisfactory. The adequacy of treatment of safeguards, fiduciary issues, and alignment and harmonization was considered satisfactory. The extent of soundness of data generating and analysis process (including rates of returns) in support of PCR assessment was considered highly unsatisfactory. There has been insufficient data generated to support the progress made in the project. The overall adequacy of the accessible evidence (from PCR including annexure and other data provided was considered highly unsatisfactory. The project is not likely to achieve its development objectives without a change in tools and systems for data collection and management. The extent to which lessons learned (and recommendations) are clear and based on the PCR assessment (evidence & analysis) was considered unsatisfactory. There has been insufficient data generated to support the progress made in the project. The extent of the overall clarity and completeness of the PCR was considered satisfactory.

The timeliness of the PCR is rated as late (1), since it was produced 7 months after project closure which is longer than the required 6 months.

8. SUMMARY OF THE EVALUATIONThis is a summary of both the PCR and IDEV ratings with justification for deviations/comments. Appropriate section of the PCR Evaluation should be indicated in the last column in order to avoid detailed comments. The evaluator must provide a reasonable explanation for each criterion the PCR rating is not validated by IDEV. Consequently, the overall rating of the project could be “equally satisfactory”.

Criteria PCR PCREN Reason for disagreement/ Comments

RELEVANCE 3.5 3.0 SatisfactoryRelevance of project development objective 4 4 The project is in line with the Poverty

Reduction Strategy Paper (PRSP) adopted in Djibouti in July 2004 and thepriority thrusts of AWF’s Operational Programme 2005-2009. It will, therefore, contribute to achieving the MDG (2015) and the African Water Vision for 2025 by supporting the establishment of appropriate and viable drinking and irrigation water services

accessible to all the rural populations of the Dikhil and Obock Regions and to respond to the need of the local population with an overall objective of improving their living conditions by harnessing water for domestic and agricultural use.

Relevance of project design 3 2 The absence of the designs and estimated costs of major water facilities (flood spreading dam and catchment pond) during project appraisal and the updating of the working designs have resulted in a too lengthy extension of theproject implementation period (87 months against 20 months initially designed). This could have been avoidedthrough better estimation of time required to carry out the civil works and to ensure that all PMU staff is fully on board from the start and avoid a high project overhead cost. In addition, no financial provision was made for the financing of a monitoring and evaluation system activities during project appraisal (by IFAD funding).

EFFECTIVENESS 3 3Development objective (DO) 3 3 It is very likely that the project has

contributed to the development objective of improving the socio-economic conditions and reducing poverty levels of the target population. However, contributions of this project todevelopment outcomes were not measured for example in a household survey carried out among the target population.

EFFICIENCY 2.3 2.0 UnsatisfactoryTimeliness 1 1 Highly unsatisfactory. Ratio between

planned project duration and project actual implementation time is 0,229, and less than 0,50

Resource use efficiency 3 3 Satisfactory. Ratio between median physical implementation of RFF outputsby all financiers and commitment rate is0, 86, and more than 0, 75 and less than 1.

Cost-benefit analysis Noscore

UTR Unable to evaluate due to no information provided in the PCR.

Implementation progress (IP) 3 2 Unsatisfactory. A number of issues have substantially affected satisfactory

implementation progress.SUSTAINABILITY 3 2 Unsatisfactory Financial sustainability 2 2 Unsatisfactory There is generally a lack

of data regarding the financial sustainability post-project, including cost-benefit analysis and a long term cost-recovery plan. This was not included in the project. There is a lack of provision of details on how the constructed water intake and diversion structures will be maintained and which budget would be used.

Institutional sustainability and strengthening of capacities

4 2 Unsatisfactory. It is apparent from the available information that the Ministry has developed its capacity to manage this project and it will be better preparedto run similar projects in the future based on the lessons learned from this project. However, this capacity strengthening was not sufficiently measured and included in an overall monitoring and evaluation framework, which is considered a missed opportunity. A challenge remains to upscale the physical interventions carried out in this project to other suitable areas in Djibouti, which is rightly acknowledged in the PCR.

Ownership and sustainability of partnerships 4 3 Satisfactory. The project reports demonstrate convincing information thatthe Ministry has taken ownership of the project including the considerable achievements made as a result of the project. However, there as indicated earlier, there is no information on the way operation and maintenance of the water structures is secured beyond the project lifeline. The PCR report that the target communities have taken ownership of the water intake structures through their involvement in their construction. However, there is no sufficient details on how this involvement was done and the perspectives of the communities were not recorded. On this basis it is not possible to determine the level of ownership as perceived by the communities.

Environmental and social sustainability 3 2 The appraisal report and PCR note that

environmental and social impact studies were carried out under component A (construction of the hydraulic structures), however results of these studies were missing in the available information, nor any corrective measures seem to have been taken to mitigate negative social or environmental impacts.

OVERALL PROJECT COMPLETION RATING

2,96 2,4 Unsatisfactory.

Bank performance: 3 3 SatisfactoryBorrower performance: 3 3 SatisfactoryPerformance of other shareholders: 3 3 SatisfactoryOverall PCR quality: 2.45 Unsatisfactory. This is the average

rating based on ratings giving in appendix 1. See appendix 1 for justification.

9. PRIORITY FOR FUTURE EVALUATIVE WORK: PROJECT FOR PERFORMANCE EVALUTION REPORT, IMPACT EVALUTION, COUNTRY/SECTOR REVIEWS OR THEMATIC EVALUATION STUDIES:

- Project is part of a series and suitable for cluster evaluation

- Project is a success story

- High priority for impact evaluation

- Performance evaluation is required to sector/country review

- High priority for thematic or special evaluation studies (Country)

- PPER is required because of incomplete validation rating

Major areas of focus for future evaluation work:

a) Performance evaluation is required for sector/ country review

b) Cluster evaluation (institutional support)

c) Sector evaluation (budgetary support or public finance management reforms)

Follow up action by IDEV: Identify same cluster or sector operations; organize appropriate work or consultation mission to facilitate a), b) and/or c).

Division Manager clearance Director signing off

Data source for validation: Task Manager/ Responsible bank staff interviewed/contacted (in person, by telephone or

email) Documents/ Database reports

Attachment:

PCR evaluation note validation sheet of performance ratings

List of references

Appendice 1

PROJECT COMPLETION REPORT EVALUATION NOTE Validation of PCR performance ratings

PCR rating scale:

Score Description4 Very Good – Fully achieved with no shortcomings3 Good – Mostly achieved despite a few shortcomings2 Fair – Partially achieved. Shortcomings and achievements are roughly balanced1 Poor – very limited achievement with extensive shortcomings

UTS Unable to score/rateNA Non Applicable

Criteria Sub-criteriaPCRworkscore

IDEVreview

Reasons for deviation/comments

RELEVANCE Relevance of the project development objective (DO) during implementation

4 4 Highly satisfactory. The project is in line with the Poverty Reduction Strategy Paper (PRSP) adopted in Djibouti in July 2004 and the priority thrusts of AWF’s Operational Programme 2005-2009. It will, therefore, contribute to achieving the MDG (2015) and the African Water Vision for 2025 by supporting the establishment of appropriate and viable drinking and irrigation water services accessible to all the ruralpopulations of the Dikhil and Obock Regions and to respond to the need of the local population with an overall objective of improving their living conditions by harnessing water for domestic and agricultural use.

Relevance of project design (from approval to completion)

3 2 Unsatisfactory. The absence of the designs and estimated costs of major water facilities (flood spreading dam and catchment pond) during project appraisal and the updating of the working designs have resulted in a too lengthy extension of the projectimplementation period (87 months against 20 months initially designed). This could have been avoided through better estimation of time required to carry outthe civil works and to ensure that all PMU staff is fully on board from the start and avoid a high project overhead cost. In addition, no financial provision wasmade for the financing of a monitoring and evaluation system activities during project appraisal (by IFAD funding).

OVERALL RELEVANCE SCORE 3.5 3.0 Satisfactory

EFFECTIVENES Effectiveness in delivering outcomes


IDEVreview


S*

Outcome: Harness surface water resources in a pilot,integrated and sustainable manner

No score

3

All planned water facilities were constructed under State control with the participation of the beneficiary populations or by local contractors. This will help to significantly address the problems of water availability for the various uses of the populations of the regions concerned by the project.However, more outcomes should have been included,in line with those listed in the project appraisal report.

Effectiveness in delivering output

Component A: Construction of Water Facilities

Output 1: Working designs for water harnessing facilities prepared

Noscore

3 Satisfactory. PDS and FDS (storage dams, impluviumand catchment pond) as well as environmental and social impact assessments conducted, which was acknowledged in the PCR

Output 2: Pilot and new facilities constructed

Noscore

3 Satisfactory. All facilities constructed within contractual time periods, except the catchment pond, which was acknowledged in the PCR. The major slippage on the provisional schedule was due to the overall delay in project implementation during the first year and the deadline extensions that were required to search for supplementary financing to construct the flood spreading dam recommended in the PDS, and the occurrence of an exceptional flood that caused considerable damage to the dam which was 80% complete.

Component B: Geophysical Prospecting

Output 1: Geophysical prospecting conducted

Noscore

3 Satisfactory. Completed with delays due to delays in the project start-up in the first year.

Component C: Capacity Building

Output 1: The capacity of MAEM-RH services built

Noscore

3 Satisfactory. The consulting firm provided the capacity building services within the contractual timeframes. The major slippage on the provisional schedule was due to the overall delay in project implementation during the first year (virtually no activity) as a result of incomplete PMU staff.

Output 2: Experience shared

Noscore

3 Satisfactory. The consulting firm conducted the studytrip within the contractual timeframe. Not clear how this experience sharing was done and which


IDEVreview


with similar countries

countries/institutions benefited from this activity.

Component D : Water Sector Study

Output 1: The inventory and the study on the updating of monitoring networks conducted

Noscore

3 Satisfactory. The services of the consulting firm were delivered within the contractual timeframes. The major slippage on the provisional schedule was due to the overall delay in project implementation during the first year (virtually no activity) as a result of incomplete PMU staff.

Output 2: A projectbank for surface water harnessing using average-capacity facilities available

Noscore

3 Satisfactory. The services of the consulting firm were delivered within the contractual timeframes. The major slippage on the provisional schedule was due to the overall delay in project implementation during the first year (virtually no activity) as a result of incomplete PMU staff.

Component E: Project Management

Output 1: Project vehicles procured

Noscore

3 Satisfactory. The service provider procured the two vehicles within the contractual timeframes. The majorslippage on the provisional schedule was due to the overall delay in project implementation during the first year (virtually no activity) as a result of incomplete PMU staff, and to the lengthy national procurement process.

Output 2: Complete project staff recruited

Noscore

3 Satisfactory. The recruitment of PMU staff by the State, with the support of IFAD, was delayed by morethan a year, which seriously affected the functioning of the project during the first year (virtually no activity) as a result of incomplete PMU staff.

Output 3: Project financial audit conducted

Noscore

2 Unsatisfactory. The first audit covering the period from 1 February 2008 to 31 December 2010 was conducted in August 2011, and its results were approved by the Bank in February 2012. The final audit was conducted in April 2015 and its results were not yet available at the date of preparation of this completion report.

Development objective (DO)

Development objective rating

3 3 Satisfactory. It is very likely that the project has contributed to the development objective of improving the socio-economic conditions and reducing poverty levels of the target population. However, contributions of this project to development outcomes were not measured for example in a household survey carried out among the


IDEVreview


target population. An important project objective, increased water availability, was not well measured or estimated in terms of actual increased availability for the different water users and uses. Considering theinterest shown in the pilot experiment by many technical and financial partners, it is important to consolidate the outcomes achieved by reinforcing the institutional framework established in order to construct similar facilities in other regions of the country, and to pay special attention to the maintenance of the facilities constructed, especially the treatment of the watershed of the Sadai threshold dam to prevent it from rapid sedimentation by torrential rain floods. The project is not likely to fullyachieve its development objectives without a change in scope.

Beneficiaries coverage

Building the capacity of the beneficiary populations of the project area by involving them in the construction of facilities

No score

2 Unsatisfactory. There seems to be a discrepancy on type of reporting of beneficiaries between appraisal report and PCR. The PCR reports that the capacity of the beneficiary populations of the project area was built by involving them in the construction of water facilities, however without including total numbers, but with a reported gender balance of 40% women. There is insufficient detail on the way communities were involved, in order to evaluate the effectiveness of this involvement in terms of capacity building. Theappraisal report includes approximately 2,400 rural people that are expected to benefit from increased water availability, however the number of people is not reported in the PCR.

Training of 7 watertechnicians, 5 works technicians ,5 hydrogeology technicians and onedriller, one study trip, procurement of drillers, two vehicles and office furniture

No score

3 Satisfactory. Building the capacity of 17 technicians from MAEM-RH, study trip, training of a driller. However, some of the items listed as beneficiaries in the PCR are equipment and should not be included here.

Four water facility execution contractors, one works control firm and one PDS, FDS and environmental

No score

3 Satisfactory. Contractors are also reported as beneficiaries which is generally unusual in reporting of development projects.


IDEVreview


assessment consulting firm IFAD, FAO, the Resilience Project involved in the construction of the water facilities of the project’s pilot experiment

Noscore

3 Satisfactory. Not sure if IFAD and FAO should be listed as beneficiaries. Apart from their mere involvement in the project. There is no clear information confirming that they participated in trainings.

Beneficiaries coverage score

Noscore

3 Satisfactory

Unanticipated outcomes (positive or negative not considered in the project logical framework) and their level of impact on the project (high, moderate, low)

Institutional development

Noscore

n/a The capacity development of institutions and their staff is acknowledged, however without qualitative data.

Gender

Noscore

3 Satisfactory. Gender consideration were well considered the project appraisal report. Building the capacity of the beneficiary populations of the project area estimated at 2.400 persons by involving them in the construction of facilities, which women represent 40%.

Environment & climate change

Noscore

3 Satisfactory. The project to harness surface water for various uses, especially in the flood spreading dam construction area, generated the unexpected outcome of attracting fauna and wild birds to the dam impact area due to the presence of water. In the long run, thiscould lead to the development of hunting tourism in the Obock Region. There is no risk of contracting malaria because the inhabited villages are located at least five kilometres from the flood spreading dam.

Poverty reductionNo

scoren/a The project is expected to reduce poverty levels of

the target population, however there is a lack of data to measure and conclude that contributions made were significant.

Private sector development

Noscore

3 Satisfactory. Contractors have reportedly been capacitated through their involvement in the project.

Regional integration

Noscore

n/a No qualitative data in the PCR to support Regional cooperation

Other (specify): Tourism and health

Noscore

3 Satisfactory. The PCR mentions the positive impact of water for wildlife and, as a result, possible development of wildlife hunting. Also the possible risk of malaria is mentioned and considered negligible.

Unanticipated outcomes overall score

Noscore

3 Satisfactory


IDEVreview


EFFECTIVENESS OVERALL SCORE

3 3 Satisfactory

EFFICIENCY Timeliness (based on the initial closing date)

1 1 Ratio between planned project duration and project actual implementation time is 0,229, and less than 0,50.

Resource used efficiency

3 3 Ratio between median physical implementation of RFF outputs by all financiers and commitment rate is 0, 86, andmore than 0, 75 and less than 1.

Cost-benefit analysis

Noscore

UTR Unable to evaluate due to no information provided in the PCR.

Implementation progress (from the IPR)

3 2 A number of issues have substantially affected satisfactory implementation progress.

Other (specify)

OVERALL EFFICIENCY SCORE 2.3 2.0 Unsatisfactory

SUSTAINABILITY

Financial sustainability

2 2 Unsatisfactory. There is generally a lack of data regarding the financial sustainability post-project, including cost-benefit analysis and a long term cost-recovery plan. This wasnot included in the project. There is a lack ofprovision of details on how the constructed water intake and diversion structures will be maintained and which budget would be used.

The major water resource harnessing facilities did not benefit from existing appropriate mechanisms designed to ensure the sustainability of investments. This is particularly true for the major Sadai dam for spreading torrential rain floods carrying sediments and riprap from the basin upstream of the dam.The provision of a budget head to finance regular dam maintenance and dredging, and the long-run anti-erosion protection of the basin upstream of the dam will be essential to permanently ensure the positive outcomesexpected from this type of facility.


IDEVreview


Institutional sustainability and strengthening of capacities

4 2 Unsatisfactory. It is apparent from the available information that the Ministry has developed its capacity to manage this projectand it will be better prepared to run similar projects in the future based on the lessons learned from this project. However, this capacity strengthening was not sufficiently measured and included in an overall monitoring and evaluation framework, which is considered a missed opportunity. A challenge remains to upscale the physical interventions carried out in this project to other suitable areas in Djibouti, which is rightly acknowledged in the PCR.

Ownership and sustainability of partnerships

4 3 Satisfactory. The project reports demonstrateconvincing information that the Ministry hastaken ownership of the project including the considerable achievements made as a result of the project. However, there as indicated earlier, there is no information on the way operation and maintenance of the water structures is secured beyond the project lifeline. The PCR report that the target communities have taken ownership of the water intake structures through their involvement in their construction. However, there is no sufficient details on how this involvement was done and the perspectives of the communities were not recorded. On this basis it is not possible to determine the level of ownership as perceived by the communities.

Environmental and social sustainability

3 2 Unsatisfactory. The appraisal report and PCR note that environmental and social impact studies were carried out under component A (construction of the hydraulic structures), however results of these studies were missing in the available information, nor any corrective measures seem to have been taken to mitigate negative social or environmental impacts. The major water resource harnessing facilities did not benefit from existing appropriate mechanisms


IDEVreview


designed to ensure the sustainability of investments. This is particularly true for the major Sadai dam for spreading torrential rain floods carrying sediments and riprap from the basin upstream of the dam. The provision of a budget head to finance regulardam maintenance and dredging, and the long-run anti-erosion protection of the basin upstream of the dam will be essential to permanently ensure the positive outcomes expected from this type of facility.

OVERALL SUSTAINABILITY SCORE

3 2 Unsatisfactory

Criteria Sub-criteriaPCRWorkscore

IDEVreview


BANK PERFORMANCE

Proactive identification and resolution of problems at differentstage of the project cycle

Notrated

3 Satisfactory. The major problem for the project implementation arise from the under-estimation of the costs of water facilities during project appraisal, which led to major changes in the initial project, resulting in a major slippage on the projectimplementation schedule and the need to search for supplementary financing to implement the revised project. The projectobjectives were achieved, and the pilot experiment carried out successfully.

Use of previous lessons learned from previous operations during design and implementation

Notrated

3 Satisfactory

Promotion of stakeholder participation to strengthen ownership

Notrated

3 Satisfactory. Grassroots communities and the staff of the Sector Ministry were involved the selection of sites and the execution of works under State control andthrough contract in order to strengthen ownership with a particular project attention in the building of their capacity and the development of know-how (training of technicians, studies for inventory and updating of hydro-climatic monitoring networks, preparation of a project bank, etc.).

Enforcement of safeguard and fiduciary requirements

Notrated

3 Satisfactory

Design and implementation of Monitoring & Evaluation system

Notrated

1 Highly Unsatisfactory

Quality of Bank supervision (mix of skills in supervisory teams, etc)

Notrated

3 Satisfactory


IDEVreview


Timeliness of responses to requests

Notrated

3 Satisfactory

OVERALL BANK PERFORMANCE SCORE 3 SatisfactoryBORROWER PERFORMANCE

Quality of preparation and implementation

Notrated

3 Satisfactory

Compliance with covenants, agreements and safeguards

Notrated

3 Satisfactory

Provision of timely counterpart funding

Notrated

3 Satisfactory

Responsiveness to supervision recommendations

Notrated

3 Satisfactory

Measures taken to establish basis for project sustainability

Notrated

2 Unsatisfactory

Timeliness of preparing requests Notrated

3 Satisfactory

OVERALL BORROWER PERFORMANCE SCORE 3 Satisfactory PERFORMANCE OF OTHER STAKEHOLDERS

Timeliness of disbursements by co-financiers

Not rated

3 Satisfactory

Functioning of collaborative agreements

Not rated

3 Satisfactory

Quality of policy dialogue with co-financiers (for PBOs only)

Not rated

3 Satisfactory

Quality of work by service providers

Not rated

3 Satisfactory

Responsiveness to client demands Not rated

3 Satisfactory

OVERALL PERFORMANCE OF OTHER STAKEHOLDERS

3 Satisfactory

The overall rating is given: Very Good, Good, Fair and Poor.

(i) Very Good (HS) : 4(ii) Good (H) : 3(iii) Fair (US) : 2(iv) Poor (HUS): 1

DESIGN, IMPLEMENTAION AND UTILIZATION OF MONITIRING ANDEVALUATION (M&E)

Criteria Sub-criteriaIDEVScore

Comments

M&E DESIGNM&E system is in place, clear, appropriate and realistic

1

Highly Unsatisfactory. M&E system was not put in place during the project appraisal. M&E activities to be undertaken under IFAD funding without its clear commitment.

Monitoring indicators and monitoring plan were duly approved

2 Not sufficiently defined at the project appraisal.

Criteria Sub-criteriaIDEVScore

Comments

Existence of disaggregated gender indicator

1No clear disaggregated gender indicators in the project appraisal.

Baseline data were available or collected during the design

1 Highly Unsatisfactory.

Other, specify

OVERALL M&E DESIGN SCORE 1,25 Highly Unsatisfactory.M&E IMPLEMENTA-TION

The M&E function is adequately equipped and staffed


OVERALL M&E IMPLEMENTATION SCORE 1 Highly Unsatisfactory.M&E UTILIZATION

The borrower used the tracking information for decision


OVERALL M&E UTILIZATION SCORE 1OVERALL M&E PERFORMANCE SCORE 1

Highly Unsatisfactory.

PCR QUALITY EVALUATION

CriteriaPCR-EVN

(1-4)Comments

QUALITY OF PCR

1. Extent of quality and completeness of the PCR evidence and analysis to substantiate the ratings of the various sections

3 Satisfactory. But absence of an efficient system of generation of monitoring and evaluation data. The project appraisal report did not provide for the financing of a monitoring and evaluation system. The PCR lack evidence and data analysis to substantiate the ratings of the various sections.

2. Extent of objectivity of PCR assessment score 3 Satisfactory.

3. Extent of internal consistency of PCR assessment ratings; inaccuracies; inconsistencies; (in various sections; between text and ratings; consistency of overall rating with individual component ratings)

3 Satisfactory.

4. Extent of identification and assessment of key factors (internal and exogenous) and unintended effects (positive or negative) affecting design and implementation

3 Satisfactory.

5. Adequacy of treatment of safeguards, fiduciary issues, and alignment and harmonization

3 Satisfactory.

6. Extent of soundness of data generating and analysis process (including rates of returns) in support of PCR assessment


7. Overall adequacy of the accessible evidence (fromPCR including annexure and other data provided)

1 Highly Unsatisfactory. The project is not likely to achieve its development objectives without a change in tools and systems for data collection andmanagement.

8. Extent to which lessons learned (and recommendations) are clear and based on the PCR assessment (evidence & analysis)

2 Unsatisfactory.

9. Extent of overall clarity and completeness of the PCR

3 Satisfactory.

Other (specify)

PCR QUALITY SCORE 2.45 Unsatisfactory.

PCR compliance with guidelines (PCR/OM ; IDEV)

1. PCR Timeliness (On time = 4; Late= 1) 1 PCR was produced 7 months after project closure which is longer than the required 6 months.

2. Extent of participation of borrower, Co-financiers & field offices in PCR preparation

UTR

3. Other aspect(s) (specify)

PCR COMPLIANCE SCORE

*** rated as Very Good (4), or Good (3), or Fair (2), or Poor (1)

References

African Development Bank Group, Policy for integrated water resources management, April 2000

African Development Bank Group, Staff Guidance on Project Completion Reporting and Rating, Quality Assurance and Results Department (ORQR), Quality Assurance Division (ORQR.2), August 2012

African Development Bank Group, Djibouti Country Strategy Paper 2011-2015. August 2011

Ministère de l’Agriculture, de l’Eau, de la Pêche, de l’Elevage et des Ressources Halieutiques (MAEPE-RH). Plan Directeur 2009-2018 En Republique de Djibouti. Fevrier 2009

Republique de Djibouti. Vision Djibouti 2035