the world bank operations evaluation department
TRANSCRIPT
Director-General, Operations Evaluation: Gregory K. Ingram Acting Director, Operations Evaluation Department: Nils Fosdvedt Manager, Sector and Thematic Evaluation Group: Alain Barbu Task Manager: Martha Ainsworth
Review of Impact Evaluation Methodologies Used By The
Operations Evaluation Department Over Past 25 Years
Anju Gupta Kapoor
T H E W O R L D B A N K O P E R A T I O N S E V A L U A T I O N D E P A R T M E N T
2002The World Bank
Washington, D.C.
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
OPERATIONS EVALUATION DEPARTMENT ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION The Operations Evaluation Department (OED) is an independent unit within the World Bank; it reports directly to the Bank’s Board of Executive Directors. OED assesses what works, and what does not; how a borrower plans to run and maintain a project; and the lasting contribution of the Bank to a country’s overall development. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank’s work, and to provide accountability in the achievement of its objectives. It also improves Bank work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. OED Working Papers are an informal series to disseminate the findings of work in progress to encourage the exchange of ideas about development effectiveness through evaluation. The findings, interpretations, and conclusions expressed here are those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent. . Contact: Operations Evaluation Department Partnerships & Knowledge Programs (OEDPK) e-mail: [email protected] e-mail: [email protected] Telephone: 202-458-4497 Facsimile: 202-522-3125 http:/www.worldbank.org/oed
Acronyms FY Fiscal year HIAL Higher Impact Adjustment Lending IER Impact Evaluation Report NEP National Extension Project OED Operations Evaluation Department OEDPK OED, Partnership and Knowledge Programs PFI Participating financial institution RWS Rural water system SENSA National Environmental Sanitation Service SCARP Salinity Control and Reclamation Project SMI Small and medium industries SSA Sub-Saharan Africa SSE Small-scale enterprises
i
Contents Introduction.................................................................................................................................... 1
OED Impact Evaluations .............................................................................................................. 1
OED Impact Evaluations with a Counterfactual ........................................................................ 3
Typology of OED Impact Evaluations with a Counterfactual ............................................ 4 Cost Analysis....................................................................................................................... 7 Task Manager Interviews.................................................................................................... 8
Summary and Recommendations................................................................................................. 9
Annexes
Annex 1. List of OED Impact Evaluation Reports ........................................................................11
Annex 2. Key Features of the Methodologies Used in Impact Evaluation Reports with a Counterfactual...............................................................................................................................................14
Annex 3. Historical Cost Data for OED Impact Evaluations.........................................................18
Annex 4. Results of Survey of OED Impact Evaluation Task Managers ......................................19
Annex 5. Project and Evaluation Objectives for OED Impact Evaluations with a Counterfactual20
Boxes
Box 1. Approaches to Impact Evaluation and the Counterfactual ...................................................4
Box 2. Key Findings of OED Impact Evaluations with a Counterfactual .......................................7
Box 3. OED Impact Evaluations that had an Impact, According to OED Task Managers..............9
Figures
Figure 1. Number of Impact Evaluations by Fiscal Year.................................................................2
Figure 2. Impact Evaluations With and Without a Counterfactual ..................................................2
Figure 3. Distribution of OED Impact Evaluations by Sector and the Use of a Counterfactual......3
Figure 4. Distribution of OED Impact Evaluations by Completion Year ........................................3
Tables
Table 1. OED Impact Evaluations with Non-experimental or Quasi-experimental Design ............5
Table 2. Impact Evaluations with Unspecified Comparison Group.................................................6
1
I. INTRODUCTION
1. The purpose of this review is to: (i) take stock of impact evaluations conducted by the Operations Evaluation Department (OED) over the past 25 years; (ii) examine the methodologies used; and (iii) extract lessons for use in future evaluations. The review is based on a portfolio of 78 impact evaluations conducted by OED since 1979, the cost data on some evaluations (provided by OED), and interviews with a few task managers.
2. An impact evaluation is an assessment of outcomes of key actions/inputs relative to what would have happened in the absence of the intervention. Counterfactual (control/comparison)1 analysis is an important element of an impact evaluation that helps in ascertaining the extent to which the positive and negative consequences observed in the treatment area can be attributed to the intervention and not to other external conditions. A counterfactual analysis “nets out” all observed development changes in the treatment area that would have been observed even if the intervention had not taken place. Of the 78 impact evaluations conducted by OED, 21 presented analysis based on comparison with a counterfactual.
3. Section II describes the nature of the 78 OED impact evaluation reports. Section III presents in some detail the methodologies adopted by 21 impact evaluation reports that were based on counterfactuals, the costs of those for which data could be found, and the results of a survey of the OED task managers for nine of the evaluations, concerning lessons for future OED impact evaluations. The final section summarizes the results and suggests some recommendations for future OED classification of impact evaluations.
II. OED IMPACT EVALUATIONS
4. OED has been conducting impact evaluations of World Bank assistance since 1979. This report analyzes the methodologies used in 77 OED Impact Evaluation Reports listed in the World Bank “ImageBank” database and one additional OED sector study that used a counterfactual.2 Annex 1 lists the 78 evaluations with their sectoral classification and completion dates. The Partnerships and Knowledge Programs Unit of OED (OEDPK) is responsible for the sectoral classification of OED products, including impact evaluations.3 This review has used the classification provided by OEDPK.
5. Each of the 78 reports has been classified under one of 10 sectors: agriculture; education; energy; industry; population, health, and nutrition; resettlement; transport; urban; water and sanitation; and other. More than half of the evaluations (43 of the 78 reports) have been conducted in the agriculture sector. The education and water and sanitation sectors had 6 reports each (8 percent). The industry and resettlement sectors had 4 reports each (5 percent); population, health, and nutrition, transport, urban, and energy sectors had 2 reports each (about 3 percent); and 7 reports (9 percent) were classified under “other” sectors.
6. The 78 evaluations cover the period FY79 to FY02, for an average 3 to 4 reports per year. However, the distribution over years was not uniform (Figure 1). Only one evaluation was conducted in FY79. The annual number of impact evaluations increased during the early 1980s but declined rapidly
1. A counterfactual analysis requires a control or comparison group of non-participants. The terms control and comparison groups are used interchangeably in many evaluations. However, a control group consists of participants that are selected at random from within a well-defined set from which participants of the treatment group are also selected, while a comparison group consists of participants that are purposively matched to the participants of the treatment group.
2. “Higher Impact Adjustment Lending (HIAL) in Sub-Saharan Africa,” Report 19797. The OED web site does not list impact evaluations as a separate evaluation category, so this study relied primarily in those listed in the ImageBank.
3. Some OED task managers indicated that the sectoral assignment was incorrect.
2
after 1984. OED decided to stop impact evaluations in the early 1990s, but restarted them in 1993 with a different focus. Between FY93 and FY98 the number of impact evaluations surged, with at least 4 evaluations per year. The number of impact evaluations then declined again—to 3 in FY99, and to only one in FY00.
Figure 1. Number of Impact Evaluations by Fiscal Year
0 2 4 6 8
10
79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 00
Fiscal Year
Num
ber o
f Eva
luat
ions
Total number of evaluations: 78 7. An initial screening of the 78 evaluations was done to determine the eligibility for in-depth review. An impact evaluation was included in the review if the evaluation design provided for counterfactual data collection. Twenty-five4 of the 78 impact evaluations involved the use of counterfactual data. The remaining 53 had no counterfactual analysis for several reasons, the most notable being nationwide application of the project; the substantial time lag between completion of the project and initiation of the evaluation such that the demonstration effect could not be distinguished from treatment effect; and the limitations of the evaluation design.
Figure 2. Impact Evaluations With and Without a Counterfactual
55%
13%4%
28%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
No counterfactual, field surveys
No counterfactual, no field surveys
Counterfactual, field surveys
Counterfactual, no field surveys
Total number of evaluations: 81
8. Reviewing the reports from a different perspective, 13 of the 78 evaluations had no fieldwork and were largely based on desk reviews. Three of the 13 reports with no fieldwork nevertheless had counterfactual analysis. Figure 2 presents the distribution of the 78 evaluations based on the presence or absence of a counterfactual.
9. Counterfactual analysis was prevalent in the urban and water and sanitation sectors, but was not prevalent in the resettlement and energy sectors (Figure 3). Seven of the 43 reports prepared for the agriculture sector had counterfactual analysis. The other 18 evaluations with counterfactual analysis included 2 reports for the industry sector; one each for the population, health, and nutrition and transport sectors; 3 for the education sector; 5 for the water and sanitation sector; and 2 for the urban sector. Further,
4. Of these 25 evaluations, 4 collected counterfactual data, but there was limited evidence that it was used. These four were therefore excluded.
3
four reports under “other” sectors had a counterfactual analysis.
Figure 3. Distribution of OED Impact Evaluations by Sector and the Use of a Counterfactual
4
5
2
1
0
1
2
0
3
7
3
1
0
1
4
1
2
2
3
36
0% 20% 40% 60% 80% 100%
Other
Water & Sanitation
Urban
Transport
Resettlement
HNP
Industry
Energy
Education
Agriculture
% of impact evaluation reports
With counterfactual Without counterfactual
10. A larger proportion of more recent OED impact evaluations used a counterfactual, compared with impact evaluations before FY95 (Figure 4). Fifty-eight percent of the evaluations (45 evaluations) were conducted before FY95, of which only 9 (20 percent) had counterfactual analysis. The proportion of evaluations with a counterfactual to total evaluations progressively increased over the years FY95 to FY01; over a third of the evaluations in each year after FY94 had analysis based on counterfactuals.
Figure 4. Distribution of OED Impact Evaluations by Completion Year
Total number of evaluations: 78
0
10
20 30
40
50
pre-FY1995 FY1995 FY1996 FY1997 FY1998 PostFY1998
Num
ber o
f Im
pact
Eva
luat
ions
Total number of evaluations Evaluations with a counterfactual
III. OED IMPACT EVALUATIONS WITH A COUNTERFACTUAL
11. This section presents a typology of the methodologies used in the 21 OED impact evaluations conducted since FY77 that have attempted to establish a counterfactual. Information is also presented on the costs of a subset of these evaluations and the results of a short survey of their OED task managers to understand better any lessons or recommendations for future OED impact evaluations.
4
Typology of OED Impact Evaluations with a Counterfactual
12. An impact evaluation typically includes:5
• Data on a sample of participants from the treatment group at different levels of analysis (e.g., household, farmers, institutions, municipalities, etc.).
• Comparable data on non-participants from a control or comparison group at each level of analysis. • Baseline data and follow-up data for the treatment and control/comparison groups. • Sample selection based on appropriate sampling methodology (e.g., probability sampling, which
attaches a specific chance of selection to each unit in the universe, normally referred as random sampling), and sufficient sample size to establish statistically valid inferences.
• Qualitative information to inform interpretation of results. 13. Based on country and project conditions, an impact evaluation can use different approaches to select the control or comparison group for assessing the impact of the intervention on the treatment group (see Box 1).
Box 1. Approaches to Impact Evaluation and the Counterfactual
Experimental or Randomized Treatment-Control Design
Participants are randomly assigned to treatment and control groups. The two groups are expected to be similar in all respects except that the treatment group receives the development intervention and the control group does not.
Non-experimental or quasi-experimental design
Participants in the comparison group are drawn from a larger population and matched to the participants in the treatment group, based on a set of observed characteristics. Propensity score matching, matching based on geographic proximity or on socio-economic proximity, instrumental variable or statistical controls, and reflexive comparisons are some of the matching methods employed in creating a comparison group.
Other Approaches to Impact Evaluation
Counterfactual generated using multiple assumptions. A counterfactual can be generated using simulation modeling, by comparing outcomes in countries or households that received an intervention with those that did not, or by trying to control statistically for differences in initial conditions.
Theory-based evaluation assesses each theory and assumption about the project during its implementation, over time. Data are collected at each step of implementation, flowing from assumptions to actions. If the intervention does not deliver as expected, it is possible to pinpoint where in the chain of assumptions and actions the problem occurred. This method is often used in situations where an intervention is applied nationwide, only “before” and “after” data are available, and the possibilities for attributing results via a counterfactual are limited.
Implicit counterfactual is based on comparing the treatment group to a comparison group with a similar intervention from another source. In such an evaluation, the counterfactual helps to address the question of the suitability of the source of the intervention rather than the suitability of the intervention itself.
Source: Judy L. Baker. 2000. Evaluating the Impact of Development Projects on Poverty: A Handbook for Practitioners. Washington, D.C.: World Bank.
14. The 21 OED impact evaluations with a counterfactual in Annex 2 can be broadly classified into 5 groups:
5. Judy L. Baker. 2000. Evaluating the Impacts of Development Projects on Poverty: A Handbook for Practitioners. Directions in Development Series. Washington, D.C.: The World Bank.
5
15. The first group, with only one evaluation, used a control group to establish the counterfactual. The evaluation of the community-based rural water systems and the development of village committees in Paraguay had data from San Juan Hospital that served 10 large villages, of which five had a rural water system (RWS) financed by the World Bank. The hospital records provided data on all children treated in the hospital for two years: 1987 (pre-intervention) and 1996 (post-intervention). The study also conducted analysis on 17,644 individuals from RWS I and II and 101,787 individuals to be in RWS IV from a survey by the National Environmental Sanitation Service’s (SENSA’s) department of operations in 1996–97. In addition, interviews were conducted with directors and employees of local organizations, representatives of the local municipality, central and municipal officials, system operators, bilateral donors, and consultants.
16. The second group, with 15 evaluations, used non-experimental or quasi-experimental design to establish a counterfactual (Table 1). These consisted of three subgroups: three evaluations had no baseline data and historical data analysis was based on annual country-level data; one evaluation had non-random or self-selected6 participants in the treatment and comparison group; the remaining 11 evaluations had baseline data and involved random selection of participants. A counterfactual was established in a similar way in all the 15 evaluations. For example, in the evaluation of the World Bank support for small-scale enterprises (SSE) in Ecuador, 145 treatment SSEs were compared to 55 SSEs that did not receive the Bank support. The accounting data of both sets of SSEs were available from secondary sources. The quantitative data were supplemented with structured interviews with officials from participating financial institutions.
Table 1. OED Impact Evaluations with Non-experimental or Quasi-experimental Design
Group Features Country Project Bangladesh Shallow Tubewells Project Pakistan SCARP Transition Pilot Project Indonesia The Legacy of the Kampung Improvement Program Ecuador World Bank Support for Small-Scale Enterprises Brazil/ Philippines
Building Institutions and Financing Local Development
Philippines Second Rural Credit Project Turkey Seyhan Irrigation Project (Stage II) Philippines Small & Medium Industries Development Projects (I–IV) Morocco Socioeconomic Influence of Rural Roads India Tamil Nadu Integrated Nutrition Project
2A
Baseline data based on existing secondary sources or recall; random selection of participants; qualitative information
Korea Delayed Development of the Cholla Region: An Institutional Study
India Early Experience with Involuntary Resettlement India West Bengal Agricultural Development Project
2B No baseline data; random selection of participants; qualitative information Kenya Development of Housing, Water Supply and Sanitation in Nairobi
2C Baseline data based on existing secondary sources or recall; non-random selection of participants; qualitative information
Brazil Learning from Best Practice in Five Urban Projects
6. Self-selection can introduce biased responses.
6
17. The third group, with one evaluation, used multiple assumptions to generate a counterfactual. The objectives of this evaluation of Higher Impact Adjustment Lending (HIAL) were to: (i) determine whether there was an improvement in adjustment lending; and (ii) assess the difference the HIAL approach has made in Sub-Saharan Africa (SSA). Thirteen of the 17 HIAL SSA countries were the treatment group and 35 non-HIAL SSA countries and 24 non-SSA countries were used as the comparison groups. The mean value comparison between the treatment and the control group assumes that external conditions affect both sets of countries in the same way. A panel was created using all fiscal data available from pre-existing secondary sources.
18. The fourth group, with one evaluation, used theory-based evaluation to evaluate World Bank agricultural extension projects in Kenya. The objective of this evaluation was to assess empirically the impact of the National Extension Projects (NEP I and NEP II). Theory-based evaluation was used since NEP was introduced nationwide between 1983 and 1998 making the estimation of a counterfactual extremely difficult without making strong assumptions. A panel on households was created by re-interviewing households from a 1982 survey (pre-intervention) conducted by the Africa Technical Department. Contingent valuation methods and regression techniques were used, maintaining a clear distinction between impact of a particular system and the impact of extension. In addition, the evaluation involved discussions with current and former extension staff, ministry staff, focus groups, and district-level staff.
19. Three evaluations made up the fifth group, in which the criteria for the selection of the comparison group were not clearly specified (see Table 2). For example, for the evaluation of the Bank’s support for the first and second education projects in Colombia, all 19 schools that had the Bank intervention were selected as the treatment group. Ten schools from various parts of the country were selected as a comparison group. The criteria for the selection of these 10 schools as comparison group is not specified.
Table 2. Impact Evaluations with Unspecified Comparison Group
Design Grouping Country Project Sri Lanka World Bank support for small & medium
industries (SMI) Colombia First and Second Education Projects
Baseline data based on existing secondary sources or recall, random selection of participants, and qualitative information
Thailand First Education Project
20. The findings of two of these evaluations—one with a control group and one using theory-based evaluation supplemented by panel data—are highlighted in Box 2.
7
Box 2. Key Findings of OED Impact Evaluations with a Counterfactual
OED impact evaluations have documented projects and programs that worked and those that have been ineffective, in each case with important recommendations for improving performance.
Community-Based Rural Water Systems and the Development of Village Committees in Paraguay
The World Bank loan led to establishment of 275 functioning community-based rural water systems (RWS) and raised the coverage of rural areas from around 1 percent to about 20 percent over a 20-year period. The National Environmental Sanitation Service (SENASA) delivered 952 training courses on health and hygiene and on water system management, benefiting 160,000 people, and helped create 424 local organizations. RWS created employment and leadership opportunities for women in rural villages, improved income distribution and local coordination capacity, reduced the disparity between rich and poor in terms of basic household amenities, improved environmental hygiene, strengthened the private sector, and produced time and energy savings and productivity gains due to health improvements at the household level.
The impact evaluation recommended that RWS should: (i) have sustainable tariffs; (ii) provide administrative help for weaker local organizations (juntas); and (iii) cover water meters in the loans for new juntas and provide access to credit to existing juntas that are up to date with their payments for universal metering of household use. In addition, SENASA should be provided with assistance in knowledge management and evaluation capacity.
World Bank Agricultural Extension Projects in Kenya
The evaluation found that the World Bank projects had no lasting improvement in the effectiveness of agricultural extension services. Institutional development was limited, with continued weak management, and the evaluation was not able to establish a positive rate of return on current expenditures for extension. The ineffectiveness of the approach adopted in the projects has led some extension agents to adopt alternative approaches to deliver extension services to their clients.
Cost Analysis
21. Data on costs financed from OED budget sources were available for 10 of the 21 evaluations with counterfactuals (Annex 3). Cost data were available for all evaluations from the first, third, and fourth groups; 6 of the 15 evaluations from the second group; and 1 of the 3 evaluations from the fifth group.7
22. The experimental or randomized control evaluation of community-based rural water systems and the development of village committees in Paraguay was the least expensive evaluation ($36,000, in 1995 U.S. dollars). On the other hand, the theory-based evaluation of World Bank agricultural extension projects in Kenya was the most expensive ($252,000). The evaluation that used a counterfactual generated using multiple assumptions, “Higher Impact Adjustment Lending (HIAL) Initial Evaluation” was also relatively low, at $88,000. The quasi-experimental evaluations ranged in cost from a low of $44,000 for the Pakistan SCARP Transition Pilot Project to a high of $179,000 for the evaluation of World Bank Support for Small-Scale Enterprises in Ecuador.
23. Is the cost related to the approach used to estimate a counterfactual? No. The costs are directly related to the type and extent of data collected for the evaluation. The evaluations that used secondary sources, such as the community-based rural water systems in Paraguay, SCARP transition pilot project in Pakistan, and HIAL evaluation, were low cost. The evaluations that were based on collection of additional data through sample surveys were more costly and the cost increased with the number of surveys and
7. The cost data covers staff time, travel costs, and consultant costs. However, expenditures out of trust funds are not included and were not available.
8
regions covered. Evaluations of the NEP projects and the housing, water supply and sanitation in Kenya, for example, covered over 500 individuals each. The evaluation of the Second Rural Credit Project in Philippines covered over 700 farmers and the evaluation of SSE in Ecuador covered 200 firms. Both the NEP project and the SSE project had large regional coverage, making them the two most costly evaluations.
Task Manager Interviews
24. In an effort to glean lessons from past OED impact evaluators, 13 of the task managers of the 21 evaluations8 with counterfactuals were asked to complete a short questionnaire. The questions included their assessment of: the extent of collaboration between OED staff, Bank operational staff, and developing country research institutions; the influence of the impact evaluation on the Bank’s and borrower’s policies and programs; 9 the adequacy of the time and budget to conduct the study and disseminate results; and advice for future OED impact evaluations. Of the 13 task managers to whom a questionnaire was sent, nine responded and their responses were tabulated (Annex 4).
25. Extent of collaboration with Bank operational staff and governments. Seven of the nine task managers indicated that the extent of collaboration between the Bank regional staff and the OED team conducting the impact evaluation was the same (or more in one case) as required, while two felt that it fell short of what was required. A similar picture emerged for the collaboration between the government officials and the OED team conducting the impact evaluation.
26. Use of local public or private research institutions. Six of the nine impact evaluations used local public or private research institutions in one or more roles: data collection (6), data analysis (3), report writing (2), and discussions (2).
27. Influence of impact evaluations on the Bank’s policies and programs. Eight of the nine task managers believed that the impact evaluation had no impact on the design of the projects that followed those evaluated. Three reasons were given: there was no new project (3 evaluations); the follow-on project was already initiated before the evaluation was completed (1); and the project design was satisfactory and required no change (3). The task manager for one impact evaluation provided no reason for the lack of impact on subsequent project design. The ninth task manager reported that the impact evaluation did have an impact by ensuring that greater emphasis was given to monitoring outcomes in the subsequent project (Box 2). Despite the consensus that there was limited impact on subsequent projects, seven task managers noted that the impact evaluations resulted in better understanding of policies for the sector. Three task managers rated the overall influence of the impact evaluation on the Bank’s policies and programs as substantial, one rated it moderate, and four rated it marginal.
28. Influence of impact evaluations on government policies and programs. Although seven task managers indicated that the evaluation had an influence on the government’s policies and programs (Box 3), only one task manager rated the overall influence as substantial; two rated the impact moderate and four rated it marginal.
29. Time, resources, and effort to disseminate findings and follow-up actions. Six task managers indicated their satisfaction with the time and resources devoted to conducting the impact evaluation. However, only two were satisfied with the level of dissemination and follow-up actions; two had mixed 8. Eight impact evaluations either did not mention the name of the task manager or the task manager could not be contacted.
9. The responses reflect the task managers’ own assessments of the impact of their evaluation but present only one perspective. A formal evaluation of the impact of past OED impact evaluations was beyond the scope and objectives of this paper.
9
feelings, and two indicated that seminars to disseminate findings had to be cancelled because of the audience resistance to the messages.
Box 3. OED Impact Evaluations that had an Impact, According to OED Task Managers
Building Institutions and Financing Local Development: Lessons from Brazil and the Philippines
Objective: To assess both direct impacts on the beneficiaries as anticipated by the projects, and indirect impacts on institutional capacity building and the development of local economies focusing on fiscal and financial management, employment, and income generation in the participating municipalities.
Impact: The task manager rated the impact of the evaluation on the policies and programs of both the Bank and the country’s as substantial. The follow-up project gave more emphasis to monitoring of outcomes in the Philippines; the Bank’s policy shifted toward greater assistance at the local level (municipalities); and government policy supported this by more support for decentralization. The task manager was very satisfied with the level of collaboration between different entities involved—Bank regional staff, government officials, and the OED team.
World Bank Agricultural Extension Projects in Kenya
Objective: (i) To assess empirically the impact of the NEP I and NEP II using theory-based evaluation and panel household data without a control group. (ii) To draw appropriate policy conclusions, maintaining a clear distinction between the impact of a particular system and the impact of extension.
Impact: The task manager rated the impact of the evaluation on the Bank’s policies and programs as substantial and on country’s policies and programs as moderate. The evaluation resulted in: (i) a decisive shift in the Bank’s policy away from monolithic and rigid agricultural extension systems; (ii) considerable debate to reconsider government policy toward extension strategy and service delivery; and (iii) thinking about good/bad practice in extension policy and project design in neighboring countries and the international community at large. However, the task manager was not satisfied with the level of collaboration between different entities involved—Bank regional staff, government officials and the OED team.
30. Recommendations for future impact evaluations. Task managers for two of the evaluations said that they would make insignificant changes to their former study if the study were to be conducted again. The other six mentioned that they would allocate more time or more resources, or have more local collaboration. Two task managers advocated greater reliance on secondary data, especially in data-rich countries. Three task managers indicated that more rigorous and thorough analysis is the key to good impact evaluation.
IV. SUMMARY AND RECOMMENDATIONS
31. The international community is increasingly calling for results-based programming and spending. This paper reviewed the methodology and design of 78 OED evaluations of the impact of World Bank assistance since 1977. This list was obtained from the World Bank “ImageBank” database and included all OED “Impact Evaluation Reports,” plus one OED Sector Study highlighted by OEDPK as having a counterfactual. There was no list of impact evaluation reports as a separate category on OED’s websites, nor were the criteria documented for classifying an evaluation as an impact evaluation.
32. This list of 78 evaluations reveals two “waves” of interest in impact evaluation in OED—from 1980 to 1985 and from 1993 to 1998. These are the years with the greatest number of impact evaluations annually. More than half of OED impact evaluations have been in the agriculture sector.
10
33. Of the 78 impact evaluations, 25 (nearly a third) compared outcomes of the project to a counterfactual. Most of those with a counterfactual also collected data in the field. Nearly two-thirds of the impact evaluations tried to estimate impact without a counterfactual, which is sometimes extremely difficult to construct—for example, when an intervention is nationwide. If the evaluation can account for the influence of other factors, this type, too, can provide some evidence of impact. Most of the impact evaluations without a counterfactual nevertheless conducted field surveys.
34. This paper examined in detail the methodology of 21 of the 25 OED impact evaluations with a counterfactual. Only one used a randomly selected control group; the vast majority (15 impact evaluations) used quasi-experimental designs. Information on the approximate costs of 10 impact evaluations with counterfactuals and dating from FY96-99 was gleaned from budget files. The cost of the evaluations ranged from $36,000 to $252,000 in 1995 dollars, although the data may not include all of the costs (such as those funded from trust funds or external sources). The impact of these evaluations on the design of subsequent projects and on government and World Bank policy has never been formally studied. The task managers for nine of the impact evaluations with a counterfactual were reached with a short questionnaire to assess their views on the impact and on lessons for future OED impact evaluation.
35. The conclusion from this study is that OED has substantial experience with impact evaluations, including those that constructed counterfactuals, but this experience is not well documented and the impact on Bank and borrower performance has not been tracked. OED should consider the following actions to improve the documentation and better understand the impact of its impact evaluations:
1. Add “Impact Evaluations” as a distinct, searchable product on OED’s website with a description of the criteria for defining an impact evaluation. This will improve access of the public to evidence of the impact of Bank assistance.
2. Maintain an on-line impact evaluation database that includes key methodological characteristics and available information on the costs of the evaluation, the extent of collaboration with local research institutions, the institutions involved, the name of the OED task manager and the task manager(s) for the project(s) under evaluation.
3. Greater effort should be made to systematically collect information on the impact of OED impact evaluations, as assessed by government, operational staff, beneficiaries and other donors, and as reflected in real changes in the way things are done.
Annex 1 11
Annex 1. List of OED Impact Evaluation Reports
Sector/ Region
Country Title Report # Date
Agriculture
AFR Kenya Smallholder Agricultural Credit Project 2968 5/1/1980 AFR Sudan Roseires Irrigation Project 3051 6/1/1980 AFR Burundi Arabica Coffee Improvement Project 3314 2/1/1981 AFR Malagasy Republic Lake Alaotra Irrigation Project 3600 8/1/1981 AFR Kenya Livestock Development Project 3622 9/1/1981 AFR Malawi Shire Valley Agricultural Consolidation Projects 4850 12/1/1983 AFR Kenya First and Second Forest Plantations Projects 4911 2/1/1984 AFR Dahomey Hinvi Agricultural Development Project 5029 4/1/1984 AFR Côte d’Ivoire First, Second, Third and Fourth Oil palm and coconut
projects 5072 5/1/1984
AFR Gambia Agricultural Development Project 5125 6/1/1984 AFR Togo, Côte d’Ivoire,
and Upper Volta Agricultural and Rural Development Projects 6878 6/30/1987
AFR Nigeria Kano and Sokoto Agricultural Development Projects 14767 6/29/1995 AFR Malawi The World Bank and the Agricultural Sector 17898 5/20/1998 AFR Tanzania Agriculture, and the World Bank: an OED review 18111 6/30/1998 AFR Mauritania Second Livestock Project 18133 6/30/1998 AFR Kenya World Bank Agricultural Extension Projects in Kenya 19523 6/30/1999 EAP Papua New Guinea New Britain Smallholder Development Pr. 3070 7/1/1980 EAP Malaysia Muda River and Kemubu Irrigation Projects 3587 8/1/1981 EAP Philippines Second Rural Credit Project 4557 6/1/1983 EAP Korea Pyongtaek-Kumgang Irrigation Project 5941 11/1/1985 EAP China North China Plain Agriculture Project 13243 6/29/1994 EAP Thailand Second Tree Crops and Third Rubber Replanting Projects 13244 6/29/1994 EAP LAC Philippines Mexico
and Thailand World Bank Experience with Irrigation Development: Socio-economic, Institutional and Technical Impact and Lessons: Philippines and Thailand
8494 3/22/1990
EAP SAS
Thailand Myanmar Vietnam; Bangladesh
Irrigation: Operation, Maintenance, and System Performance in Southeast Asia: an OED Impact Study
15824 6/27/1996
ECA Turkey Seyhan Irrigation Project – Stage Two 5745 6/1/1985 LAC Mexico Third Irrigation Project 2559 6/1/1979 LAC Peru San Lorenzo Irrigation and Land Settlement (stage three)
Project 3933 5/1/1982
LAC Colombia Third Atlantico Irrigation Project 3959 6/1/1982 LAC Dominican Republic Livestock Development Project 4732 10/1/1983 LAC Mexico and Morocco World Bank Experience with Irrigation Development:
Socio-economic, Institutional and Technical Impact and Lessons: Mexico and Morocco
7876 6/15/1989
MNA Tunisia Reduction Flood Hazards and Traffic Congestion: SFAX Flood Protection Project
16777 6/20/1997
SAS Indonesia Irrigation Rehabilitation Project 4575 6/1/1983 SAS Ceylon Lift Irrigation Project 5634 5/1/1985 SAS Bangladesh Foodgrain Storage Project 6567 12/31/1986 SAS India and Indonesia Seeds Projects 6575 12/31/1986
Annex 1 12
Sector/ Region
Country Title Report # Date
SAS Bangladesh Shallow Tubewells Project 11031 8/22/1992 SAS India Karnataka Irrigation Project 12132 6/29/1993 SAS India West Bengal Agricultural Development Project 12140 6/29/1993 SAS Sri Lanka Smallholder Rubber Rehabilitation and Fourth Tree Crops
Projects 14760 6/30/1995
SAS Bangladesh, India and Pakistan
Seed Projects 14761 6/30/1995
SAS Pakistan On-Farm and Command Water Management and Irrigation Systems Rehabilitation Projects
15863 6/28/1996
SAS Sri Lanka Kurunegala and Second Rural Development Projects 16418 3/28/1997 SAS India Karnataka & Rajasthan Dairy Development Projects and
Madhya Pradesh, and First and Second National Dairy Projects
16848 6/30/1997
Education
AFR Malawi Education Project 7919 6/30/1989 EAP Philippines Agricultural Education Project 4472 5/1/1983 EAP Thailand Vocational Education Project 5069 5/1/1984 EAP Indonesia First and Second Education Projects 7767 5/22/1989 LAC Colombia First and Second Education Projects 3062 6/1/1980 LAC Colombia First and Second Sustainability of Education Projects 7926 6/30/1989
Energy
AFR Ghana Kpong Hydroelectric Project 12141 6/30/1993 EAP Thailand Khao Laem Hydroelectric Project 12131 6/29/1993
Industry
EAP Philippines First, Second, Third and Fourth Small & Medium Industries Development Projects
18041 6/18/1998
MNA Arab Republic of Egypt
El-Dikheila Reinforcing Bar Project 15555 4/23/1996
SAS India Maharashtra Petrochemical Project 13260 6/30/1994 SAS Sri Lanka World Bank Support for Small & Medium Industries in Sri
Lanka: An Impact Evaluation 16790 6/24/1997
Health, Nutrition & Population
LAC Brazil The Brazil Health System 18142 6/30/1998 SAS India Tamil Nadu Integrated Nutrition Project 13783 12/12/1994
Resettlement
AFR Senegal First and Second Terres Neuves Resettlement Projects 5170 6/1/1984 EAP Malaysia First, Second, and Third Jengka Triangle Land Settlement
Projects 5988 12/1/1985
SAS India Early Experience with Involuntary Resettlement: Impact Evaluation Report on India Second Maharashtra Irrigation Project
12133 6/29/1993
AFR; EAP; SAS
Ghana, Thailand, India
Early Experience with Involuntary Resettlement: Overview 12142 6/30/1993
Transport
LAC Brazil Secondary and Feeder Roads Project and the Second Feeder Roads Project
16738 6/13/1997
MNA Morocco Socioeconomic Influence of Rural Roads: Fourth Highway Project
15808 6/28/1996
Annex 1 13
Sector/ Region
Country Title Report # Date
Urban
EAP Indonesia Enhancing the Quality of Life in Urban Indonesia: the Legacy of Kampung Improvement Program.
14747 6/29/1995
LAC Brazil Learning from Best Practice in Five Urban Projects: Medium-Sized Cities, the Recife Metropolitan Region, Preparations of Metropolitan Development Programs for Fortaleza and Salvador, the Parana Market Towns Improvement and Northeast Urban Flood Reconstruction Projects
16736 6/17/1997
Water and Sanitation
AFR Kenya Development of Housing, Water Supply and Sanitation in Nairobi
15586 4/25/1996
LAC Paraguay Community Based Rural Water Systems and the Development of Village Committees
17923 5/29/1998
SAS India Water Supply and Waste Water Services in Bombay: First, Second and Third Bombay Water Supply and Sewerage Projects
15849 6/28/1996
SAS Pakistan SCARP Transition Pilot Project 16840 6/30/1997 SAS Sri Lanka Community Water Supply and Sanitation Project 18113 6/30/1998 SAS India Comparative Review of Rural Water Systems Experience:
The Rajasthan Water Supply and Sewerage Project, and the Rural Water Supply and Environmental Sanitation Projects for Maharashtra and Karnataka
18114 6/30/1998
Other
AFR Sub-Saharan Africa Higher Impact Adjustment Lending (HIAL) in Sub-Saharan Africa
19797 6/29/1999
EAP Indonesia First, Second and Third Transmigration Projects 12874 3/22/1994 EAP Korea Delayed Development of the Cholla Region: An
Institutional Study 16211 12/31/1996
LAC Ecuador World Bank Support for Small Scale Enterprises in Ecuador - An Impact Evaluation
17953 6/3/1998
LAC Brazil and the Philippines
Building Institutions and Financing Local Development: Lessons from Brazil and the Philippines
18727 12/18/1998
SAS Pakistan First, Second and Third Income Generating Projects for Refugee Areas
15862 6/28/1996
Global The World Bank and Microenterprise Finance: From Concept to Practice
19895 11/15/1999
Ann
ex 2
. Key
Fea
ture
s of t
he M
etho
dolo
gies
Use
d in
Impa
ct E
valu
atio
n R
epor
ts w
ith a
Cou
nter
fact
ual
Qua
ntita
tive
Ana
lysi
s Fe
atur
es
Cou
nter
fact
ual
Title
and
Rep
ort
Num
ber
Cou
ntry
D
atab
ase
Type
U
nit o
f an
alys
is
Sam
ple
Size
Sa
mpl
ing
Con
trol
Com
paris
onO
ther
Unc
lear
Estim
atio
n St
rate
gy
Qua
litat
ive
Ana
lysi
s A
ttem
pted
Agric
ultu
re
Pane
l dat
a R
ural
ban
ks47
trea
tmen
t & a
ll ru
ral b
anks
(23%
of
all t
reat
men
t ban
ks
Ran
dom
Ph
ilippi
nes:
Sec
ond
Rur
al C
redi
t Pro
ject
(4
557)
Ph
ilippi
nes
Post
-inte
rven
tion
surv
ey w
ith
retro
spec
tive
data
Fa
rmer
s 73
8 tre
atm
ent
farm
ers
(9%
of a
ll tre
atm
ent f
arm
ers)
R
ando
m
Mea
n va
lue
com
paris
on
Yes
Turk
ey: S
eyha
n irr
igat
ion
proj
ect (
stag
e II)
(574
5)
Turk
ey
Post
-inte
rven
tion
surv
ey w
ith
retro
spec
tive
data
Farm
H
ouse
hold
129
treat
men
t & 2
1 co
ntro
l far
m
hous
ehol
ds (3
% o
f al
l tre
atm
ent
hous
ehol
ds)
Ran
dom
M
ean
valu
e co
mpa
rison
Ye
s
Bang
lade
sh: S
hallo
w
Tube
wel
ls P
roje
ct
(110
31)
Bang
lade
sh
Post
-inte
rven
tion
surv
ey; 1
989
seco
ndar
y ba
selin
e da
ta
Farm
ers
150
STW
ow
ner/
man
ager
s fro
m 2
an
d 3
Upa
zila
from
2
zone
s (o
f 4) &
50
wat
er-p
urch
aser
s &
60 n
on-ir
rigat
ors.
36 fr
om 1
989
surv
ey &
114
ra
ndom
ly
sele
cted
.
Mea
n va
lue
com
paris
on
Yes
Indi
a: E
arly
Exp
erie
nce
with
Invo
lunt
ary
Res
ettle
men
t (12
132)
In
dia
Post
-inte
rven
tion
surv
ey
Hou
seho
ld
210
hous
ehol
ds
(28%
sam
ple)
from
6
subm
erge
d vi
llage
s (1
7%
sam
ple)
, 120
ho
useh
olds
from
6
parti
ally
sub
mer
ged
(11%
sam
ple)
and
110
hous
ehol
ds
from
3 c
ontro
l vi
llage
s.
Ran
dom
M
ean
valu
e co
mpa
rison
Ye
s
Indi
a: W
est B
enga
l Ag
ricul
tura
l D
evel
opm
ent P
roje
ct
(121
40)
Indi
a Po
st-in
terv
entio
n su
rvey
Fa
rmer
s
190
STW
farm
ers
(less
than
1%
of t
he
tota
l) fro
m 3
-5
deve
lopm
ent b
lock
s w
ithin
4 (o
f 6)
dist
ricts
.
Prox
imity
&
acce
ss to
di
stric
t he
adqu
arte
rs
Mea
n va
lue
com
paris
on
Yes
14 Annex 2
Qua
ntita
tive
Ana
lysi
s Fe
atur
es
Cou
nter
fact
ual
Title
and
Rep
ort
Num
ber
Cou
ntry
D
atab
ase
Type
U
nit o
f an
alys
is
Sam
ple
Size
Sa
mpl
ing
Con
trol
Com
paris
onO
ther
Unc
lear
Estim
atio
n St
rate
gy
Qua
litat
ive
Ana
lysi
s A
ttem
pted
Wor
ld B
ank
Agric
ultu
ral
Exte
nsio
n Pr
ojec
ts in
Ke
nya
(195
23)
Keny
a
Post
-inte
rven
tion
surv
ey; 1
982
seco
ndar
y ba
selin
e da
ta
Hou
seho
ld
562
treat
men
t ho
useh
olds
(les
s th
an 1
% o
f all
treat
men
t ho
useh
olds
)
Pane
l cr
eate
d
CVM
; R
espo
nse
rate
; M
ultiv
aria
te
anal
ysis
Yes
Indu
stry
Wor
ld B
ank
Supp
ort f
or
smal
l & m
ediu
m
indu
strie
s (S
MI)
in S
ri La
nka:
An
Impa
ct
Eval
uatio
n (1
6790
)
Sri L
anka
Post
-inte
rven
tion
surv
ey; b
asel
ine
data
fro
m 1
985
& 19
92
reco
rds
Firm
s
149
treat
men
t (SM
I II
& III
) & 1
55
cont
rol f
irms
(3%
of
all t
reat
men
t firm
s)
Not
cle
arly
sp
ecifi
ed
Mea
n va
lue
com
paris
on;
Res
pons
e ra
te a
naly
sis
Yes
Firs
t, Se
cond
, Thi
rd a
nd
Four
th S
mal
l & M
ediu
m
Indu
strie
s D
evel
opm
ent
Proj
ects
in P
hilip
pine
s (1
8041
)
Philip
pine
s
Se
cond
ary
pane
l dat
aFi
rms
Mor
e th
an 3
00
firm
s, h
alf o
f whi
ch
are
cont
rol
SMIL
ES-
NSO
dat
aset
M
ean
valu
e co
mpa
rison
Ye
s
Educ
atio
n
Col
ombi
a: F
irst a
nd
Seco
nd E
duca
tion
Proj
ects
(306
2)
Col
ombi
a Po
st-in
terv
entio
n su
rvey
; ret
rosp
ectiv
e da
ta
Scho
ol
offic
ials
19 tr
eatm
ent &
10
cont
rol s
choo
l of
ficia
ls
All
Mea
n va
lue
com
paris
on
Scho
ol
offic
ials
13 tr
eatm
ent &
11
cont
rol s
choo
l of
ficia
ls
Ran
dom
-
Mea
n va
lue
& re
gres
sion
an
alys
is
Thai
land
: Firs
t Ed
ucat
ion
Proj
ect
(506
9)
Thai
land
Po
st-in
terv
entio
n su
rvey
; ret
rosp
ectiv
e da
ta
Stud
ents
20
2 st
uden
tsN
ot c
lear
ly
spec
ified
Res
pons
e ra
te
Yes
Wat
er a
nd S
anita
tion
Keny
a: D
evel
opm
ent o
f H
ousi
ng, W
ater
sup
ply
and
sani
tatio
n in
N
airo
bi (1
5586
)
Keny
a Po
st-in
terv
entio
n su
rvey
; ret
rosp
ectiv
e da
ta
Indi
vidu
al
100
(1-2
%)
indi
vidu
als
each
fro
m 3
pla
nned
site
s (fu
nded
mai
n w
ater
su
pply
line
) & 2
un
plan
ned
site
s (fu
nded
kio
sks)
Ran
dom
se
lect
ion
(exc
eptio
n:
30 o
wne
r in
terv
iew
s in
on
e pl
anne
d si
te)
Mea
n va
lue
com
paris
on
Yes
Paki
stan
: SC
ARP
Tran
sitio
n Pi
lot P
roje
ct
(168
40)
Paki
stan
Po
st-in
terv
entio
n su
rvey
; bas
elin
e da
ta
from
199
0 da
tase
t Fa
rmer
s
391
treat
men
t (3%
of
all
treat
men
t fa
rmer
s) &
100
co
ntro
l are
a fa
rmer
s
Pane
l cr
eate
d
M
ean
valu
e co
mpa
rison
Ye
s
15 Annex 2
Qua
ntita
tive
Ana
lysi
s Fe
atur
es
Cou
nter
fact
ual
Title
and
Rep
ort
Num
ber
Cou
ntry
D
atab
ase
Type
U
nit o
f an
alys
is
Sam
ple
Size
Sa
mpl
ing
Con
trol
Com
paris
onO
ther
Unc
lear
Estim
atio
n St
rate
gy
Qua
litat
ive
Ana
lysi
s A
ttem
pted
Seco
ndar
y pa
nel d
ata
(198
7 &
1996
) C
hild
ren
5 tre
atm
ent (
2% o
f th
e to
tal)
& 5
cont
rol
villa
ges
Pane
l cr
eate
d
Pa
ragu
ay: C
omm
unity
-ba
sed
rura
l wat
er
syst
ems
and
the
deve
lopm
ent o
f villa
ge
com
mitt
ees
(179
23)
Para
guay
Se
cond
ary
surv
ey d
ata;
re
trosp
ectiv
e in
form
atio
n
Indi
vidu
al
17,6
44 (6
%)
indi
vidu
als
from
R
WS
1 &
II an
d 10
1,78
7 (3
0%)
indi
vidu
als
to b
e in
R
WS
IV
Ran
dom
Mea
n va
lue
com
paris
on
Yes
Urb
an
Braz
il: L
earn
ing
from
Be
st P
ract
ice
in F
ive
Urb
an P
roje
cts
(167
36)
Braz
il Po
st-in
terv
entio
n su
rvey
; ret
rosp
ectiv
e da
ta
Indi
vidu
al
Mee
tings
with
8-1
6 in
divi
dual
s(le
ss th
an
1%) f
rom
(2%
) tre
atm
ent &
con
trol
grou
ps fr
om 1
1 m
unic
ipal
ities
Parti
cipa
tory
-
Mea
n va
lue
com
paris
on
Yes
Indo
nesi
a: E
nhan
cing
th
e Q
ualit
y of
Life
in
Urb
an In
done
sia:
the
Lega
cy o
f Kam
pung
Im
prov
emen
t Pro
gram
(1
4747
)
Indo
nesi
a
Post
-inte
rven
tion
surv
ey; b
asel
ine
data
fro
m 1
82 th
esis
by
Dr.
Tayl
or; r
etro
spec
tive
data
H
ouse
hold
25 (l
ess
than
1%
) ho
useh
olds
eac
h fro
m 5
trea
tmen
t (fr
om 3
of 1
1 ci
ties)
&
25 h
ouse
hold
s ea
ch fr
om 2
con
trol
(from
2 c
ities
) ka
mpu
ngs
Stra
tifie
d ra
ndom
Mea
n va
lue
com
paris
on
usin
g t-
stat
istic
s; C
hi-
squa
re
stat
istic
s
Res
iden
ts
12 re
side
nts(
less
th
an 1
% o
f the
tota
l) in
eac
h of
the
two
site
s &
serv
ice
area
s
Ran
dom
M
ean
valu
e co
mpa
rison
Yes
Tran
spor
t
King
dom
of M
oroc
co:
Soci
oeco
nom
ic
influ
ence
of r
ural
road
s (1
5808
)
Mor
occo
Po
st-in
terv
entio
n su
rvey
; ret
rosp
ectiv
e da
ta
Farm
ers
29-5
0 fa
rmer
s in
ea
ch o
f the
4 (o
f 10)
tre
atm
ent r
oad
villa
ges;
& 5
-13
farm
ers
in e
ach
of
the
4 co
ntro
l roa
d vi
llage
Stra
tifie
d ra
ndom
Diff
eren
ce-in
-di
ffere
nce;
di
ffere
nce-
in-
mea
ns
Yes
16 Annex 2
Qua
ntita
tive
Ana
lysi
s Fe
atur
es
Cou
nter
fact
ual
Title
and
Rep
ort
Num
ber
Cou
ntry
D
atab
ase
Type
U
nit o
f an
alys
is
Sam
ple
Size
Sa
mpl
ing
Con
trol
Com
paris
onO
ther
Unc
lear
Estim
atio
n St
rate
gy
Qua
litat
ive
Ana
lysi
s A
ttem
pted
Hea
lth, N
utrit
ion
& Po
pula
tion
Indi
a: T
amil
Nad
u In
tegr
ated
Nut
ritio
n Pr
ojec
t (13
783)
In
dia
Seco
ndar
y da
ta
colla
ted
by T
INP
Mon
itorin
g of
fice.
C
hild
ren
1600
; 160
9 an
d 12
91 (2
%) c
hild
ren
enro
lled
in p
hase
I,
II &
III, r
espe
ctiv
ely
of T
INP-
I
Stra
tifie
d ra
ndom
Ye
s
Oth
ers
Post
-inte
rven
tion
surv
ey; r
etro
spec
tive
data
Fi
rms
44 tr
eatm
ent (
over
10
%) &
36
cont
rol
firm
s
Stra
tifie
d ra
ndom
M
ean
valu
e co
mpa
rison
Ye
s Ko
rea:
Del
ayed
D
evel
opm
ent o
f the
C
holla
Reg
ion:
An
Inst
itutio
nal S
tudy
(1
6211
)
Kore
a
Seco
ndar
y pa
nel d
ata
villa
ges
villa
ges
near
2
treat
men
t brid
ges,
co
mpa
red
to
neig
hbor
ing
villa
ges
with
out b
ridge
s
All b
ridge
s
M
ean
valu
e co
mpa
rison
Wor
ld B
ank
Supp
ort f
or
Smal
l Sca
le E
nter
pris
es
in E
cuad
or (1
7953
) Ec
uado
r Po
st-in
terv
entio
n su
rvey
; bas
elin
e da
ta
from
ann
ual r
ecor
ds
Firm
s 14
5 tre
atm
ent (
1%
of th
e to
tal)
& 55
co
ntro
l firm
s R
ando
m
Mea
n va
lue
com
paris
on
Yes
Braz
il
448
treat
men
t (73
%
of th
e to
tal)
& 18
5 co
ntro
l m
unic
ipal
ities
Stra
tifie
d ra
ndom
Annu
al m
unic
ipal
ity
data
M
unic
ipa-
lity 2
1 tre
atm
ent (
68%
of
the
tota
l) &
15
cont
rol
mun
icip
aliti
es
Stra
tifie
d ra
ndom
Mea
n va
lue
com
paris
on
Build
ing
Inst
itutio
ns a
nd
Fina
ncin
g Lo
cal
Dev
elop
men
t: Le
sson
s fro
m B
razi
l and
the
Philip
pine
s (1
8727
) Ph
ilippi
nes
Post
-inte
rven
tion
surv
ey; r
etro
spec
tive
data
Stal
l hol
ders
&
owne
rs
60 s
tall
hold
ers
(20%
of t
he to
tal)
& 15
sho
p ow
ners
(3
8% o
f the
tota
l) fo
r ea
ch th
e tre
atm
ent
& th
e co
ntro
l gro
up
Ran
dom
Mea
n va
lue
com
paris
on
with
t-st
atis
tics
Yes
Hig
her I
mpa
ct
Adju
stm
ent L
endi
ng
(HIA
L) In
itial
Eva
luat
ion
(197
97)
17 c
ount
ries
in A
frica
Se
cond
ary
pane
l dat
a C
ount
ry
13 (o
f 17)
HIA
L &
35 N
on-H
IAL
SSA
& 24
Non
-SSA
co
untri
es
Dat
a av
aila
bilit
y
Reg
ress
ion
anal
ysis
; D
iffer
ence
-in-
diffe
renc
e
17 Annex 2
Annex 3 18
Annex 3. Historical Cost Data for OED Impact Evaluations
Total Costs (US$ 000)
Evaluation Report Title and Number FY CPI Current Constant 1995 dollars
World Bank Agricultural Extension Projects in Kenya (19523) 1999 109 267.10 252 Higher Impact Adjustment Lending (HIAL) Initial Evaluation (19797) 1999 109 92.70 88 World Bank Support for Small Scale Enterprises in Ecuador (17953) 1998 107 186.10 179 First, Second, Third and Fourth Small & Medium Industries Development Projects in Philippines (18041) 1998 107 149.10 144
Paraguay: Community-based rural water systems and the development of village committees (17923) 1998 107 37.70 36
World Bank Support for small & medium industries (SMI) in Sri Lanka: An Impact Evaluation (16790) 1997 105 115.90 114
Brazil: Learning from Best Practice in Five Urban Projects (16736) 1997 105 100.70 99 Pakistan: SCARP Transition Pilot Project (16840) 1997 105 41.40 41 Kenya: Development of Housing, Water supply and sanitation in Nairobi (15586) 1996 103 144.00 144
Kingdom of Morocco: Socioeconomic influence of rural roads (15808) 1996 103 72.40 72
Annex 4 19
Annex 4. Results of Survey of OED Impact Evaluation Task Managers Comparison between the actual and required extent of collaboration between: Bank Regional staff and the OED
team conducting the impact evaluation
Government officials and the OED team conducting the impact evaluation
Actual less than required 2 2 Actual same as required 6 7 Actual more than required 1 0
Use of local public or private research institutions: Used Data collection Analysis Report writing Discussions Yes 6 6 3 2 2
No 3 3 6 7 7 Influence of impact evaluation on World Bank’s or country’s policies & programs Design of next Bank
project Bank Policy in the sector
Government policy in the sector
Policies of the international community
Yes 2 6 5 2
Maybe/Small 0 2 1 2 No 7 1 2 3 Don’t know 0 0 1 2
Overall, rating of the impact evaluation on: Bank’s policies & programs Government’s policies & programs Substantial 3 1
Moderate 2 2 Marginal 4 4 Only to extent staff interviewed 0 1
Don’t know 0 1 Level of satisfaction with: Adequacy of resources
& time Effort to disseminate findings Follow up actions
Satisfied 6 2 2 Unsatisfied 2 1 2 Mixed 0 3 2
Indifferent 0 1 2 Hostility 0 2 0 Don’t know 1 0 1
Recommendations if the impact evaluation were conducted for the same project again Insignificant
changes More Time More Resources Better Data More local
collaborations Yes 2 1 1 2 1 No 7 8 8 7 8
Lessons or advice about conducting impact evaluation Adequate
Time Adequate Resources
Good Data Use of local collaborators Rigorous and thorough analysis
Yes 0 0 1 0 3
No 9 9 8 9 6
Ann
ex 5
. Pro
ject
and
Eva
luat
ion
Obj
ectiv
es fo
r O
ED
Impa
ct E
valu
atio
ns w
ith a
Cou
nter
fact
ual
Title
& R
epor
t N
umbe
r N
ame
of P
roje
ct
eval
uate
d #
Obj
ectiv
es o
f the
Pro
ject
#
Obj
ectiv
es o
f the
Impa
ct E
valu
atio
n
1 To
incr
ease
inst
itutio
nal c
apac
ity o
f mun
icip
aliti
es &
sta
te u
rban
de
velo
pmen
t age
ncie
s to
pla
n, fi
nanc
e, &
exe
cute
inve
stm
ent
prog
ram
s.
1
2 To
impr
ove
the
fisca
l & fi
nanc
ial m
anag
emen
t cap
acity
of
mun
icip
aliti
es.
To a
sses
s im
pact
s of
mun
icip
al d
evel
opm
ent p
roje
cts
(MD
Ps) o
n th
e in
stitu
tiona
l cap
acity
bui
ldin
g of
loca
l gov
ernm
ents
for f
isca
l an
d fin
anci
al m
anag
emen
t and
for p
lann
ing
and
impl
emen
tatio
n of
inve
stm
ent p
rogr
ams.
3 To
pro
vide
bas
ic e
cono
mic
& s
ocia
l inf
rast
ruct
ure
in u
rban
are
as.
2
Mun
icip
al
Dev
elop
men
t pr
ojec
t in
the
stat
e of
Par
ana
& R
io
Gra
nde
do S
ul
4 To
impr
ove
targ
etin
g of
urb
an p
rogr
ams
to lo
wer
-inco
me
popu
latio
ns.
To
ass
ess
both
dire
ct im
pact
s on
the
bene
ficia
ries
as a
ntic
ipat
ed
by th
e pr
ojec
ts, a
nd in
dire
ct im
pact
s on
the
deve
lopm
ent o
f loc
al
econ
omie
s fo
cusi
ng o
n em
ploy
men
t and
inco
me
gene
ratio
n in
th
e pa
rtici
patin
g m
unic
ipal
ities
.
1 To
est
ablis
h an
inst
itutio
nal m
echa
nism
, the
Mun
icip
al
Dev
elop
men
t Fun
d (M
DF)
, to
prov
ide
loca
l gov
t. w
ith d
irect
ac
cess
to lo
ng-te
rm d
evel
opm
ent f
inan
ce.
2 To
est
ablis
h a
natio
nal-l
evel
tech
nica
l int
erm
edia
ry, t
he c
entra
l pr
ojec
t offi
ce.
3 To
stre
ngth
en lo
cal t
echn
ical
& fi
nanc
ial c
apac
ity fo
r pro
ject
im
plem
enta
tion
& se
rvic
e m
anag
emen
t thr
ough
a tr
aini
ng
prog
ram
.
Build
ing
Inst
itutio
ns
and
Fina
ncin
g Lo
cal
Dev
elop
men
t: Le
sson
s fro
m B
razi
l an
d th
e Ph
ilippi
nes
(187
27)
Mun
icip
al
Dev
elop
men
t pr
ojec
t in
Philip
pine
s &
Seco
nd M
unic
ipal
D
evel
opm
ent
proj
ect i
n Ph
ilippi
nes
4 To
impr
ove
loca
l fis
cal p
erfo
rman
ce th
roug
h R
eal P
rope
rty T
ax
Adm
inis
tratio
n (R
PTA)
pro
gram
.
1 ID
of t
he e
xten
sion
ser
vice
by
adop
tion
of T
&V.
1 Em
piric
al a
sses
smen
t of t
he im
pact
of t
he N
EP-1
and
NEP
-II
usin
g th
eory
-bas
ed e
valu
atio
n ap
proa
ch.
Nat
iona
l ext
ensi
on
proj
ect,
NEP
-I 2
Sust
aine
d in
crea
ses
in a
gric
ultu
ral p
rodu
ctiv
ity in
30
of 4
1 di
stric
ts, c
over
ing
all m
ediu
m/h
igh
arab
le a
reas
2
1 St
imul
ate
dev
& ad
optio
n of
tech
pac
kage
s th
at e
nabl
e sm
allh
olde
rs to
incr
ease
pro
duct
ivity
& in
com
e
Wor
ld B
ank
Agric
ultu
ral
Exte
nsio
n Pr
ojec
ts
in K
enya
(195
23)
Seco
nd N
atio
nal
exte
nsio
n pr
ojec
t, N
EP-II
2
Con
solid
ate,
impr
ove
& fo
rtify
gai
ns u
nder
NEP
-1
Dra
w a
ppro
pria
te p
olic
y co
nclu
sion
s, m
aint
aini
ng c
lear
dis
tinct
ion
betw
een
impa
ct o
f a p
artic
ular
sys
tem
and
impa
ct o
f ext
ensi
on.
With
/with
out d
iffic
ult s
ince
T&V
sys
tem
intro
duce
d on
a n
atio
nal
scal
e; B
efor
e/af
ter l
imite
d si
nce
syst
em w
as in
trodu
ced
rapi
dly,
15
yea
rs a
go.
SMI I
1
Soug
ht to
eng
age
the
Gov
ernm
ent (
GO
SL) i
n a
dial
ogue
abo
ut
effic
acy
of e
cono
mic
pol
icie
s re
gard
ing
trade
, app
ropr
iate
role
of
publ
ic a
nd p
rivat
e se
ctor
s.
1 Im
pact
of c
redi
t and
non
lend
ing
serv
ices
on
smal
l and
med
ium
in
dust
ries
(SM
Is)
SMI I
I
SMI I
II
2 So
ught
t w
ork
with
GO
SL to
rest
ruct
ure
finan
cial
sec
tor f
rom
one
th
at s
ervi
ced
a ce
ntra
lly p
lann
ed in
dust
rial e
cono
my
into
one
that
re
spon
ds to
the
dem
and
of e
ntre
pren
eurs
.
2 Ef
fect
s of
ban
k le
ndin
g on
par
ticip
atin
g fin
anci
al in
stitu
tions
(P
FIs)
Wor
ld B
ank
Supp
ort
for s
mal
l & m
ediu
m
indu
strie
s (S
MI)
in
Sri L
anka
: An
Impa
ct E
valu
atio
n (1
6790
) SM
I IV
3 So
ught
to g
ener
ate
jobs
on
a co
st-e
ffect
ive
basi
s, a
ssum
ing
that
sm
all f
irms
wou
ld b
e m
ore
labo
r-int
ensi
ve.
20 Annex 5
Title
& R
epor
t N
umbe
r N
ame
of P
roje
ct
eval
uate
d #
Obj
ectiv
es o
f the
Pro
ject
#
Obj
ectiv
es o
f the
Impa
ct E
valu
atio
n
1
Exam
ine
the
deve
lopm
ent o
f the
pro
ject
sch
ools
ove
r the
pas
t 4-
year
per
iod.
Fi
rst e
duca
tion
proj
ect
Prov
ide
10 s
econ
dary
com
preh
ensi
ve o
r div
ersi
fied
scho
ols
(INEM
s) th
at w
ould
pre
pare
stu
dent
s fo
r fur
ther
st
udie
s/em
ploy
men
t. Ea
ch IN
EMs
was
exp
ecte
d to
offe
r gu
idan
ce c
ouns
elin
g, li
brar
y an
d he
alth
ser
vice
s, a
nd h
ave
larg
e ad
min
istra
tive
staf
f, &
train
ed te
ache
rs.
2 C
olom
bia:
Firs
t and
Se
cond
Edu
catio
n Pr
ojec
ts (3
062)
Se
cond
edu
catio
n pr
ojec
t
Prov
ide
9 ad
ditio
nal s
econ
dary
INEM
s
Exam
ine
enro
llmen
t gro
wth
, int
erna
l effi
cien
cy, a
bsor
ptio
n of
gr
adua
tes,
teac
hing
/lear
ning
env
ironm
ent,
spill-
over
effe
cts
& pu
blic
ass
essm
ent o
f the
pro
ject
sch
ools
.
Wor
ld B
ank
Supp
ort
for S
mal
l Sca
le
Ente
rpris
es in
Ec
uado
r (17
953)
SSE
I – IV
Sm
all S
cale
Ent
erpr
ise
(SSE
) pro
ject
aim
ed to
incr
ease
su
bsta
ntia
lly th
e av
aila
bilit
y of
term
fina
ncin
g fo
r mic
ro a
nd s
mal
l fir
ms,
whi
le im
prov
ing
the
bank
ing
syst
em’s
abi
lity
to le
nd to
this
se
ctor
.
Ex
amin
e th
e ef
fect
s of
the
loan
s an
d of
oth
er B
ank
inte
rven
tions
, bo
th le
ndin
g an
d no
n-le
ndin
g on
SSE
s an
d pa
rtici
patin
g fin
anci
al
inst
itutio
ns (P
FIs)
.
Seco
ndar
y ci
ties
regi
onal
pro
ject
1
Supp
ort t
he g
over
nmen
t’s e
fforts
to d
evel
op th
e C
holla
regi
on &
re
duce
inte
r-reg
iona
l dis
parit
ies.
1
“Dire
ct” p
roje
ct im
pact
s on
the
bene
ficia
ries
as a
ntic
ipat
ed
Seco
nd G
wan
gju
regi
onal
pro
ject
2
Long
-term
“sid
e-ef
fect
s” o
n th
e in
dust
rializ
atio
n pr
oces
s an
d in
stitu
tiona
l lea
rnin
g.
Kore
a: D
elay
ed
Dev
elop
men
t of t
he
Cho
lla R
egio
n: A
n In
stitu
tiona
l Stu
dy
(162
11)
Jeon
ju re
gion
al
deve
lopm
ent
proj
ect
2 In
dust
rial d
evel
opm
ent t
hrou
gh in
vest
men
t in
indu
stria
l est
ates
an
d in
crea
sing
em
ploy
men
t & in
com
e of
the
isla
nd p
opul
atio
n.
3 Im
pact
of b
ridge
s on
isla
nd e
cono
my.
Firs
t, Se
cond
, Thi
rd
and
Four
th S
mal
l &
Med
ium
Indu
strie
s D
evel
opm
ent
Proj
ects
in
Philip
pine
s (1
8041
)
SMI I
– IV
Fi
nanc
ing
urba
n-ba
sed
smal
l ind
ustry
adm
inis
tere
d by
Indu
stria
l G
uara
ntee
Loa
n Fu
nd o
r IG
LF.
Ev
alua
te im
pact
of t
he B
ank’
s st
rate
gy o
n SM
Is.
1 D
ecom
mis
sion
ing
213
publ
ic S
CAR
P w
ells
To s
tudy
the
impa
ct o
f pro
ject
in P
akis
tan?
? Pa
kist
an: S
CAR
P Tr
ansi
tion
Pilo
t Pr
ojec
t (16
840)
SCAR
P Tr
ansi
tion
Pilo
t Pro
ject
2
Rep
laci
ng S
CAR
P w
ells
with
210
0 pr
ivat
e w
ells
, pro
vidi
ng
elec
trica
l sup
plie
s fo
r priv
ate
wel
l ope
ratio
n an
d im
prov
emen
ts to
m
inor
irrig
atio
n ca
nals
.
Thai
land
: Firs
t Ed
ucat
ion
Proj
ect
(506
9)
Thai
land
Firs
t Ed
ucat
ion
Proj
ect
In
crea
sing
out
put &
impr
ovin
g qu
ality
of m
iddl
e le
vel s
kille
d w
orke
rs &
tech
nici
ans
requ
ired
in th
e in
dust
rial &
agr
icul
tura
l se
ctor
s by
pro
vidi
ng e
quip
men
t. In
frast
ruct
ure
was
pro
vide
d by
N
atio
nal G
ovt.
& te
ch a
ssis
tanc
e by
USA
ID.
To
stu
dy th
e im
pact
of t
he fi
rst v
ocat
iona
l edu
catio
n pr
ojec
t
21 Annex 5
Title
& R
epor
t N
umbe
r N
ame
of P
roje
ct
eval
uate
d #
Obj
ectiv
es o
f the
Pro
ject
#
Obj
ectiv
es o
f the
Impa
ct E
valu
atio
n
Med
ium
-siz
ed
citie
s
Impr
ove
urba
n in
frast
ruct
ure
and
serv
ices
; stre
ngth
en fe
dera
l ag
enci
es re
spon
sibl
e fo
r urb
an p
olic
y.
1 To
lear
n ho
w b
est p
ract
ice
mun
icip
aliti
es a
ddre
ss &
reco
ncile
ef
ficie
ncy
& po
verty
redu
ctio
n eq
uity
goa
ls in
urb
an s
ervi
ce
deliv
ery.
Th
e R
ecife
m
etro
polit
an
regi
on
Im
prov
e ho
usin
g &
urba
n se
rvic
es: I
nduc
e de
sire
d pa
ttern
s of
m
etro
dev
elop
men
t; in
com
e ge
nera
tion
of th
e po
or; i
nstit
utio
nal
deve
lopm
ent.
2
To id
entif
y &
asse
ss: s
usta
inab
le d
irect
& in
dire
ct p
roje
ct im
pact
s up
on li
ving
con
ditio
ns o
f the
urb
an p
oor;
spec
ific
proj
ect a
ctio
ns
asso
ciat
ed w
ith B
razi
lian
mun
icip
aliti
es th
at s
ucce
eded
in
impr
ovin
g th
ese
cond
ition
s ef
ficie
ntly
; inn
ovat
ions
intro
duce
d by
th
e pr
ojec
ts th
at c
an b
e re
plic
ated
els
ewhe
re.
Prep
arat
ions
of
met
ropo
litan
de
velo
pmen
t pr
ogra
ms
for
Forta
leza
&
Salv
ador
Pr
epar
e de
velo
pmen
t pro
gram
s fo
r For
tale
za &
Sal
vado
r; st
reng
then
pla
nnin
g se
rvic
es.
The
Para
na
mar
ket t
owns
im
prov
emen
t
Lo
ng-te
rm b
orro
win
g fo
r mun
icip
aliti
es; s
yste
mat
ic s
elec
tion
of
inve
stm
ent c
osts
; bet
ter l
ivin
g st
anda
rds
thro
ugh
infra
stru
ctur
e.
3 To
dra
w le
sson
s ab
out p
roje
ct d
esig
n &
impl
emen
tatio
n fe
atur
es
that
mos
t hel
p im
prov
e th
e liv
ing
cond
ition
s of
the
urba
n po
or.
Braz
il: L
earn
ing
from
Bes
t Pra
ctic
e in
Fiv
e U
rban
Pr
ojec
ts (1
6736
)
Nor
thea
st u
rban
flo
od
reco
nstru
ctio
n pr
ojec
ts
R
ehab
ilitat
e flo
od-d
amag
ed m
unic
ipal
ities
; stre
ngth
en p
lann
ing
for f
lood
pre
vent
ion.
4
To p
rovi
de p
ract
ical
& fo
cuse
d re
com
men
datio
ns fo
r Bra
zilia
n &
Bank
pol
icy
mak
ers
on w
ays
of b
ringi
ng s
usta
inab
le b
enef
its to
th
e po
or e
ffici
ently
.
Jaka
rta U
rban
D
evel
opm
ent
proj
ect
To
est
ablis
h a
natio
nal u
rban
dev
elop
men
t pro
gram
that
wou
ld
rais
e th
e liv
ing
cond
ition
s of
the
urba
n po
or b
y im
prov
ing
thei
r ac
cess
to b
ette
r phy
sica
l inf
rast
ruct
ure
& ho
usin
g.
Seco
nd u
rban
de
velo
pmen
t pr
ojec
t
To
initi
ate
the
KIP
for i
mpr
ovin
g th
e ki
ving
con
ditio
ns o
f the
urb
an
poor
out
side
Jak
arta
(Sur
abay
a).
Third
urb
an
deve
lopm
ent
proj
ect
To
exp
and
the
exis
ting
KIP
in J
akar
ta &
Sur
abay
a &
to e
xten
d it
to 4
oth
er s
econ
dary
citi
es &
to b
road
en th
e KI
P, in
clud
ing
inve
stm
ents
for i
mpr
ovin
g ge
nera
l pub
lic h
ealth
(e.g
. sol
id-w
aste
m
anag
emen
t).
Indo
nesi
a:
Enha
ncin
g th
e Q
ualit
y of
Life
in
Urb
an In
done
sia:
th
e Le
gacy
of
Kam
pung
Im
prov
emen
t Pr
ogra
m (1
4747
) Fo
urth
urb
an
deve
lopm
ent
proj
ect
To
impl
emen
t a n
atio
nwid
e KI
P; a
site
s &
serv
ices
pro
gram
; and
a
stra
tegy
for s
treng
then
ing
the
man
agem
ent c
apab
ilitie
s of
the
Bank
Tab
unga
n N
egar
a (B
TN) &
Nat
iona
l Urb
an D
evel
opm
ent
Cor
pora
tion
(PER
UM
NAS
).
To
und
erst
and
the
med
ium
-to lo
ng-te
rm im
pact
s (5
to 1
0 ye
ars
afte
r com
plet
ion)
of e
ight
een
year
s of
Ban
k le
ndin
g fo
r urb
an
deve
lopm
ent.
1 To
pro
vide
med
ium
and
long
term
cre
dit t
o ab
out 8
680
farm
ers,
fis
herm
en a
nd ru
ral e
ntre
pren
eurs
, thr
ough
abo
ut 2
00 ru
ral
bank
s.
To
eva
luat
e th
e se
cond
rura
l cre
dit p
roje
ct.
Philip
pine
s: S
econ
d R
ural
Cre
dit P
roje
ct
(455
7)
Philip
pine
s se
cond
rura
l cre
dit
proj
ect
2 To
fina
nce
tract
ors,
pow
er ti
llers
, irri
gatio
n eq
uipm
ent,
on-fa
rm
stor
age
and
proc
essi
ng fa
cilit
ies,
hou
sing
, equ
ipm
ent &
pr
oduc
tion
stoc
k fo
r liv
esto
ck d
evel
opm
ent &
sm
all f
ishi
ng b
oats
&
equi
pmen
t.
22 Annex 5
Title
& R
epor
t N
umbe
r N
ame
of P
roje
ct
eval
uate
d #
Obj
ectiv
es o
f the
Pro
ject
#
Obj
ectiv
es o
f the
Impa
ct E
valu
atio
n
Turk
ey: S
eyha
n irr
igat
ion
proj
ect
(sta
ge II
) (IE
R57
45)
Turk
ey: S
eyha
n irr
igat
ion
proj
ect
(sta
ge II
)
To
incr
ease
agr
icul
tura
l pro
duct
ion
and
to e
xplo
it th
e fu
ll po
tent
ial
of th
e re
gion
To a
cces
s th
e m
ajor
effe
cts
of th
e Se
yhan
irrig
atio
n pr
ojec
t on
the
deve
lopm
ent o
f the
Ada
na p
lain
.
Firs
t Nai
robi
wat
er
supp
ly p
roje
ct
1 To
exp
and
& im
prov
e th
e w
ater
sys
tem
in N
airo
bi to
mee
t the
in
crea
sing
dem
and,
and
to s
treng
then
WSD
incl
udin
g its
fina
ncia
l op
erat
ions
.
Ac
cess
the
med
ium
& lo
ng-te
rm im
pact
s of
mor
e th
an 2
0 ye
ars
for t
hese
pro
ject
s.
Nai
robi
site
s &
serv
ices
pro
ject
2
To d
emon
stra
te fe
asib
ility
of p
rovi
ding
affo
rdab
le h
ousi
ng
serv
ices
to th
e ur
ban
poor
on
a la
rger
sca
le &
at l
ower
cos
ts th
an
prev
ious
gov
ernm
ent p
roje
cts,
and
to e
xpan
d tru
nk s
ewer
in
frast
ruct
ure
to m
eet t
he p
ress
ing
sani
tatio
n ne
eds
of th
e ci
ty,
and
to im
prov
e pr
ojec
t im
plem
enta
tion
capa
bilit
ies
of G
OK.
Seco
nd N
airo
bi
wat
er s
uppl
y pr
ojec
t
3 To
incr
ease
the
wat
er s
uppl
y ca
paci
ty in
Nai
robi
, im
prov
e &
expa
nd th
e di
strib
utio
n sy
stem
, & p
rovi
de tr
aini
ng &
ass
ista
nce
with
the
desi
gn &
impl
emen
tatio
n of
impr
ovem
ents
to W
SD’s
ac
coun
ting
& m
anag
emen
t sys
tem
s.
Seco
nd u
rban
pr
ojec
t 4
To e
nabl
e th
e G
over
nmen
t to
acce
lera
te im
plem
enta
tion
of lo
w-
inco
me
hous
ing
prog
ram
s in
Nai
robi
and
initi
ate
sim
ilar p
rogr
ams
in K
isum
u an
d M
omba
sa to
incl
ude
squa
tter u
pgra
ding
, low
er
stan
dard
s in
site
and
ser
vice
s pl
ots,
inst
itutio
nal s
treng
then
ing,
as
sist
ance
to fi
nanc
ial m
anag
emen
t of l
ocal
aut
horit
ies,
and
im
prov
emen
t of i
ncom
e ge
nera
ting
activ
ities
and
nut
ritio
n an
d fa
mily
pla
nnin
g se
rvic
es fo
r low
-inco
me
grou
ps.
Keny
a:
Dev
elop
men
t of
Hou
sing
, Wat
er
supp
ly a
nd
sani
tatio
n in
Nai
robi
(1
5586
)
Nai
robi
third
wat
er
supp
ly
engi
neer
ing
proj
ect
5 To
hel
p W
SD p
repa
re a
third
pha
se w
ater
sup
ply
inve
stm
ent
proj
ect t
o m
eet t
he ra
pidl
y ex
pand
ing
wat
er d
eman
ds o
f Nai
robi
C
ity u
p to
mid
-90s
, to
stre
ngth
en th
e op
erat
ions
and
fina
ncia
l m
anag
emen
t of W
SD to
incr
ease
its
effic
ienc
y an
d ca
paci
ty to
un
derta
ke th
e th
ird p
hase
, and
to p
repa
re re
com
men
datio
ns fo
r lo
ng-te
rm in
stitu
tiona
l dev
elop
men
t.
King
dom
of
Mor
occo
: So
cioe
cono
mic
in
fluen
ce o
f rur
al
road
s (1
5808
)
Mor
occo
Fou
rth
high
way
pro
ject
To
impr
ove
and
expa
nd th
e pr
ovin
cial
road
net
wor
k to
hel
p al
levi
ate
rura
l pov
erty
.
To u
nder
stan
d th
e im
pact
of r
ural
road
s on
tran
spor
t in
frast
ruct
ure
& se
rvic
es, a
s w
ell a
s on
the
regi
on’s
eco
nom
y an
d so
cial
wel
fare
.
23 Annex 5
Title
& R
epor
t N
umbe
r N
ame
of P
roje
ct
eval
uate
d #
Obj
ectiv
es o
f the
Pro
ject
#
Obj
ectiv
es o
f the
Impa
ct E
valu
atio
n
1 To
pro
vide
chl
orin
ated
wat
er to
rura
l com
mun
ities
.
Rur
al w
ater
su
pply
pro
ject
2
To p
rovi
de s
anita
ry u
nits
in ru
ral c
omm
uniti
es a
nd h
elp
cons
truct
sa
fe la
trine
s.
To
ass
ess
the
impa
cts
of th
e W
orld
Ban
k’s
assi
stan
ce p
rogr
am
on th
e pe
rform
ance
of P
arag
uay’
s ru
ral w
ater
sub
sect
or.
3 To
pro
mot
e be
tter h
ygie
ne h
abits
& p
ublic
hea
lth im
prov
emen
ts.
4 To
stre
ngth
en th
e m
anag
eria
l & o
pera
tiona
l cap
abilit
ies
of
SEN
ASA
5 To
enc
oura
ge c
omm
unity
par
ticip
atio
n in
the
proj
ect,
rais
e a
com
mun
ity c
ontri
butio
n to
initi
al in
vest
men
t, an
d fo
ster
loca
l co
mm
itmen
t to
mai
nten
ance
.
Para
guay
: C
omm
unity
-bas
ed
rura
l wat
er s
yste
ms
and
the
deve
lopm
ent o
f vi
llage
com
mitt
ees
(179
23)
Rur
al w
ater
su
pply
and
sa
nita
tion
proj
ect
II &
III
6 To
ens
ure
parti
al re
cove
ry o
f the
inve
stm
ent.
Hig
her I
mpa
ct
Adju
stm
ent L
endi
ng
(HIA
L) In
itial
Ev
alua
tion
(197
97)
21 o
pera
tions
in
17 c
ount
ries
in
Afric
a
To
impr
ove
the
resu
lts o
f adj
ustm
ent l
endi
ng (A
L) in
Sub
-Sa
hara
n Af
rica
(SSA
) thr
ough
the
appl
icat
ion
of g
reat
er c
ount
ry
sele
ctiv
ity a
nd im
prov
ed d
esig
n.
To
det
erm
ine
whe
ther
ther
e w
as a
n im
prov
emen
t in
adju
stm
ent
lend
ing
in S
ub-S
ahar
an A
frica
and
to a
sses
s th
e di
ffere
nce
HIA
L ap
proa
ch h
as m
ade.
1 To
stu
dy th
e im
pact
of t
he p
roje
ct o
n th
e nu
tritio
n &
heal
th s
tatu
s of
the
targ
et p
opul
atio
n, in
depe
nden
t of o
ther
fact
ors
that
hav
e in
fluen
ced
the
outc
ome.
2
To s
tudy
if s
ervi
ce d
eliv
ery
was
of s
uffic
ient
qua
lity
so a
s to
mak
e it
plau
sibl
e to
hav
e po
sitiv
e im
pact
. 3
To s
tudy
whi
ch p
rogr
am in
put c
reat
ed th
e im
pact
. 4
To e
xam
ine
the
set o
f ben
efic
iarie
s an
d pa
rtici
pant
s.
5 To
stu
dy th
e co
st-e
ffect
iven
ess
of th
e pr
ojec
t.
Indi
a: T
amil
Nad
u In
tegr
ated
Nut
ritio
n Pr
ojec
t (13
783)
Tam
il N
adu
Inte
grat
ed
Nut
ritio
n Pr
ojec
t
To
impr
ove
the
nutri
tiona
l and
hea
lth s
tatu
s of
pre
scho
ol c
hild
ren,
pr
imar
ily th
ose
6-36
mon
ths
old
and
preg
nant
and
nur
sing
m
othe
r.
6 To
exa
min
e si
gns
that
indi
cate
sus
tain
abilit
y of
the
resu
lts
achi
eved
so
far.
Indi
a: E
arly
Ex
perie
nce
with
In
volu
ntar
y R
eset
tlem
ent
(IER
1213
2)
Karn
atak
a Irr
igat
ion
Proj
ect
1 To
con
stru
ct tw
o da
m a
s pa
rt of
the
larg
er U
pper
Krs
hina
Irr
igat
ion
proj
ect (
thou
gh o
nly
one
dam
, at N
ayap
ur, w
as
com
plet
ed a
t tha
t tim
e).
1
To e
xam
ine
how
rese
ttlem
ent w
as im
plem
ente
d an
d to
det
erm
ine
the
effe
cts
of re
settl
emen
t on
the
lives
of t
he a
ffect
ed fa
milie
s.
2 To
relo
cate
and
rese
ttle
peop
le a
ffect
ed b
y th
e pr
ojec
t 2
To c
ompa
re th
e re
settl
emen
t exp
erie
nce
in d
iffer
ent c
ount
ries
and
unde
r var
ying
circ
umst
ance
s an
d re
late
this
exp
erie
nce
to
the
Bank
’s e
volv
ing
settl
emen
t pol
icie
s an
d gu
idel
ines
ove
r the
la
st te
n ye
ars.
3
To d
raw
less
ons
from
the
findi
ngs
and
conc
lusi
ons
for m
ore
effe
ctiv
e re
settl
emen
t pol
icie
s an
d pr
actic
es b
y th
e Ba
nk.
24 Annex 5
25 Annex 5
Title
& R
epor
t N
umbe
r N
ame
of P
roje
ct
eval
uate
d #
Obj
ectiv
es o
f the
Pro
ject
#
Obj
ectiv
es o
f the
Impa
ct E
valu
atio
n
Indi
a: W
est B
enga
l Ag
ricul
tura
l D
evel
opm
ent
Proj
ect (
IER
1214
0) In
dia:
Wes
t Be
ngal
Ag
ricul
tura
l D
evel
opm
ent
Proj
ect
To
inst
all 1
8000
sha
llow
tube
wel
ls.
1 To
stu
dy th
e sh
ort-t
erm
impa
ct o
f sha
llow
tube
wel
l (ST
W)
irrig
atio
n on
farm
er c
osts
and
ben
efits
.
2 To
reex
amin
e co
mpl
etio
n re
ports
for m
ediu
m- a
nd lo
nger
-term
im
pact
s on
farm
er c
osts
and
ben
efits
, mac
hine
ry li
fe a
nd re
pair
cost
s, a
nd th
e in
tera
ctio
n of
the
STW
inve
stm
ents
with
the
dura
bilit
y of
the
grou
ndw
ater
reso
urce
itse
lf.
Bang
lade
sh:
Shal
low
Tub
ewel
ls
Proj
ect (
IER
1103
1) Ba
ngla
desh
: Sh
allo
w
Tube
wel
ls P
roje
ct
1 To
inst
all 1
0000
sha
llow
tube
wel
ls.
3 To
exa
min
e pe
rcep
tions
of I
rriga
ted
agric
ultu
re a
t the
tim
es o
f pr
ojec
t app
rova
l and
com
plet
ion
repo
rting
.