Document of

The World Bank

FOR OFFICIAL USE ONLY

Report No: 22533

IMPLEMENTATION COMPLETION REPORT(TF-21949; TF-21948)

ON

GRANTS

IN THE AMOUNT OF US$15.8 MILLION

TO

BRAZIL

FOR A

SCIENCE AND TECHNOLOGY SUBPROGRAM:SCIENCE CENTERS & DIRECTED RESEARCH -- PHASE I

ANDEMERGENCY ASSISTANCE

Dcc 17. xool

This document has a restricted distribution and may be used by recipients only in the performance of theirofficial duties. Its contents may not otherwise be disclosed without World Bank authorization.

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS

(Exchange Rate Effective )

Currency Unit = Cruzeiro (Cr$) (at Appraisal)Real (R$) (at Completion)

Cr$ 1 = US$ 0.0011Real (R$) 1 US$1

US$ 1 = Cr$ 913US$ 1 Real (R$) 1

FISCAL YEARBrazil: January 1 to December 31 Project: January 1 to December 31

ABBREVIATIONS AND ACRONYMSABC Agencia Brasileira de Cooperaqdo

(Brazilian Cooperation Agency, Ministry of Foreign Affairs)CAPES CoordenaXao de Aperfei oamento de Pessoal de Nivel Superior

(Coordinating Agency for Graduate Training)CEC Commission of the European CommunitiesCNPq Conselho Nacional de Desenvolvimento Cientifico e Tecnol6gico

(National Scientific and Technological Development Council)COAP Coordenaqdo de Apoio de Programas Prioritarios

(Priority Programs Support Coordination)CORPAM Comissao Coordenadora Regional de Pesquisas na Amaz6nia

(Regional Coordinating Commission for Research in the Amazon)CPATU Centro de Pesquisa Agropecuairia do Tr6pico Umido

(EMBRAPA's Agricultural Research Center for the Wet Tropics)CTC Comite Tecnico Cientifico

(Technic-Scientifical Committee)DCTEC Departamento de Cooperaqdo Tecnica

(Technical Cooperation Department)EC Evaluation CommitteeEEZ Economic and Ecological ZoningEMBRAPA Empresa Brasileira de Pesquisa Agropecuairia

(Brazilian Agricailtural Research Company)FESA Foreign Exchange Expenditure Special AccountFINEP Financiadora de Estudos e Projetos

Studies and Projects Financing Agency

Vice President: David De FerrantiCountry Manager/Director: Gobind Nankani

Sector Manager/Director: John RedwoodTask Team Leader/Task Manager: Judith Lisansky

FOR OFFICIAL USE ONLY

FNDCT Fundo Nacionalpara o Desenvolvimento Cientfico e Tecnol6gico(National Fund for Science and Technology Development)

FUAP Formuldtio (Inico de Apresentaqdo de Propostas(Standard Researsh Proposal Form)

GEA Grupo Especial de Acompanhamento(High Level Advisory Group)

GIAC Grupo Internacional de Acompanhamento Cientifico(International Advisory Group - IAG).

G-7 Group of Seven Industrialized CountriesLAG Intemational Advisory GroupIBAMA Instituto Brasileiro de Meio Ambiente e dos Recursos Naturais Renovaveis

(Brazilian Institute for the Environment and Renewable Resources)INMETRO Instituto Nacional de Metrologia

(National Metrology Institute)INPA Instituto Nacional de Pesquisa da Amaz6nia

National Institute of Amazon ResearchLCB Local Competitive BiddingLIB Limited Intemational BiddingMCT Minist&io de Ciencia e Tecnologia

(Ministry of Science and Technology)MIS Management Infornation SystemMMA Ministerio do Meio Ambiente e da Amaz6nia Legal

(Ministry of the Environment and Legal Amazon)MOD Memorandum of DirectorMPEG Museu Paraense Emilio Goeldi

(Emilio Goeldi Museum of Para)MRE Minist&rio de Relateoes Exteriores

(Ministry of Foreign Affairs)NGO Non Governmental OrganizationODA Overseas Development AgencyPADCT Programa de Apoio ao Desenvolvimento Cientifico e Tecnol6gico

(Scientific and Technological Development Support ProgramPIU Project Implementation Unit)RHAE Programa de Desenvolvimento de Recursos Humanos em Areas Estrategicas

(Development of Human Resources in Strategic Areas Program)RNP Rede Nacional de Pesquisa

(National Research Network)RFT Rain Forest Trust FundSEAIN Secretaria de Assuntos Internacionais

(International Affairs Secretariat)SECOP Secretaria de Coordenaqdo de Programas

(Program Coordination Secretariat)ST Secretaria Tecnica

(Technical Secretariat)SUDAM Szuperintendencia de Desenvolvimento da Amazonia

(Amazon Development Bureau)TOR Terms of ReferenceTS Technical Secretariat

This document has a restricted distribution and may be used by recipients only in theperformance of their official duties. Its contents may not otherwise be disclosed withoutWorld Bank authorization.

BRAZIL

SCIENCE CENTERS & DIRECTED RESEARCH PROJECT-PHASE I

CONTENTS

Page No.

1. Project Data 1

2. Principal Performance Ratings 13. Assessment of Development Objective and Design, and of Quality at Entry 1

4. Achievement of Objective and Outputs 5

5. Major Factors Affecting Implementation and Outcome 10

6. Sustainability 127. Bank and Borrower Performance 13

8. Lessons Learned 15

9. Partner Comments 18

10. Additional Information 20Annex 1. Key Performance Indicators/Log Frame Matrix 21

Annex 2. Project Costs and Financing 22Annex 3. Economic Costs and Benefits 23

Annex 4. Bank Inputs 24

Annex 5. Ratings for Achievement of Objectives/Outputs of Components 25Annex 6. Ratings of Bank and Borrower Performance 26

Annex 7. List of Supporting Documents 27

Annex 8. Beneficiary Survey Results 28

Annex 9. Stakeholder Workshop Results 29

Annex 10. List of Directed Projects. First Call 30

Annex 11. Directed Research Projects, According to the Priorities Identified by CORPAN 31

Annex 12. Directed Research Projects. Executing and Cooperating Institutions 32

Annex 13. Directed Research. Indicators and Implementation Difficulties 33

Project ID: P006568 Project Name: RF-Science Centers & DirectedResearch Project-Phase I

Team Leader: Judith M. Lisansky TL Unit: LCSRF

ICR Type: Intensive Learning Model (ILM) of ICR Report Date: December 1 7, 2001

1. Project Data

Name: RF-Science Centers & Directed Research L/C/TFNumber: TF-21949; TF-21948Project-Phase I

Country/Department: BRAZIL Region: Latin America andCaribbean Region

Sector/subsector: VY - Other Environment

KEY DATESOriginal Revised/Actual

PCD: 09/01/1993 Effective: 12/01/1994 02/01/1995Appraisal: 10/01/1993 MTR: 12/01/1995 09/01/1997Approval: 06/01/1994 Closing: 03/01/1997 12/01/1999

Borrower/Implementing Agency: Federative Republic of Brazil/SCTOther Partners: EC, USAID, ODA, RFT

STAFF Current At AppraisalVice President: David De Ferranti Shahid J. BurkiCountry Manager: Vinod Thomas Rainer B. SteckhanSector Manager: John Redwood Kreszentia M. DuerTeam Leader at ICR: Judith M. Lisansky Claudia SobrevilaICR Primary Author: Ivo Marzall; Loretta Sprissler

2. Principal Performance Ratings

(HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HL=Highly Likely, L=Likely, UN=Unlikely, HUN=HighlyUnlikely, HU=Highly Unsatisfactory, H=High, SU=Substantial, M=Modest, N=Negligible)

Outcome: S

Sustainability: L

Institutional Development Impact: M

Bank Performance: S

Borrower Performance: U

QAG (if available) ICRQuality at Entry: S S

Project at Risk at Any Time: Yes

3. Assessment of Development Objective and Design, and of Quality at Entry

3.1 Original Objective:

The Science Centers and Directed Research Subprogram of the Pilot Program to Conserve the BrazilianRain Forest aimed to promote the generation and dissemination of scientific knowledge relevant to

conservation and sustainable development activities in the Amazon region. This goal was to be pursuedthrough a two-pronged strategy: (i) support for a grants program for funding research projects on acompetitive basis in the Amazon region; and (ii) strengthening of two established research institutions ofthe Amazon, the National Institute for Amazon Research (INPA) in Manaus, and the Emilio GoeldiMuseum of Para (MPEG) in Belem.

The subprogram was divided into two separate, complementary, and simultaneous operations:(a) Emergency Assistance, aimed at rehabilitating/updating deteriorated infrastructure and equipment inboth centers; and (b) the Science Centers and Directed Research Project - Phase I, with two components:(i) Directed Research - aimed at financing selected research projects, and (ii) Science Centers - designed tocomplement the Emergency Assistance rehabilitation program and support the institutions' reorganization,improved management and decision making, enhanced human capacity, and dissemination of information.

Note: In view of the closely interlinked and complementary nature of the two operations under the Science Centersand Directed Research Subprogram (Phase I and Emergency Assistance), they are assessed together in the presentICR.

3.2 Revised Objective:N/A

3.3 Original Components:

3.3.1 Directed Research

The Phase I Directed Research component was designed to support a grants program to fund researchprojects evaluated and selected on a transparent, competitive basis in response to established priorities.Qualified grantees would include scientists from universities and other research institutions, both publicand private, located in the Amazon or working in close collaboration with an Amazon-based institution.Research priorities would address current gaps in research and the demand for research results by naturalresource users and decision makers. Research priorities for the program were identified during aparticipatory workshop led by CORPAM (Comissao Coordenadora Regional de Pesquisas naAmaz6nia--Regional Coordinating Commission for Research in the Amazon) and through additional expertconsultations with workshop participants, other members of the scientific community and the TechnicalSecretariats of other Pilot Program projects.

The following four broad research themes were identified: (i) Amazonian ecosystems structure andfunction (e.g. impact of land use changes on ecosystem function, biodiversity assessments and conservationstrategies); (ii) sustainable natural resource management (e.g. baseline studies on productivity of differentland and water use systems, studies on new or traditionally used forest products with potential economicvalue); (iii) socioeconomic and cultural systems (e.g. cultural and social base for the sustainable use ofAmazonian environment, impact of pollution on health); and (iv) low environmental impact infrastructure(e.g. adaptive research on renewable energies for direct use in the Amazon, studies on transport modes withlow environmental impact). The priorities for the first call for proposals were limited to the first threeareas mentioned above.

Proposals would be assessed by Evaluation Commissions (ECs) composed of highly qualified scientists,who would use two main criteria for evaluating proposals and making recommendations: (i) priorityresearch focus, i.e., the extent to which the proposals addressed the pre-identified priority research themes;and (ii) scientific merit, based on the innovative character of the proposed research, soundness ofmethodology, the qualifications of the proposed research team, the relevance of the research to sustainable

- 2-

development, and adequate infrastructure for the research proposed. Individual grants would not be largerthan US$300,000 for a two-year subproject.

The operational rules and procedures for both the Directed Research and Science Centers components wereincorporated in an Operational Manual, which was prepared in 1995 and revised in 1997.

3.3.2. Science Centers/Emergency Assistance

The objective of the Science Centers component of the Phase I project and Emergency Assistance projectwas to improve the scientific research and dissemination capacity of the region's most important sciencecenters - the National Institute for Amazonian Research (INPA) and the Emilio Goeldi Museum of Para(MPEG).

The design of the Science Centers and Emergency Assistance operations addressed the institutional issuesraised by the 1992 Needs Assessment Report prepared by an advisory group formed by the NationalAcademy of Science (USA). The main deficiencies of INPA and MPEG identified in the report included:(i) lack of clear institutional guidelines and focused research goals; (ii) low salaries and exodus of bestqualified staff; (iii) lack of administrative efficiency in research administration; (iv) lack of coordinatedactivities with other institutions; (v) deteriorated infrastructure and lack of facilities for field research; (vi)lack of necessary equipment; and (vii) lack of a performance evaluation system. The NSA advisory grouprecommended the preparation of a strategic plan by each center to help inform the objectives of a sciencecenters project. The strategic plans were completed in 1994 and were integrated, as recommended, into theproject design.

The Science Centers/Emergency Assistance operations were designed to: (a) strengthen institutionalmanagement and administrative functions; (b) support the renovation and expansion of researchinfrastructure and equipment; (c) increase human resource capacity in scientific research and education;(d) disseminate research results; (e) provide technical assistance in science and dissemination management;(f) provide support to the project implementation unit; and (g) support key policy studies and plans atMPEG.

3.4 Revised Components:N/A

3.5 Quality at Entrv:

3.5.1 Overall

The objectives of both projects were sound and consistent with the general objectives of the Pilot Programto Conserve the Brazilian Rain Forest and, consequently, with the region's overall conservation andsustainable development aims, insofar as they responded to the most pressing needs of the Amazonianresearch sector. The project design was simple and straightforward. With two distinct components underthe Phase I project, in addition to the infrastructural support provided under the Emergency Assistanceproject, it addressed the urgent need for rehabilitating and upgrading the research capacity of the two majorAmazonian research institutes, and introduced an innovation to the region's research environment - namely,a competitive scheme for financing research projects which aimed to (i) direct research incentives to apre-established, limited list of crucial, relevant themes, and (ii) encourage the participation of all regionaltesearch entities.

-3 -

While project objectives and the technical strategy were sound, the project administrative and financingarrangements were less adequately designed. Complex disbursement arrangements hampered projectimplementation performance. Furthermore, the limitations of the internal bureaucracy of FINEP (thefinancial management entity for both projects), as well as the inexperience of local disbursement agencies,should have been more carefully evaluated during project preparation.

Considering that the Government's contribution to the project was limited, the initial threshold of theSpecial Accounts set up under the projects (US$500,000) eventually was deemed to be insufficient.Although the Bank raised the thresholds two times, ultimately to US$1.3 million (RFT-021949) andUS$1.0 million (CEC-21948), respectively, this did not fully resolve the project's slow disbursements. Inaddition, the financial management constraints were further addressed at mid-term through simplification ofthe fund-transfer arrangements and the creation of an adequately staffed and trained autonomous financialmanagement structure within FINEP, which subsequently improved implementation performance.

3.5.2 Directed Research Component

The objectives of the Directed Research component coincided with overall Phase I project objectives. Thestrategy adopted for selecting, financing, and monitoring the directed research projects was rational,innovative, straightforward and effective. It sought to ensure the funding of projects that effectivelyaddressed the priority problems identified by a wide range of regional stakeholders, including technologyusers, scientists and policy makers. By channeling project funds to a limited list of eligible researchpriorities, the project intended to optimize the use of funds and prevent their pulverization into activities oflesser importance.

The establishment of a competitive basis for financing the research program aimed to ensure support forthe best proposals and encourage the development of regional research capacity by targeting regionalinstitutions. The Directed Research component also provided for the development of well-defined selectioncriteria and monitoring and evaluation plans by specialized Evaluation Commissions (ECs) composed ofinternationally respected scientists to ensure both the judicious selection of subprojects and effectiveimplementation quality control according to project objectives. Responsibility for overall assessment andmonitoring of Directed Research subprojects was given to the International Scientific Advisory Group(GIAC), composed of top scientists who would monitor subproject implementation progress through annualinterfacing with the ECs, interviews with the execution teams and visits to subproject sites.

The strategy for generating relevant research outputs was sound. However, the dissemination strategy wasless adequate, despite the allocation of Directed Research financing for publication of the expected resultsand technologies to be generated by the subprojects. The strategy provided for the dissemination ofresearch results to the scientific community, the government, and large-scale producers such as the bigsawmills. It did not, however, seek to facilitate the dissemination of information on new technologies tosmall producers, which would have required a targeted knowledge and technology transfer strategy thatwas not included in the project. The project lacked a stated commitment to farmers and forest usersscattered throughout the Amazon region who could have benefited from any innovative technologiesgenerated by the project.

3.5.3 Science Centers/Emergency Assistance

The Science Centers component under the Phase I project and the Emergency Assistance project aimed tostrengthen and improve the research capacity of INPA and MPEG, whose efficiency in generating basicand strategic research would to a large extent determine the efficacy of the Amazonian research network at

- 4 -

large. The design of both operations was adequate. It aimed to optimize the two institutions as Centers ofExcellence through: (i) targeted investments in infrastructure; (ii) the introduction of structural adjustmentswith the intensification of strategic planning; (iii) a renewed emphasis on interdisciplinary andinter-institutional research approaches; (iv) the improvement of dissemination and fund-raising functions;(v) the introduction of staff improvement and performance evaluation schemes; (vi) and the improvement ofrelevant administrative procedures.

While the design of these operations was generally satisfactory, it underestimated the difficulties andchallenges involved in the implementation of an effective institutional and staff performance evaluationsystem and an aggressive fund-raising scheme under the current legal framework goveming bothinstitutions, which limits their capacity to promote or dismiss employees according to performance, andrestricts their financial management autonomy.

3.5.4 Organizational Arrangements

MCT's Secretariat for Scientific Development acted as the Technical Secretariat (TS) of the projects. TheTS was to be responsible for overall project execution and management, working in close coordination withFINEP, the project's Financing Agency. The main task of the TS was to ensure interaction betweenFINEP, the coordinators of the project components, and the project assessment and advisory groups.

FINEP was responsible for financial management and accounting, including the processing of timelydisbursements to executing agencies according to Annual Work Plans and agreed timetables. FINEP wasalso responsible for the analysis and contracting of Directed Research subprojects and for assisting withlocal subproject management. Project Implementation Units (PIUs) were also established at each ScienceCenter to manage and monitor the implementation of the Science Centers component and EmergencyAssistance project.

In addition, project advisory groups were established to assist the TS with defining relevant policies,formulating implementation plans, and analyzing Directed Research proposals, as well as defining theproject monitoring and evaluation plan. These groups included CORPAM, the International ScientificAdvisory Group (GIAC) and the Directed Research Evaluation Commissions (ECs).

CORPAM assisted the TS with defining the general norms of the project and the research priorities to beadopted. GIAC. was created to carry out independent scientific evaluations of the project, while the ECswere mainly responsible for selection and implementation quality control of Directed Research subprojects.

While the project implementation arrangements were logical in their design, they did not sufficientlycompensate for the institutional weaknesses of FINEP and the two Science Centers, which contributed tofunding delays and the generally slow pace of project execution. However, project implementationimproved significantly after the Mid-Term Review.

4. Achievement of Objective and Outputs

4.1 Outcome/achievement of objective:

Overall the Phase I and Emergency Assistance projects achieved the main objectives. The projects helped torevitalize the research activities in the Amazon, hampered since the late 1980's by funding and personnelconstraints. Through support for directed research, the Phase I project also helped to expand scientificknowledge of the region's natural resources, assess the impact of human interventions on the region, andopen new opportunities for the sustainable management of Amazonian plants and animals. It is expected

- 5 -

that these results will contribute to improved development and preservation strategies for the Amazon. ThePhase I and Emergency Assistance projects also contributed to an expanded and improved researchcapacity at the two major Amazon research institutions through support for critical infrastructureimprovements, the introduction of strategic planning schemes, the training of researchers, and thedissemination of research results.

4.2 Outputs by components:

4.2.1 Directed Research

Twenty three research projects were selected from 116 applications under the first call for proposals,published in January 1995. Only projects under the first call for proposals are considered in this ICR; projectsunder the second call were done under a separate, bi]ateral EC-FINEP project in which the Bank did notparticipate. The call and selection procedures were transparent and followed the rules established by theproject. The great majority (70%) of the 23 selected projects were classified under the sustainable naturalresources management priority line, indicating the relevance of this theme among the priority Amazonianissues as defined under this component. Eight projects (35%) were listed under the Amazonian ecosystemsgroup, six of which were double-classified with the previous group; and four (17%) corresponded to thesocioeconomic and cultural systems group, There were no proposals under the low environmental impactinfrastructure category.

Directed Research subproject executors included INPA, the Brazilian Agricultural Research Corporation(EMBRAPA), the National Institute for Space Research (INPE), and the Brazilian Cocoa DevelopmentAgency (CEPLAC), along with other Amazonian institutions -- mainly universities and one NGO. About26 regional institutions, 17 national agencies from other states, and nine intemational institutionsparticipated in the implementation of the projects, indicating a strong inter-institutional andinterdisciplinary integration. In addition, 51 national and 26 international institutions cooperated with thesubproject executing institutions, indicating both a world-wide interest in Amazonian issues as well as anevident effort towards multi-institutional cooperation.

The majority of the subprojects (14) produced results or partial results, while most of the remainingprojects reporting no results were mainly long-term projects needing further research before reliableconclusions could be drawn. The Directed Research subprojects also contributed to the introduction ofinnovative methodological techniques for wider use in Amazon research activities. (See Annexes 10-13 formore detailed information.)

There were a number of technical difficulties in relation to the directed research subproject selectionprocess. Because of insufficient information and guidance to the candidates about the selection proceduresduring the proposal preparation phase, as well as some rigidity during the screening phase, a number ofexcellent, otherwise eligible proposals were rejected. Moreover, the proposal format adopted under theproject led to unnecessarily long and detailed proposals, requiring excess work by both proposers andreviewers. The call for proposals also tended to prioritize and encourage the financing of alreadyestablished, successful projects, so that most of the selected proposals involved ongoing, long-term researchprojects. As a result, only a few projects involved original research initiated with PPG-7 support. Inaddition, the thematic emphasis of the first call for proposals was subject to some criticism by, amongothers, the International Advisory Group of the Pilot Program to Conserve the Brazilian Rain Forest, whichheld that the demand for knowledge by other components of the Pilot Program should have been given morepriority.

-6 -

Implementation problems included funding delays, and irregular performance by some execution teams,resulting from coordination problems and the low commitment of executing staff, which jeopardized theimplementation and, occasionally, the very scope, of some subprojects. Other subprojects were affected byclimatic events and, in one case, by the partial destruction of a research trial by fire. In these cases, most ofthe previous work was lost, and in the case of agricultural and agroforestry projects, the trials had to bere-started from scratch. Overall, eight projects had fully satisfactory implementation performance, while inthe remaining 15 projects the performance was uneven or deficient.

4.2.2 Science Centers/Emergency Assistance

Both Science Centers endeavored to implement the project objectives and targets under the Phase I andEmergency Assistance projects. The main achievements were:

(a) Improved management: (i) the restructuring and prioritization of research programs; (ii) definitionof strategic objectives of the Science Centers; and (iii) introduction of competitive fund allocation systemsto support research projects within each institution. These improvements resulted in more rational researchmanagement performance and contributed to an increased execution efficiency.

(b) Infrastructure and equipment expansion and rehabilitation: (i) improvement of the hydraulic,electric, sanitary and fire-control systems; (ii) upgrading of the libraries and collection storage facilities;(iii) establishment of modem computer networks; and (iv) expansion and improvement of laboratories andoffices. Overall, the infrastructure improvements financed under the Phase I and Emergency Assistanceprojects had an energizing effect on the centers' staff and contributed to an improved working environment.

(c) Human resources upgrading. Staff training and staff participation in congresses and seminarshad been neglected over most of the 1980s and 1990's, mainly as a result of reduced funding. Training ofstaff at both institutions was clearly revitalized under the Phase I project. The number of PhDs was raisedby 34% at INPA and by 54% at MPEG, while, at project completion, most of the remaining research staffof both institutes were registered in graduate courses. The participation of the researchers in national andinternational scientific congresses was significantly expanded, and the post-graduate program for non-staffresearchers given by both Centers was considerably increased. Both Centers failed, however, to establisheffective staff and institutional performance evaluation systems, as envisioned under the project.

(d) Dissemination of research results. The achievements of this activity under the Phase I projectwas remarkable in both institutes. Besides the updating of the publication of their traditional scientificbulletins, largely neglected over the previous decade for lack of funds, both centers invested in a number ofdissemination activities, such as the publication of books, folders, videos, media material, and theorganization of a number of events aimed at the general public as well as scientists, students and children.Nevertheless, dissemination activities appear to be targeting a relatively limited audience, mainly scientistsand other specialized audiences, because neither of the two centers apparently view as part of their centralmissions the extension and technology transfer activities so lacking in the region.

(e) Scientific collections. Infrastructural improvements under the Phase I and Emergency Assistanceprojects were essential to the preservation of the various scientific collections at INPA and MPEG. Bothcenters also expanded their collections, and modernized and computerized their management.

(f) Inter-institutional cooperation. While much more could have been achieved in this area, bothINPA and MPEG expanded their cooperation with national and international institutions during Phase Iproject implementation. As a result, a number of important research activities were carried out with the

- 7 -

technical and financial support of research institutions, universities, governmental entities, privatecompanies, NGOs, and others.

Strategic Planning. The Centers endeavored to implement the improvements proposed in the 1994 SPguidelines, whose overall objectives were to define and establish: (i) general institutional guidelines; (ii)broad thematic areas of research; and (iii) administrative and institutional mechanisms to transform theinstitutions into Centers of Excellence. Both Centers were unable, however, to establish an ongoinganalysis of research priorities, potential benefits and targeted clients. With the continued absence of suchanalysis, it remains unclear how research resources are allocated among the various organizational levels.

4.2.3 Project Organization and Management

Project coordination

The Technical Secretariat was composed of a Technical Secretary and a number of assistants. Its mainactivities included: (i) the organization and dissemination of project-related statistical and managerialinformation; (ii) coordination of project monitoring and evaluation; (iii) organization of advisory groups forthe selection of research proposals; (iv) supervision of the budgetary analysis of the proposals; and (v)supervision of physical and financial monitoring and follow-up on the implementation of projectcomponents. The TS also produced annual implementation reports and the project completion report.

The performance of the TS was mixed. It played an important role in addressing financial managementissues together with FINEP. The TS also organized successful seminars for the discussion of projectimplementation progress and problems. The TS did not, however, organize regular annual meetings of theInternational Scientific Advisory Group (GIAC) as well as follow-up visits by the Directed ResearchEvaluation Committees (ECs), as foreseen by the project, except towards the end of the project. The TSreporting performance was deficient, as was its performance in establishing appropriate physical/financialperformance indicators and project impact indicators.

Local management

Each Science Center created an internal Project Implementation Unit (PIU) to manage and monitor theimplementation of the Science Centers component and Emergency Assistance project. Each PIU wasassisted by a foundation -- the Djalma Batista Foundation (FDB) in the case of INPA, and the Universityof Para Foundation assisting MPEG. The implementation of the Directed Research subprojects wascoordinated by the TS directly.

The performance of both PIUs was less than satisfactory. The performance of INPA's PIU was especiallycompromised by the performance of the FDB, which lacked the necessary administrative and financialmanagement capacity to manage project funds. The University of Para Foundation, on the other hand,functioned somewhat better in assisting MPEG's PIU.

Financial management

FINEP's performance was initially below expectations and contributed to continued implementation delays.While some changes introduced at mid-term significantly improved FINEP's performance, it remained lessthan fully satisfactory throughout the project implementation period. At the local level, procurementprocessing, fund transfers to PIUs and subproject executors, project accounting and financial reporting,and the hiring of personnel were all handled by the foundations contracted by the two Science Centers for

- 8 -

this purpose -- one of which functioned satisfactorily, and one which did not. FINEP also did not providesustained technical support to local level implementors.

Advisory bodies

Project advisory groups composed of representatives of the TS, FINEP, and the scientific, technologicaland private-enterprise communities, made significant contributions to defining relevant policies,formulating implementation planning, and analyzing Directed Research proposals, as well as defining themonitoring and evaluation strategy of the project. These groups included the Regional CoordinatingCommission for Research in the Amazon (CORPAM), the International Scientific Advisory Group (GIAC)and the Directed Research Evaluation Committees (ECs).

CORPAM is an advisory organ of MCT. It assisted the TS in defining the general norms of the Phase Iand Emergency Assistance projects and the research priorities adopted. However, CORPAM did not meetfor several years after project start-up and contributed little to project implementation. MCT is nowdeveloping a new mechanism to deal with the scientific research strategy for the Amazon.

GIAC. The Intemational Scientific Advisory Group (GIAC) was created to carry out independent scientificevaluations of the projects. The group was composed of five preeminent national and internationalscientists. To carry out its mandate, GIAC relied on progress reports; interviews with the TS, FINEP, localfoundations, Science Center directors and Directed Research project coordinators; and direct observationsthrough field visits.

GIAC was originally expected to meet every year. However, as GIAC's activities were supposed to befinanced with counterpart funds, the shortage of funds was cited by the TS as the reason why the annualmeetings were not held. As a result, GIAC carried out only two evaluations during the life of the project, atmid-term and completion, both partly financed with Bank supervision funds. The outputs of GIAC'sreviews, while relevant, seemed to reflect operational constraints imposed by the group's Terms ofReference (which were prepared by the TS but not submitted for the Bank's no objection), as well as timelimitations. The lack of an efficient project monitoring scheme and the absence of a central capacity tosynthesize and consolidate the individual field reports, combined with the late delivery of these reports andthe limited time given to GIAC to assess implementation progress, contributed to the preparation ofincomplete evaluation reports. In particular, while GIAC's assessment of the Science Centers componentand Emergency Assistance was accurate and the implementation problems were generally well identified inboth evaluation reports, the outputs of many Directed Research subprojects, their relevance to theobjectives of the Pilot Program and their potential environmental and socioeconomic impact were notadequately addressed. Thus, while the institution of an independent evaluation and advisory group was animportant design feature of the Phase I project, the actual impact of GIAC evaluations andrecommendations on project implementation was limited.

ECs. The mandate of the ECs was to analyze and select Directed Research proposals, monitor subprojectexecution, and provide the necessary guidance to the implementation teams. Each EC was responsible forone or more Directed Research subprojects. The ECs were supposed to perform their task through annualfield visits to the projects, accompanied by TS and FINEP members. These field visits, however,commenced only after mid-term (due in part to delayed start-up of Directed Research projects).

Monitoring and evaluation

Monitoring of project physical and financial implementation was generally adequate in both Science

-9-

Centers. However, performance monitoring was weak. The centers have not internalized the need tomonitor the results of the strategic planning process and to establish baselines from which progress can beassessed. The same applies to project investments. It appears that no baseline data or markers formonitoring have been established to monitor the impact of project-financed activities on the quality andproductivity of research, dissemination of research results, integration with other national and internationalresearch institutions, or performance of administrative activities. Moreover, no procedure for evaluating theimpact of the research on clients was ever established.

It must be recognized, however, that in the final implementation phase, MCT made a formidable effort tocompile and synthesize the results of Directed Research subprojects. But due to the lack of adequatetraining of the project coordinators, the evaluation guidelines were not uniformly applied. As a result, theindividual subproject evaluation reports were uneven. While some were reasonably complete and included avaluable, critical implementation analysis that was very useful for GIAC's assessment work, others weresuperficial, thus increasing the difficulty of a full assessment of subproject results.

GIAC evaluations identified a series of problems related mainly to Directed Research subproject planning,execution and coordination, which were not, however, corrected during implementation, indicating thatGIAC's authority was not duly recognized. Legal clauses, or some other mechanism, should be consideredfor future operations to strengthen the application of GIAC recommendations. Although the institutions hadtaken full responsibility for Directed Research project execution, they generally failed to make the requiredimplementation follow-up.

4.3 Net Present Value/Economic rate of return:N/A

4.4 Financial rate of return:N/A

4.5 Institutional development impact:

The impact of the project on the institutional development of both INPA and MPEG was evident. Theirorganizational structure was improved and the research strategy in both centers was rationalized towardinterdisciplinarity. The research projects ceased to be individual operations to become the responsibility ofthe institution. This involved a remarkable behavioral change at the level of both researchers and directors.The administration was computerized and became more efficient. The libraries and the scientific collectionswere organized and computerized to facilitate their management and expand their use.

5. Major Factors Affecting Implementation and Outcome

5.1 Factors outside the control of government or implementing agency:

There were no major exogenous factors that affected the implementation performance. The originalthreshold level of the Special Account which was later deemed to be insufficient initially contributed to aslow funding flow. This was corrected upon Bank's identification of the problem.

The Asian/Russian economic crisis that deeply affected the Brazilian economy in 1997 and 1998 may havemarginally contributed to a reduced funding flow in the late implementation phase. And the Government'sReal devaluation policy in early 1999 reduced both centers' budgets in real terms.

- 1 0 -

5.2 Factors generally subject to government control:

The late budget approval each year by the Brazilian Congress and the consequent delay in the transfer offunds to the executing agencies were serious constraints on the timely implementation of both infrastructureinvestments and directed research activities. There is no easy solution to this issue, as public institutions inBrazil are prohibited by law from holding on to counterpart funds from one year to the next in order tocover the 4-5 month funding gap experienced at the beginning of every fiscal year.

The Government's budget and personnel policies were determinant constraints on the implementation ofstaff expansion and performance evaluation programs in both Science Centers. The current policy alsolimited the Centers' autonomy to implement alternative fund-raising schemes. These adverse policies forboth public service in general and the scientific research sector in particular are currently being reviewedby the Brazilian Congress.

5.3 Factors generally subject to implementing agency control:

The fund transfer arrangement initially adopted under the project was complex, involving variousintermediary institutions (MMA, MCT, FINEP) and the corresponding need for annually renewedagreements between these institutions. This was considerably improved at mid-term through the adoption ofdirect transfers to FINEP.

The routine delays in Brazilian budget approvals also posed a serious constraint on FINEP's fundingperformance. This was reportedly corrected for the second call for research proposals financed by the EC,using FINEP's private sector financing system, through which the institution uses its own funds charging,matter-of-factly, the corresponding interests.

FINEP's slow disbursement procedures contributed to delayed implementation early in the project. Thiswas eventually corrected with the creation within FINEP of a special project managing group composed ofwell-trained staff, and the acceleration of the procedures. However, administrative performance at the locallevel, particularly at INPA, remained less than fully satisfactory throughout the life of the project.

Inadequate monitoring results were detrimental to the impact evaluation of Directed Research activities.Similarly, the lack of adequate monitoring schemes within the Science Centers posed a serious constraint tothe achievement of excellence.

5.4 Costs andfinancing:

Total project costs were estimated during appraisal at US$20.1 million, of which US$17.1 million wouldbe funded from grants and US$3.0 million in Government counterpart financing. The total cost of theEmergency Assistance project was estimated at US$5.0 million (US$2.5 M from grants and US$2.5 Mfrom the Treasury). The total cost of Phase I was estimated at US$15.1 million (US$14.6 M from grantsand US$0.5 M from the Government). Actual expenditures totaled the equivalent of US$26.7 million(133% of the estimates). This was partly due to inflation and exchange rate variations (four distinctcurrencies were involved), and, to a great extent, to the higher than expected contribution of the BrazilianGovernment, as shown below. The considerably higher than foreseen contribution of the Federal Treasury(308% of the expected total participation) is a clear indication of the high priority given to the project bythe Government, as shown in the following table:

US$ million equivalentSource Appraisal Actual % of Appraisal

EmergencyAssistanceUSAID 2.00 2.00 100RFTF 0.50 0.50 100Government 2.50 3.90 156Sub-total 5.00 6.40 128

Phase IRFTF 8.50 8.50 100CEE 5.40 5.62 104ODA 0.70 0.81 116Government 0.50 5.33 1,066Sub-total 15.10 20.26 134

Total Grants 17.10 17.43 102Total Gov. 3.00 9.23 308Grand Total 20.10 26.66 133

6. Sustainability

6.1 Rationale for sustainability rating:

The project contributed to the sustainability of both Science Centers. Improved infrastructure contributedto the sustainability of libraries and collections. The implementation of strategic planning, organizingactivities according to a long term research strategy, is an evident sustainability factor. Expandeddissemination, through publications, seminars, conferences and the general media, leads to the adoption,and thus to the sustainability, of research results.

Sustainability is likely -- but there are risks. In terms of the sustainability of the Science Centers, along-term maintenance plan for the infrastructural improvements is lacking. Also missing is a formal,committed link with regional extension schemes to ensure the adoption of research results at the smallproducer level. This is a serious constraint on the widespread use of research results by the farming andforest user community and, therefore, a limiting factor to sustainability.

Other major constraints to sustainability are of a legal nature, consisting of government policies over whichthe project had no influence. They are, mainly, the current personnel policy, which is responsible fornoncompetitive salaries, limited scope for promotion, and institutional impediments to rewardingproductivity and performance; and budgetary constraints, related to the insufficient Government supportand to the limited administrative autonomy of the institutions.

On the other hand, the future financial sustainability of science and technology investments in the Amazonregion appear to be more promising than ever before due to recent major policy shifts in the Ministry ofScience and Technology (MCT) including the approval of major long-term sectoral and infrastructural

- 12 -

S&T investment programs which for the first time will be allocated in part by percentages to specificregions of Brazil, with some targeting of the Northeast and Amazon regions.

6.2 Transition arrangement to regular operations:

An interim bilateral project financed by the European Commission to support a second set of researchprojects is currently under implementation. A follow-up Phase 2 project is being proposed by the BrazilianGovernment. The project preparation process will depend on the joint definition by the Government, theBank and the donors (particularly USAID), of the project's strategy and scope, the integration of lessonslearned from the Phase 1 and Emergency Assistance projects, current needs, funding limitations and donorpreferences.

7. Bank and Borrower Performance

Bank7.1 Lending:

7.1.1 Disbursements

The Bank and Donor disbursement performance was generally satisfactory. The Bank responded to theRecipient's requests aimed at accelerating administrative flows and improving project implementationperformance. With this purpose, the Bank/donors raised the amounts of the Special Accounts when itbecame evident that their original size was insufficient, and also made exceptional concessions for theRecipient's direct contracting of goods and services up to established cost limits. However, sporadicreplenishment delays by the donor agencies sometimes contributed to delayed disbursements.

7.2 Supervision:

Bank supervisions were regular, usually two per year. The Bank generally detected early emergentimplementation problems and helped the Recipient to find relevant solutions. Bank assessment of theinstitutional strengthening problems faced by the Science Centers during implementation were pertinent andprecise. However, the Bank was somewhat slow to take action to solve the Special Account issue, whichwas initially addressed in PY2.

7.3 Overall Bank performance:

Overall, Bank and Donor performance was satisfactory.

Borrower7.4 Preparation:

Project preparation performance was satisfactory.

7.5 Government implementation performance:

The extended project implementation period (five years instead of two) is an indicator that theGovernment's implementation performance was problematic. A case could be made that the originallyplanned implementation period was too short, especially in light of the complexity of implementationarrangements. But a longer project cycle would have done little to change the slow execution pace causedby the aforementioned bottlenecks in financial flow related to FINEP's slow disbursement performance andlocal financial management deficiencies.

- 13-

Slow disbursements contributed to delayed start-up of the Directed Research component. AlthoughFINEP's performance improved after mid-term, the accumulated delays and the local administrativedeficiencies contributed to the need for various project extensions. F1NEP was also slow in complying withthe submission of reports, work plans and studies agreed with the Bank.

MMA, which had the overall responsibility for the Pilot Program and the Science Subprogram, had asatisfactory perfornance in the funding process. Following MMA's initiative, the originally complicatedfund transfer scheme was simplified. However, MMA's interest in project monitoring and research resultswas below the level expected from a particularly concerned major partner.

7.6 Implementing Agency:

MCT. MCT's project management performnance was initially hampered by the financing scheme, whichthe TS eventually managed to satisfactorily improve. The TS successfully organized and coordinated amid-term and a completion seminar with participation of all stakeholders, to discuss project implementationachievements and problems. The TS satisfactorily coordinated GIAC's mid-term and final evaluationmissions. However, MCT did not organize the foreseen annual meetings for GIAC or the ECs, nor was itable to act upon GIAC's mid-term recommendations. The TS was also unable to establish appropriateimplementation performance indicators for project monitoring and evaluation, and did not effectivelysupervise the timely delivery or quality of project implementation reports by the Science Centers andsubproject executors. The TS itself was slow in reporting and in initiating preparation of the agreed Phase2 project. Its performance in attracting new funds to the subprogram as well as in getting consensus aboutneeded future activities was also below expectations.

FINEP. FINEP's performance was initially deficient, due both to the financing scheme and to the agency'sinternal difficulties, which negatively affected the project's implementation pace. FINEP's effectivenessconsiderably improved after mid-term, when the detected deficiencies were satisfactorily corrected.However, FINEP did not adequately assist the local management entities with their procurementdifficulties.

Science Centers. Both INPA and MPEG successfully achieved the planned foreseen infrastructure andequipment upgrades/rehabilitation/upgrading investments. Both centers did not, however, set up along-term maintenance plan for the rehabilitated infrastructure. If this gap is not urgently corrected, thesustainability of the project-financed investments will be at risk.

The slower than foreseen implementation pace of Science Center activities was due mainly to the irregulardisbursement performance, aggravated by deficiencies of the PIUs. The functioning of the PIU at INPAwas largely unsatisfactory, while the PIU at MPEG performed better.

In general, both Centers made considerable progress with the implementation of their respective strategicplanning guidelines. Any deficiencies in the achievement of certain key aspects of the Centers' institutionaldevelopment schemes, however, can be attributed to externally placed legal and administrative constraints,not easily overcome. In particular, both institutes could not implement a satisfactory staff performanceevaluation system, thus limiting their overall scope for excellence. Also due to their limited administrativeautonomy, the Centers' fund-raising capacity is still limited, although significant progress has been made inthis direction.

Advisory bodies. CORPAM, after playing a significant role in the definition of the project's researchpriorities, participated little in actual project implementation. GIAC produced two very useful evaluation

- 14 -

reports, but its recommendations were never really addressed. The effectiveness of both GIAC and the ECswas hampered by their limited access to project performance on the ground, since the annual evaluationsforeseen in the project design report were not carried out due to counterpart funding limitations.Furthermore, a uniforn evaluation of the Directed Research subprojects was hindered by the lack of awell-defined monitoring scheme.

7.7 Overall Borrower performance:

While the project ultimately achieved its main development objectives, an overall rating of unsatisfactoryfor Recipient performance has been given, due to management deficiencies at various levels thatcontributed to a slow implementation pace. Moreover, legal constraints and related government policiescontributed to limited progress in terms of staff performance evaluation and the development of an effectivefund-raising capacity.

8. Lessons Learned

Financing scheme

* The irregular funding flow resulting from federal budget management constraints and the complexfund transfer procedures adopted by the project negatively affected the implementation of both the researchprojects and the Science Centers' construction schedule. As funding delays may cause irrecoverable lossesto some research projects, the programmed financing timetable must be respected to meet the biologicalcycle constraints of crops, animals and forest-related trials.

* Budget-management rules and FINEP's own operational structure were designed to deal withexternal loans, but not with grants. Therefore, when project funding schemes differ from normal financingrules, they must be carefully planned, and executing officers trained accordingly before projecteffectiveness, to avoid funding delays.

* Some of FINEP's financial intermediary arrangements, such as those restricting the currency ofpayment of international vendors to Reais, were incompatible with efficient implementation. A solution tothis impasse and other identified restrictions must be established in a possible future operation.

Directed Research

* The calls for competitive research funding were based on very broad thematic lines. This led to awide range of thematic proposals distributed over a limited number of projects with a consequent dilutedimpact. Eligibility criteria should therefore be more strict to encourage concentration and ensure solutionsto very precise problems.

* The first call for proposals, which was financed under the Phase I project, lacked integration andsynergy with the Pilot Program as a whole. Mechanisms should be included in future directed researchprojects to make them more responsive to demand and help identify opportunities for greater synergy withprogram objectives.

* Misunderstandings about the presentation of research proposal budgets appears to have led to therejection of proposals which may have been of good technical quality. Better clarification should be givento proposers on exactly what is expected from the proposer and how proposals are rated. A description ofselection criteria and any weighted point system used in the evaluation of proposals should also be

- 15 -

presented. In addition, the proposers should have access, through telephone or e-mail, to further assistanceduring project preparation. Detailed feedback should be provided upon request in cases of proposalrejection.

* The sheer bulk of proposals - about 100 pages each for 116 proposals - made both preparationand review an onerous process. The detailed proposal should be more closely specified, in terms of ashorter and more precise proposal of a maximum of 15 pages.

* In most cases, it was not possible to detect whether the research proposal was innovative, built onresults of previous research, a continuation of an existing project, or just a repetition of previousexperimental work. It is therefore important that the call for proposals describe the state of the art in termsof scientific knowledge for each theme, identifying the most important gaps in information needing researchand those eligible for project support. In turn, each proposal should include a brief examination of the stateof the art of its particular area of concentration, identifying the relevant information gap to be addressed bythe proposed intervention. The relation of the financed projects with accumulated knowledge, with regionalresearch objectives, and with ongoing research in the Amazon should be clear and visible.

* In general, the research proposals did not identify the possible users of the scientific knowledge tobe generated. Each proposal should include a description of the expected results and their respective area ofapplication, identifying the prospective users and possible information dissemination strategies.

* During implementation of Directed Research subprojects, it was observed that a number ofchanges in the subproject execution teams, including the replacement of project coordinators, wereundertaken. This appears to indicate an artificial composition of the teams, as well as a lack of institutionaland personal commitment to the projects. In future operations, coordinator replacement requests should bewell-founded and justified, and should be thoroughly analyzed by MCT.

* The evaluation of the performance and impact of the research projects was hampered by theabsence of pre-established performance and impact indicators. In future operations, each project proposalshould include a detailed monitoring and evaluation plan, including the identification of adequateperformance and impact indicators, as well as a post-project monitoring scheme, since the impact ofscientific research, especially in relation to biodiversity conservation or natural resource management, cangenerally be measured only in the long run. Such a scheme should be capable of evaluating, over anadequate period of time, the impact of the research results on the environment and the sustainable use ofAmazon resources.

* Low per diems discouraged researchers to carry out field work, in detriment to research quality.Per diems should be adjusted to encourage researchers to make the necessary field control. Mechanisms forincreasing the proportion of researchers' salaries eligible for financing to ensure the inclusion of NGOs,private institutions, and autonomous researchers (affiliated or not with institutions) would be desirable.

GIAC

* GIAC's assessment reports, while very useful, did not follow an established M&E framework,lacked uniformity and were incomplete. In future operations, assessment reports should follow clear M&Eguidelines. Directed Research project coordinators should receive M&E training to ensure consistentapplication of performance and impact indicators, which would permit comparison among the projects, andfacilitate independent assessment of Directed Research performance. Any future operation must also ensure

- 16 -

that adequate funding is available so that entities responsible for M&E are able to meet on a regular basis.

* GIAC evaluations identified a series of project implementation problems, which were not, however,corrected during implementation, indicating that GIAC's authority was not duly recognized. Futureoperations should include legal clauses, or some other mechanism, to reinforce the application ofrecommendations made by independent evaluators during project implementation. It is also recommendedthat, in order to optimize its independence, any future GIAC should be composed only of scientists who donot belong to any project beneficiary institution.

Strategic planning

* It is impossible to rationally allocate resources within and between the various organizationallevels, in the continued absence of ongoing analyses of research priorities, timing/costs, potential benefits,and potential clients. A fiuture operation should, therefore, revive the strategic planning exercise initiated in1993-94 and support a continued participatory strategic planning process for interaction amongstakeholders, as a key activity towards excellence. The permnanent assistance of external expertise might bedesirable, to ensure continuity and the correct application of the strategic planning methodology.

* Neither Science Center seems to have filly recognized that the nature and size of future externalsupport may be conditioned on the ability to present rational arguments, in terms of benefits from solutionsof the priority problems of identified clients. Therefore, the analysis of potential and actual institutionalroles must be completed, as well as the identification of clients, their priority problems in the managementof the region's natural resources, and the specific institutional response to priority problems.

Inter-institutional cooperation and scientific exchange

* Inter-institutional approaches to problem solving should be encouraged, with distribution ofproblems to the most appropriate institution or group of organizations. Demands for research should not beignored because they are beyond current capacity or mandate.

* Currently, there appear to be mixed cooperation types, ranging from foreign groups who treat theScience Centers as a stage rather than a partner, to individual foreign projects that provide significantcontributions to the institutes, from one-time infrastructural improvements to ongoing support forpeer-review-driven research. The Centers must endeavor to coordinate foreign cooperation agreementstoward partnerships that are advantageous to the Amazon.

Staff performance evaluation

* The attempts of both INPA and Goeldi to introduce staff evaluation schemes to improve researchperformance under the project were frustrated by the lack of motivation of the researchers to accept beingevaluated. Legal restrictions that prevent performance-based reward or sanctions of civil servants renderstaff performance evaluations ineffective. Personnel incentives for improved research activity will be scarceso long as the outlook for promotion is poor and productivity-based salary supplements are non- existent.An improved personnel policy at the Science Centers will depend on either changing the law that governspublic institutions or changing the juridical status of the Centers to give them the necessary autonomy toestablish their own personnel policies.

- 17 -

Training

* As there have been problems which jeopardized the best use of international training events, it isrecommended that, in the future, staff sent overseas should be fluent in the dominant language of thetraining country. Moreover, staff should have qualifications and experience sufficient to make use of theopportunities.

* To optimize staff experience in training events, their back-to-office reports should identify theaspects which were of most value and explain how the ideas/techniques will be incorporated intocurrent/future research at their institution.

* Staff should be encouraged to develop links with professional societies, participate in relevanttechnical conferences and training opportunities in Latin American countries. This would contribute to anincreased national and international visibility of both INPA and MPEG.

Graduate courses

* Most of the hundreds of persons educated at INPA and MPEG in MSc and PhD courses arereported to be still working in the Amazon. It would be useful to follow-up on their activities and organizerelevant refresher training courses.

* Low per diem rates have discouraged fieldwork of trainees, with negative effects on the graduateprogram. Realistic per diems would contribute to optimized results.

* It would be better to involve students to fill staff gaps in high-priority projects rather than createlittle empires of low-priority research. Departmental plans should provide the rationale for increasing thenumbers of postgraduate students, and indicate the priority ranking of research lines and projects to whichstudents might be attached.

Fund-raising

While substantial progress has been made, overall fund-raising efforts foreseen under the project werelimited in both centers, and results were below expectations. In general, the fund-raising capacity of publicinstitutions has been very limited in Brazil. Semi-autonomous, administratively more flexible institutionssuch as EMBRAPA have been more aggressive and successful in establishing financing and cooperationschemes with a great variety of public and private, national and international institutions. This hascontributed to overall improved and diversified research performance, and to the enhanced efficiency andcompetitiveness of these institutions. It appears that, as mentioned above, for the Centers to becomemodern, entrepreneurial institutions, either the current law has to be changed giving the institutions moreflexibility and autonomy, or the Centers have to change their juridical status.

9. Partner Comments

(a) Borrower/implementing agency:(Note: The Govermnent's 39-page project completion report (submitted in August 2000) is in the project files.

The report, which does not include an executive summary, provides an overview of project outputs.)

A draft ICR was circulated for Government review on December 1, 2000. MCT comments on the draft aresunmmarized below.

* Section 3.3.1: The limited size of the Special Account was due primarily to the Banks guidelineson the size of Authorized Allocations to the Special Account, and not to the limited size of Governmentcontributions to the project.

* Section 3.3.2: The assertion that the projects lacked a stated commitment to farmer and forestusers is not entirely accurate, since small farmers played an active role in some Directed Researchsubprojects, which also produced pamphlets for the dissemination of information.

* Section 4.3.4: The GIAC could not meet more frequently due to scheduling problems of theindividual members, and not due to the lack of Government funding.

* Section 9.2.3: Annual meetings of the Evaluation Committees under the Directed Researchcomponent were not foreseen by the original project design.

* Section 10: The problem of the "sheer bulk" of Directed Research proposals making both"preparation and review an onerous process" was addressed during the second call for proposals fundedbilaterally by the EC, when the size of the proposals was reduced and managed electronically by MCT.

* Section 10: The absence of pre-established performance and impact indicators was also addressedduring the EC-funded second call for proposals.

(b) Cofinanciers:N/A

(c) Other partners (NGOs/private sector):The following paragraphs constitute a synthesis of the comments and recommendations of participatingresearchers, NGO representatives, Pilot Program representatives, donors and other stakeholders in twoseminars held respectively in June and December 1999, on the project results. They express their positiveopinion on the project and their concern for an improved implementation performance in possible futureoperations.

Comments

* The project was fundamental for the development of scientific activity in the Amazon.

* The research themes were appropriate, the selection system transparent, the development strategyadequate, and the amount of resources were satisfactory.

* As a result of the project, INPA and MPEG were able to raise additional resources, throughagreements with national and international partners, to match with project funds.

* Project monitoring was important. Thanks to monitoring, a number of implementation problemsand constraints were detected and could, therefore, be timely corrected.

Recommendations

* Socioeconomic research under the project was insufficient and should, therefore, be expanded.

- 19-

* Administrative and financial procedures must be considerably improved and simplified.

* The monitoring system must be improved, to identify bottlenecks as well as implementationperformance, results and impacts.

* Efforts should be made to improve and intensify the integration of PPG7-supported research withthe other research activities in the Amazon.

* Research projects should reflect more effectively the local and regional needs and demands.

* Research projects should more explicitly take into account the accumulated knowledge, results ofpast research and the existing data-base on the Amazon.

* Research results should be made easily accessible to the final users.

* More emphasis should be put on participatory, interdisciplinary and inter-institutional researchprojects.

* Dissemination of the results should be included in the research proposals; if needed, additionalfunds should be considered for this activity.

* The financing of research projects should include support to young researchers and students,through integrated scholarship programs, which could be accredited as counterpart funds.

* Emphasis should be put on institutional investments leading to institutional and financialsustainability of the research centers in general, and to INPA and MPEG in particular.

* An ex-post evaluation of the project is essential before planning a follow-up project.

10. Additional Information

N/A

- 20 -

Annex 1. Key Performance Indicators/Log Frame Matrix

I. Science CentersA. Performance Indicators1. Institutional Development* Organizational restructuring * Unsatisfactory* Introduction of Internal rules * Satisfactory* Creation of Research Councils * Satisfactory* International projects * Satisfactory* Introduction of research matrix * Satisfactory* Introduction of research agenda * Satisfactory* Commitment to users * Satisfactory* Thematic laboratories * Satisfactory* Internal competition * Satisfactory* Decentralization * Unsatisfactory* Computerization * Satisfactory* Improvement Scientific Collections * Highly satisfactory* Administrative procedures * Unsatisfactory* Inter-institutional cooperation * Satisfactory* Diversification of funding sources * Satisfactory* Strategic planning * Unsatisfactory2. Construction/upgrading of works* Target-achievement and quality * Satisfactory* Execution Timetable * Satisfactory* Maintenance Plans * Unsatisfactory3. Human Resources Strengthening* Staff training * Satisfactory* Graduate courses * Satisfactory* Staff expansion * Unsatisfactory* Staff performance evaluation * Unsatisfactory4. Dissemination* To public * Satisfactory* To end-users * UnsatisfactoryB. Result Indicators* Improved research performance * Satisfactory* Improved dissemination * Satisfactory* Improved interdisciplinarity * Satisfactory* Improved inter-institutionality * Satisfactory* Increased budget * SatisfactoryC. Impact Indicators * Not evaluated, as impact indicators

were never developed by recipientII. Directed Research ProjectsA. Performance Indicators* Execution performance * Satisfactory* Monitoring and reporting * Unsatisfactory* Dissemination * UnsatisfactoryB. Result Indicators * Partial resultsC. Impact Indicators * Not evaluated, as impact indicators

were never developed by recipient

- 21 -

Annex 2. Project Costs and Financing

Annex 2a. Project Costs by Component.

Appraisal Estimates and Actual, Including Contingencies.(US$ ,000 equivalent)

Actual PercentageProject Component Appraisal Estimate Estimate of Appraisal

(%)Directed Research 5,400 5,600 104Science Centers-Emergency Assistance 5,000 6,400 128-Phase 1 9,700 14,660 150Total 20,100 26,660 132

Annex 2b. Project Cost by Procurement Arrangement(US$ ,000 equivalent)

Expenditure Procurement Method Procurement MethodCategories App aisal Estimate Actual **

ICB NCB Other NBF Total ICB NCB 0th. NBF Total1. Works _ 1,440 2,459 3,8992. Goods 1,342 3,287 2,250 452 7,3313. Services _ 2,302 737 3,0394. Misc.* I__ 15,820 5,820Total 1,342 4,727 12,831 1,189 20,089

*Includes Directed Research subproject contracts, administrative expenses and financial agent fee**MCT/FIN1EP were responsible for maintaining complete procurement records, but the relevantinformation was never sent to the Bank for inclusion in Annex 2b of the ICR.

Annex 2c. Project Financing by Component

(in US$ ,000 equivalent)

Component Appx aisal Estimate Actual Estimate Percent ge of Ap praisal__._ Donors Gov. Total Donors Gov. Total Donors Gov. Total

Directed 5,400 0 5,400 5,620 0 5,620 104 104ResearchScience Centers-Emergency 2,500 2,500 5,000 2,500 3,900 6,400 100 156 100-Phase 1 9,200 500 9,700 9,310 5,330 14,640 101 1,066 151-Sub-total 11,700 3,000 14,700 11,810 9,230 21,040 101 308 143Total 17,100 3,000 120100 17,430 9,230 26,660 102 308 133

- 22 -

Annex 3. Economic Costs and Bnwnr -

N/A

Annex 4. Bank Inputs

(a) Missions:Stage of Project Cycle No. of Persons and Specialty Performance Rating

(e.g. 2 Economists, 1 FMS, etc.) Implementation DevelopmentMonth/Year Count Specialty Progress Objective

Identification/Preparation8/92-2/93 1 (technical (env'tl science)

discussionsiDC

6/93 2 (env'tl science, strategicplanning)

Appraisal/Negotiation10/93 8 (env'tl science, legal,

operations, inst'l analysis,engineering)

Supervision2/95 4 (env'tl science, economics, S S

engineering)3/96 4 (env'tl science) S S12/96 5 (env'tl science, engineering,

social science)4/97 2 (social science, env'tl science)

9/97 (M I R) 6 (social science, env'tl science,operations, financial mgt.)

2/99 5 (social science, env'tl science,economics, operations, financialmgt.)

6/99 4 (social science, env'tl science,operations, economics)

ICR11/00 I (inst'l. analysis) S S

(b) Staff

Stage of Project Cycle Actual/Latest Estimate________________ _ __ No. Staff weeks US$ ('000)

Identification/Preparation 40 79.8Appraisal/Negotiation 72 236.2Supervision 97 317.6ICR 33 75.0Total 242 708.6

- 24 -

Annex 5. Ratings for Achievement of Objectives/Outputs of Components

(H=High, SU=Substantial, M=Modest, N=Negligible, NA=Not Applicable)Rating

* Macro policies O H OSUOM O N * NAI Sector Policies OH OSUOM ON * NA

* Physical O H *SUOM O N O NAX Financial O H OSUOM O N * NA21 Institutional Development 0 H O SU 0 M 0 N 0 NAZ Environmental O H OSUOM O N O NA

SocialE Poverty Reduction O H OSUOM O N * NAE Gender O H OSUOM O N * NALIi Other (Please specify) O H OSUOM O N * NA

IZ Private sector development 0 H O SU O M 0 N 0 NA* Public sector management 0 H O SU *M 0 N 0 NAD Other (Please specifyJ O H OSUOM O N * NA

- 25 -

Annex 6. Ratings of Bank and Borrower Performance

(HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HU=Highly Unsatisfactory)

6.1 Bankperformance Rating

Z Lending OHS OS OU OHUN Supervision OHS OS OU OHUEl Overall OHS *S OU O HU

6.2 Borrowerperformance Rating

2 Preparation O HS * S O U O HUI Government implementation performance 0 HS 0 S 0 U 0 HU

Z Implementation agency performance 0D HS 0 S 0 U 0 HUZ Overall 0 HS * S O U O HU

The unsatisfactory rating listed above is due to the project financial management agency's (FINEP)inadequate performance during most of the project. It should be noted, however, that FINEP's performanceimproved significantly after the Mid-Term Evaluation (September 1997).

- 26 -

Annex 7. List of Supporting Documents

1. Staff Appraisal Report2. Loan Agreement3. Aide-memoires4. Forms 5905. Letters to the Government6. Back-to-Office Reports7. Recipient's Implementation Completion Report8. Mid-term Review9. Planejamento Estrategico do MPEG. 199410. Planejamento Estrategico do INPA. 199411. Relat6rio Final MPEG. Dezembro 199912. Relat6rio Final INPA. Maio 200013. PPG-7 C&T. Resuitados. Dez. 199914. PPDs. Resultados15. Edital16. Edital17. Relatorio de Finalizacao

- 27 -

Annex 8. Beneficiary Survey Results

N/A

- 28 -

Annex 9. Stakeholder Workshop Results

Please see Section 9(c) for the results of two stakeholder workshops held in June and December 1999.

- 29-

Additional Annex 10. List of Directed Research Subprojects

Number Title6 Evaluation of wild cocoa (Theobroa cacao L.) gernmplasm2 Data base of aTomatic plants and fruits: oils and scents3 Information base for the sustainable use of forest resources in Para State4 Biorhnsical chantes associated with apricultural activities in Westey Amazonia5 Ecological studies: native forest management, and rehabilitation of degraded

pastures and cTopping areas in Acre State6 The Ticuna universe: territory, health and envioronment7_ biEntomology of malaria in Amazonian settlements8 Characterization of ecosystems in oil palm plantations9 Biology dynamics of forest fratmentation14 Environmental idmpacts of agroforestry activities on Amazonian ecosystems, and

sustainable optionsl l Phlebotominae, Triatiminae and Trypanosomatinae: diagnostic and monitoring of

biological diversity and interactions with Amazonian populations12 Conservation and management of Trichechus inunguis (cow fish) in captivityl13 Developmen of agricultural technologies adapted to Amazonian conditions14 Development and evaluation of agroforestry systems for the Amazon15 Population dynamics and generation of endemic diseases16 Developrnent of agroforestry systems in Roraima State1 7 Edapho-climatic zoning for fast-growing forest species in the Amazon1 8 Conservation and use of Amazon phytogenetic resources19 Influence of tide pulsing on the ecologic dynamics of floodable armas in the

Amazon

20 Rehabilitation ofdegraded pastures through agroforestry in Western Amazon21 Advanced geoprocessing technologies for supporting sustainable development in

the Amazon22 Sequestration of carbon emissions from land use and soil cover changes23 Energy, water vapor and C02 balance in Central Amazon forest areas

- 30 -

Additional Annex 11. Directed Research Projects, According to the Priorities Identified byCORPAM

Priority C tegoriesLow

Socioeconomic EnvironmentalAmazonian Sustainable NR and Cultural Impact

Projects Ecosystems Management Systems Infrastructure

1 x2 X3 X_ _ __ __

4 _ _ _ _X

32 _ __ X

6 - X -7 X8 X X_ _ _

9 X X10 X11 x12 X X13 X14 X15

16 16

18 X19 X20 X _ _ _ _ _

21 x22 X X23 X

Total 8 16 4 0% 35% 70% 17% 0%

Mixed 6 6 0 0% 26% 26% 0% 0%

Primary source: PPDs Relat6rio Final

- 31 -

Additional Annex 12. Directed Research Projects. Executing and Cooperating Institutions

Executing Institutions

CooperatingProject MCT CNPq INPA MPEG EMBRAPA INP CEPLAC Other Institutions Agencies

E Rea Nat'l Int'l Nat'l Int'l1 _____ P P(2) C P(4) __ 3 32 P C P (-) 13 P P P C P (1) 5 34 C 35 _ C __6 .P -C - 2

7 C = P(2) P(1) 38 C P - P (1) 1 29 _ C _ P (1) 1 110 P_ = C_ (2) P (2) 1 1_1 ___ Cc __ P(5) P(3) 2 112 C _ __- _ < P (1) _2

13 . C - P P (2) 214 _ P ___ __C ___ 715 - 1 1

16 C _______ __=_

17 C -= C = = P(6)P(l)P(1) 3 ____

1 8 P C 1_ _ _ _ _ _ _ _ _ _ _ _______ ~~P (1) _ _ _ _ _ _

19 _ _ C _ P (I) P (1) P (2) 4 220 ___ P(3) c___ C _(1P ( PP(1) P (2) 7 721 _ _ P ___ C _ _ P (2)22 P ___ c C ________ 4 723 P _ C P () P 1 2

Total C 0 0 10 1 5 3 1 5 0 0Total 1 1 8 1 4 1 0 26 17 9

Total Cooperation 51 26

C = coordination; P = participationTle figure in parenthesis indicates: (a) for the specified institutions: the number of participating intra-institutional units, inaddition to the coordinating unit; and (b) for other institutions: the number of participating agencies.Primary source: PPDs Relat6rio Final

- 32-

Additional Annex 13. Directed Research Indicators

_______ Indicators Self-Reported Difficulties (5)Resultsl Results/

Project Cost Co- Initial Perfor- Objec- Imple-No. R$ Financing Situation Users mance tives ment. Impact Financial Admin. Tech.

(1) _2 < (3_ (4)1 265,000 OK OK D OK N X X X2 265,135 CNPq OK OK S N OK OK X X X3 261,108 WWW OK N D N OK X X X4 70,995 Embrapa OK N S OK N X X5 200,000 Ford Fdn. OK OK D N _ E X X X6 205,880 Funai OK N D p OK X X X7 156,600 PPI-3190 OK OK S P OK OK N N X8 265,000 Orstom OK OK D OK OK D X X N9 265,000 Smithson. N N D OK OK OK X X N10 265,000 CEC OK N D N E X X X11 319,563 FIOCRUZ OK N S P OK D X X X

CIDEIMUFRS.

1 2 158,219 .- OK N D .p OK D X. X X_13_ 223,000 INPA OK N D P OK OK X X X14 265,000 OK OK D N E X X15 186,484 Municip. OK N D N N X X X

Airao ._ ._.__16 100,237 OK N D N OK N N X17 300;000 _ OK N D N - M X X18 210,000 Embrapa _ OK N D N OK OK x x x19 265,000 INPA OK OK S OK D X N X

M. PlanckSHIFT _ _ _ _ _

20 295,611 - OK OK D P OK OK X X N21 265,000 - OK N S OK OK OK N N N22 306,464 - OK OK S OK OK OK X X N23 300,000 UKTiger E E S OK OK D X X N

LBA _ _ _ _Total _ 23 23 23 23 23 23 23 23 23

D 15 5E I__ __ 1 __ _ _ 3

M 1_ _ 9 _3 1 3 5 OK 21 9 8 12 1 l =__

S ___ _ _ _ __ 8 _ _ _ __ _

X = _ - 20 18 16

(1) Initial situation & Users: OK correctly stated in the report; N not stated; E = incorrectly stated.(2) Performance: Implementation performance: S = satisfactory; D = deficient.(3) Results/Objectives: Project results vis-a-vis objectives: OK = final results achieved;

P = partial results achieved; N = no results mentioned.(4) Results/implementation: Institutional-organizational results achieved as a consequence of project implementation:

OK = correctly stated; = inclusion of results from pre-project accumulated research.(5) Difficulties: X = identified; N = not identified difficulties.Primary source: PPDs - Relat6rio Final

- 33 -