Document of The World Bank

Report No: 29151

IMPLEMENTATION COMPLETION REPORT(CPL-38870 SCL-38876 SCPD-3887S)

ON A

LOAN

IN THE AMOUNT OF US$ 132.5 MILLION

TO THE

GOVERNMENT OF INDONESIA

FOR A

BOOK AND READING DEVELOPMENT PROJECT

June 16, 2004

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS

(Exchange Rate Effective March 1995)

Currency Unit = Rupiah (Rp) Rp 1 million = US$ 456

US$ 1.00 = Rp 2,193Academic YearJuly 1 - June 30FISCAL YEAR

January 1 December 31

ABBREVIATIONS AND ACRONYMS

ADB - Asian Development BankICB - International Competitive BiddingICR - Implementation Completion ReportIKIPs - Teacher Training InstitutionsINT - World Bank Department of Institutional IntegrityLCB - Local Competitive BiddingNTEC - National Textbook Evaluation CommitteePIU - Project Implementation UnitPMU - Project Management UnitPusTekKom - Center for Educational Technology, Commication and InformationQAG - Quality at Entry

Vice President: Jemal-ud-din Kassum, EAPVPCountry Director Andrew D. Steer, EACIFSector Manager Emmanuel Y. Jimenez, EASHD

Task Team Leader/Task Manager: Alfonso F. de Guzman, EASHD

INDONESIABOOK AND READING DEVELOPMENT PROJECT

CONTENTS

Page No.1. Project Data 12. Principal Performance Ratings 13. Assessment of Development Objective and Design, and of Quality at Entry 24. Achievement of Objective and Outputs 45. Major Factors Affecting Implementation and Outcome 86. Sustainability 97. Bank and Borrower Performance 108. Lessons Learned 119. Partner Comments 1110. Additional Information 13Annex 1. Key Performance Indicators/Log Frame Matrix 14Annex 2. Project Costs and Financing 16Annex 3. Economic Costs and Benefits 19Annex 4. Bank Inputs 20Annex 5. Ratings for Achievement of Objectives/Outputs of Components 22Annex 6. Ratings of Bank and Borrower Performance 23Annex 7. List of Supporting Documents 24

Project ID: P003968 Project Name: BOOK & READING DEVELOPMENTTeam Leader: Alfonso F. de Guzman TL Unit: LCSHEICR Type: Core ICR Report Date: June 16, 2004

1. Project DataName: BOOK & READING DEVELOPMENT L/C/TF Number: CPL-38870; SCL-38876;

SCPD-3887SCountry/Department: INDONESIA Region: East Asia and Pacific

Region

Sector/subsector: Secondary education (47%); Primary education (47%); Tertiary education (2%); Sub-national government administration (2%); Central government administration (2%)

Theme: Education for all (P); Social analysis and monitoring (S)

KEY DATES Original Revised/ActualPCD: 05/14/1992 Effective: 09/27/1995 08/24/1995

Appraisal: 11/28/1994 MTR: 03/15/1998 11/01/1998Approval: 05/23/1995 Closing: 10/01/2000 12/31/2001

Borrower/Implementing Agency: Government of Indonesia/MOECOther Partners: Provincial Offices of the Dept. of Education and Culture; Book Center

STAFF Current At AppraisalVice President: Jemal-ud-din Kassum Russell CheethamCountry Director: Andrew D. Steer Marianne HaugSector Manager: Emmanuel Y. Jimenez Himelda MartinezTeam Leader at ICR: Alfonso F. de Guzman Brigitte DucesICR Primary Author: David A. Klaus

2. Principal Performance Ratings

(HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HL=Highly Likely, L=Likely, UN=Unlikely, HUN=Highly Unlikely, HU=Highly Unsatisfactory, H=High, SU=Substantial, M=Modest, N=Negligible)

Outcome: U

Sustainability: L

Institutional Development Impact: M

Bank Performance: U

Borrower Performance: U

QAG (if available) ICRQuality at Entry: U

Project at Risk at Any Time: YesSustainability for this project is modest. See section 6.1

3. Assessment of Development Objective and Design, and of Quality at Entry

3.1 Original Objective:The project aimed to improve the quality of basic education in Indonesia (Grades 1-9) through producing and distributing more and better-quality textbooks, increasing their availability in the classroom, and increasing their use by teachers in delivering the curriculum. A secondary objective, in line with the Government objectives for all sectors, was to decentralize and privatize some of the textbook production and distribution functions which had theretofore been carried out centrally by the Ministry of National Education (MoNE) (formerly the Ministry of Education and Culture).

Three policy changes deriving from the project objectives were: (1) the provision of all required textbooks to all children in basic education (replacing a previous de facto policy of about one textbook per every four children); (2) competitive procurement of junior secondary textbooks from private sector publishers and Balai Pustaka, the state-owned publishing house of the Department of Education and Culture (replacing the previous policy of centralized production of some textbooks and shopping for others); and (3) the decentralization of junior secondary textbook purchasing decisions to the provincial level (replacing the previous policy of centralized purchasing decisions at the national level). These objectives also reflected Bank’s Country Assistance Strategy which places emphasis on: (i) the importance of quality in education, especially in basic education, (ii) ensuring poor children had access to the necessary educational materials and (iii) supporting decentralization in the delivery of educational services. It complemented a series of parallel investments in school facilities and teachers, made through the six primary and junior secondary school projects.

3.2 Revised Objective:Not Applicable

3.3 Original Components:Component 1. Improving Textbook Availability and Quality. This component, accounting for over 90 percent of estimated total project cost at appraisal, was intended to assist in the production and distribution of more and better-quality textbooks to primary and junior secondary schools nationwide. It consisted of the following subcomponents:

Improving Textbook Availability. Under this sub-component, textbooks and teachers guides were to be provided to all children in basic education. In primary schools, student textbooks and teachers guides were to be provided in all subjects (civics, mathematics, Indonesian language, science, and social science) for all grades; teachers’ guides only were to be provided for three subjects (arts and crafts; local content; and sports and health). In junior secondary schools, student textbooks and teachers’ guides were to be provided for eight subjects (Indonesian language, mathematics, physics, biology, world and national history, economics, geography, and English); teachers’ guides but not textbooks were to be provided for another three subjects (arts and crafts, local content, and sports and health). The Government was to fund, from its counterpart contribution to project funding, the entire requirement of primary textbooks and teachers guides. During the project period, most of the junior secondary textbooks and teachers’ guides were to be funded from the proceeds of the Bank loan. The project was initially intended to provide about 157 million textbooks to primary schools and about 85 million textbooks to junior secondary schools.

Improving Textbook Quality. A National Textbook Evaluation Committee (NTEC) was to be established within the Department of Education and Culture, to evaluate junior secondary textbooks submitted by private publishers and Balai Pustaka and ensure that only textbooks of sufficiently high

- 2 -

quality would be purchased under the project, following detailed and transparent evaluation criteria. Technical assistance was to be provided to support the working of this committee.

Changing the Role of the Book Center. The Book Center, an agency of the Department of Education and Culture, had been primarily concerned with the production and distribution of textbooks. Under the project, it was to phase out its involvement in the physical production of textbooks, except for some subjects in which there is a limited demand, and develop into the government body responsible for setting and maintaining quality control standards for the education publishing industry. Technical assistance and training, including study overseas, was to be provided under the project to assist the Book Center in this transition.

Component 2. Reading Promotion. This component was to consist of a series of activities intended to improve the quality of children’s books and promote reading. Subcomponents were (a) the holding of workshops and training in writing, editing, illustrating and publishing children’s books, targeted at interested teachers, writers, editors, and publishers in all provinces; (b) the production and distribution of a guidebook for schools and teachers giving suggestions and guidance on how to promote reading among children; and (c) the preparation of multi-media educational programs aimed at the general public and containing messages promoting reading, to be broadcast on educational channels.

Component 3. Teacher Education. This component was to consist of the production of multi-media educational programs for the in-service training of teachers demonstrating how textbooks and reference materials can be used in the classroom.

Component 4. Monitoring and Evaluation. This component was to consist of activities to provide timely and accurate information on the actual receipt of books in the schools and their use, including (a) self-reporting from the schools, (b) on-site visits to randomly selected schools, and (c) two in-depth studies, one for primary education and one for junior secondary education, to assess the long-term effects of improving learning in basic education. Technical assistance was to be provided to carry out the monitoring and evaluation.

Component 5. Project Implementation. The project was to be managed by a Project Management Unit at the national level and implemented by Project Implementation Units in each province and in the Book Center. Technical assistance was to be provided to help manage the project and to build capacity at the provincial level.3.4 Revised Components:The fourth round of tendering for textbooks was cancelled due to inability of the Bank and GOI to reach agreement on procurement procedures.

3.5 Quality at Entry:Quality at entry is judged to have been unsatisfactory. No QAG review was undertaken at project inception. However, the available documentation indicates that an appropriate level of resources were provided for project preparation from both the Bank (for Bank staff and missions) and other donors, in particular, a Japanese grant for preparation which was used to hire consultants to undertake a number of relevant studies. The project was subject to review within the Bank, and comments were taken into account in the project design. General guidelines for the quality evaluation of textbooks were developed during appraisal, with the understanding that specific guidelines for this technical evaluation, which would provide transparency during the evaluation process, were to be developed during project implementation. On the Government side, the project benefited from experienced and dedicated individuals who had worked with the Bank previously and had been involved in all aspects of project preparation. Detailed arrangements had

- 3 -

been made to strengthen the various implementing agencies through training and technical assistance.

In retrospect, the major flaw in project design at entry was the arrangements for the procurement of textbooks, which were exceptions to the Bank’s standard international competitive bidding procedures and the failure to design, in detail, alternative arrangements on procurement which would have minimized opportunities for collusion in textbook pricing. Another factor contributing to the unsatisfactory rating is the lack of a monitoring and evaluation (M&E) system that systematically looked at the impact of textbooks on student learning. The absence of school-level procedures to ensure that students return textbooks at the end of each school year is also a failing.

4. Achievement of Objective and Outputs

4.1 Outcome/achievement of objective:Although the appraisal report suggests that student achievement, retention, and dropout might be worth monitoring towards the end of the project period, there was no monitoring to determine whether the project improved the quality of basic education in Indonesia. The appraisal document provided evidence from other countries demonstrating that an increase in the availability of textbooks in particular, provision of textbooks to each and every student--had had a major positive impact on students learning outcomes. It was therefore taken as axiomatic that if good-quality textbooks were provided on a one-to-one basis to students in basic education in Indonesia, the quality of education and students learning outcomes would improve. It would have been very difficult in any case to develop an indicator to measure changes in education quality deriving solely from the provision of textbooks over the project period, since the Indonesian education system was growing rapidly, and many other factors were also having an impact.

The appraisal team therefore developed indicators which measured outputs and processes, rather than impact. The quantitative indicators, in particular, the number of textbooks and teachers guides produced and distributed, are set forth in Annex 1 and discussed below in Section 4.2.

In addition to the quantitative indicators, three qualitative indicators were developed during appraisal:

How effectively books are distributed to schools. Books were distributed to schools and the achievement of this indicator is satisfactory. The contracts with publishers required distribution of the books to individual schools, and spot checks in one province during the completion mission indicated a one-to-one correspondence between the number of books procured and the number of books arriving in the schools.

How effectively teachers and students use textbooks in the classrooms. Independent monitoring done by teacher training institutes during the project period and spot checks done during the completion mission indicated that teachers and students used the books heavily, although their impact on learning was not measured. The condition of the books after three or four years of use also indicate that they have been used thoroughly. Discussions with teachers during the completion mission revealed that the teachers found the textbooks to be of high quality (thorough, good presentation of the subject matter) but in some cases too comprehensive to be taught in a single school year. In many cases, teachers were using supplementary materials (other textbooks available on the open market) which presented less material and had more examples and more problems/questions to help students learn. The achievement of this indicator is rated partially satisfactory.

How well teachers, parents, and administrators understand the objectives and procedures of the system. Spot checks during the completion mission indicated that teachers, parents, and school administrators probably did not understand that students were required to return the books at the end of

- 4 -

every year so that they could be used by the students in the following year’s class. Although books were provided in junior-secondary education at a 1:1 book-student ratio, as of six months after the Closing Date, many of the books (one third to over half of those purchase in the first round of procurement) were not returned or accounted for. At least in some schools, there was no system developed by school administrators or teachers to keep track of the distribution of textbooks to individual teachers and students, and no sanctions for students who did not return books at the end of the school year. All parties (administrators, teachers, parents, and students) apparently believe that the Government will eventually resupply the missing books. Meanwhile, students share the remaining textbooks on a 2:1 or higher student-book ratio and/or use supplementary materials bought by the school or the parents on the open market. Due to the high rate of loss and apparent misunderstanding of procedures for returning books the achievement of this indicator is rated unsatisfactory.

The project’s decentralization objective was achieved modestly: the responsibility for the procurement of textbooks was devolved from the national to the provincial level. It is now the Department of Education’s intention to decentralize this responsibility further, to the district level. However, a system of checks and balances on procurement and monitoring have yet to be put in place. Moreover, during the project itself alleged procurement irregularities led to an investigation by the World Bank Department of Institutional Integrity (INT) and a declaration of misprocurement. At a very minimum, the irregularities led to higher prices paid for textbooks than should have been the case, and fewer books reaching the schools. The project’s privatization objective was also achieved modestly: many key activities provided by the government before the project are now provided by the private sector. Before the project, all textbooks were written by Department of Education teams. The books were not popular and often were judged to be too theoretical. Now, private publishers find their own authors; the Department of Education has only to provide the curriculum and exercise quality control. Publishing also used to be done by the Government, but now it is done by the private sector.

The three policy changes deriving from the project objectives: (1) the provision of all required textbooks to all children in basic education on a one-to-one basis; (2) competitive procurement of junior secondary textbooks from private sector publishers; and (3) the decentralization of junior secondary textbook purchasing decisions to the provincial level were implemented with varying degrees of success, as described elsewhere in this report.

4.2 Outputs by components:Component 1. Improving Textbook Availability and Quality. Improving Textbook Availability. During project implementation, the targets for textbook provision were revised upwards to about 197.5 million textbooks for primary schools and about 101.5 million textbooks and 1.9 million teachers’ guides for junior secondary schools. The outputs of this sub-component are mixed. The primary education portion of this component was implemented by the Government of Indonesia with its own funding, its own schedule and its own arrangements because the interdepartmental issues and budgetary issues related to primary school books were considerably more complex and difficult to solve than the junior secondary schools. In primary education, 243.7 million copies (120 percent of the revised target) of 70 titles (70 percent of the target) were produced and distributed to schools in the early years of the project at a 1:1 book-pupil ratio. The Department of Education estimates that about 70 percent of these books are still available, the others having disappeared or become unusable because of damage. Despite the loss of books there was progress made in the availability of textbooks at the primary level.

In junior secondary education, 60.3 million textbooks (59 percent of the revised target) and 1.4 million teachers guides (74 percent of the revised target) were produced and distributed in three annual rounds of procurement. Coverage in these first three rounds approached a 1:1 book-pupil ratio. However, many of

- 5 -

the books provided to junior secondary schools in the first round have also disappeared. The reasons the overall target of a 1:1 ratio was not met in junior secondary education were: (1) an under-estimation of the rapid growth in junior-secondary enrollments and (2) the cancellation of the fourth of the four annual rounds of procurement due to the disagreements between the Bank and Government on procurement methods to be used.

In both primary and junior secondary schools progress was made toward achieving numerical targets in the provision of the desired number of books. Despite progress toward numerical targets, the output of this subcomponent is rated partially satisfactory. This is due to the fact that the fourth round of procurement did not take place thereby ensuring that the targets in junior secondary education would not be achieved.. In addition, both primary and junior secondary schools experienced reduction in the overall numbers of books available due to loss and damage. Some of this can be attributed to the fact that the project design did not include school-level procedures to ensure that students returned textbooks at the end of each school year.

Improving Textbook Quality. The National Textbook Evaluation Committee (NTEC) was established with a professional staff of 11 drawn from senior teachers, university professors, and staff of teacher training institutes hired as consultants under the project. Based on the government-approved curriculum, private publishers submitted textbooks and teachers guides to the NTEC for review. Perhaps because books were needed in the Indonesian language, the publishers who submitted books for review were all Indonesian, with the exception of one publisher in Singapore who submitted a textbook to be used for teaching English. The NTEC reviewed the books on the basis of content, presentation, language (ease of understanding), and graphics, and then informed the publishers whether they had been approved. Books not approved initially could be re-submitted by the publishers after they had dealt with the comments of the NTEC on their earlier submissions.

One problem with this process was that the evaluation criteria and the weights given to each were not known to publishers, nor was the fact that the NTEC was willing to approve books in some subjects which covered only 80 percent of the curriculum. This meant that the process was not as transparent as it should have been aggravating deficiencies in the procurement process. Some progress was made in enhancing the quality of textbooks and teacher guides (not previously available) but the fact that the procurement process did not function meant that in the final analysis, financial considerations overrode technical ones. Therefore, the output of this subcomponent must be rated unsatisfactory.

Changing the Role of the Book Center. The role of the Book Center changed during the project period as envisioned at appraisal. It no longer produces textbooks. Instead, it has taken over the role of the NTEC (which existed only during the project period) to exercise quality control over textbooks. In addition, the Book Center is developing a book information system, eventually to be on-line, which provides information on textbooks including grade level of use, author, publisher, and printer. Consultants services from Indonesian universities were procured successfully to assist the Book Center during the project; internationally recruited technical assistance was sought but not found. Training for Book Center staff was carried out successfully, including training overseas (in Singapore, Malaysia, and Thailand). Book Center staff reported to the completion mission that they learned useful things abroad which were used to improve their own procedures. The output of this subcomponent is rated satisfactory.

Component 2. Reading Promotion. The Book Center provided training, as foreseen: a team from Jakarta visited all the provinces and held workshops in writing, editing, illustrating and publishing children’s books, targeted at interested authors, editors, illustrators, and publishers. The Book Center also produced a guidebook for schools and teachers giving suggestions and guidance on how to promote reading

- 6 -

among children, although it is not clear how widely the guidebook was distributed or what use has been made of it. The largest part of this component was the production of promotional materials, which was the responsibility of the Department of Education’s Center for Educational Technology, Communication, and Information -PusTekKom. PusTekKom produced large numbers of materials for several media, including radio, television, film, and print. Radio spots and stories were broadcast throughout Indonesia. Four series, each with 12 topics, were produced for television in each of the four years of the project. Thirty-minute films were produced for television, as were 30- to 40-second fillers. A jingle and a logo were developed as a musical introduction to the films. The jingle, "You never read it!", indicating that individuals in various life situations had missed something important by not reading, is familiar to virtually all Indonesians with access to the media. (The Project Implementation Unit estimates that about half of Indonesia’s students have access to television.) Leaflets, posters, and calendars promoting reading were sent to all schools, nutrition centers, and other village government organizations. The private sector has picked up and is using some of the messages to promote the buying of books. The output of this component is rated satisfactory.

It is impossible to measure the impact that these activities have had on reading in Indonesia. In any case, changes in a society’s reading habits are likely to occur slowly and take a generation or more. Some anecdotal evidence is available. Independent monitoring and evaluation teams working out of universities in every province have concluded that the promotional activities have increased awareness of the importance of reading, but it is not clear whether this means that the public is actually reading more. Most teachers interviewed during the completion mission said, however, that they believed their students were reading more now than students did before the project started.

Component 3. Teacher Education. PusTekKom produced 1,250 audiocassettes and an equal number of videocassettes in different junior-secondary-school subjects. These cassettes were sent to some 2,500 central schools in subdistricts throughout Indonesia, where they were used in teachers’ workshops. The output of this component is rated satisfactory.

Component 4. Monitoring and Evaluation. Monitoring and evaluation were carried out in the first instance by the junior secondary schools themselves, who sent postcards upon receipt of the books, reporting whether the books had arrived on time, at the right place, and in sufficient number. Staff from provincial Project Implementation Units made random visits to schools in their provinces to verify that books had arrived. Site visits were also made by six-member teams from universities employed with funding from the project but independent of the Department of Education. In each province, the university team visited 30 primary schools and 30 junior secondary schools, chosen at random and visited without prior warning. The university teams reported on the receipt of books and promotional materials in schools, their use, and on the tendering process in the province. Each university team did two in-depth studies, one on the primary schools and one on the secondary schools visited, each year books were procured. The provincial reports were used to prepare five national reports, one for each year of the project, with the assistance of a national consultant from the tertiary-level Teacher Training Institute of Bandung hired by the (national-level) Project Management Unit. The output of this component is rated satisfactory.

Component 5. Project Implementation. A Project Management Unit (PMU) was established at the national level and functioned effectively in terms of coordinating the project, providing guidance to the provinces, and moving ahead with project implementation. Project Implementation Units (PIUs) were created in all provinces and, with assistance from the national Project Management Unit, carried out the first three rounds of procurement. The NTEC, the Book Center, PusTekKom, and the independent monitoring and evaluation teams functioned as foreseen at appraisal and as described above. Although the national PMU, the provincial PIUs, and the NTEC appeared to be functioning effectively, an investigation

- 7 -

by the Bank's INT department has provided evidence that led the Bank to declare misprocurement on a very significant portion of the loan. The otherwise effective performance of the national PMU and the provincial PIUs is tainted by this factor, and the output of this component must therefore be rated unsatisfactory.

4.3 Net Present Value/Economic rate of return:Not applicable.

4.4 Financial rate of return:Not applicable.

4.5 Institutional development impact:The project had a substantial impact on the provinces' capacity to undertake procurement of textbooks, a skill which will become increasingly important as responsibilities for the provision of government services are increasingly decentralized from the national to the provincial, district, and local levels. However, much more needs to be done to ensure development of a comprehensive and transparent process with a system of internal controls is developed. The Book Center’s capacity to evaluate textbooks and teachers’ guides and to provide training to authors, illustrators, editors and publishers was strengthened significantly under the project. PusTekKom’s capacity to prepare promotional materials was also built up under the project. One university in each province has developed its ability to do monitoring and evaluation. Department of Education staff who worked in the Project Management Unit learned many new skills related to project implementation and reported that their expertise is sought by many others in the Department. Standard bidding documents in Indonesian for the procurement of textbooks were developed which could be used in future tendering of textbooks with financing from the Government and other sources provided there are internal controls to monitor transparency and fairness. Finally, each year, more private-sector publishers, whose main business is the publishing of textbooks, have submitted textbooks to the Department of Education for a technical review. To date, some 150 publishers, or about 40 percent of the publishers in Indonesia, have participated, however in some cases their experience has been tainted by irregularities experienced in the procurement process. The project’s institutional development impact has therefore been decidedly mixed, and for purposes of this ICR is rated modest.

5. Major Factors Affecting Implementation and Outcome

5.1 Factors outside the control of government or implementing agency:The East Asian financial crisis slowed and made more difficult the implementation of all development activities in Indonesia, including this project. The provision of books for primary education, which was financed entirely by the Government, took place before the crisis started. Thereafter, no further books were provided for primary education. Towards the end of the project, the Government found it increasingly difficult to provide counterpart funding, particularly to finance the propose acquisition of copyrights.

The devaluation of the Rupiah which accompanied the East Asian financial crisis enabled the Government to purchase more books for fewer dollars (the World Bank Loan was denominated in dollars) which suddenly bought far more Rupiahs than previously. Domestic costs rose during the crisis, but not nearly as much as the devaluation. However, the overall scope of the potential impact of the project was reduced when Government cancelled large portions of the loan due to debt concerns.

Near the end of the project, the uncertain security situation in East Nusa Tenggara (West Timor) and Maluku made project supervision impossible in those provinces and resulted in a suspension of disbursements to activities in those provinces.

- 8 -

5.2 Factors generally subject to government control:The MoNE was unwilling to provide extrabudgetary funding in the middle of 2001 to buy copyrights which could have helped to break the log jam over the fourth round of procurement. Government should have exercised greater diligence with respect to oversight of procurement and investigated complaints of irregularities more aggressively and earlier on.

5.3 Factors generally subject to implementing agency control:Following complaints at the end of the second round of tendering about the lack of transparency in the textbook evaluation process carried out by the NTEC, the Department of Education could have ensured that the criteria and weights for determining how textbooks would be selected were articulated in detail and made known to any and all publishers. This would have ensured that the process was fairer and more transparent.

Similarly, following complaints about the fairness and transparency of the third round of tendering, the MoNE could have investigated the situation and reported its finding to the appropriate government authorities and to the Bank. Its failure to do so contributed to the Bank withdrawing from the fourth round of procurement, and an investigation by the Bank’s INT department into project procurement. That investigation concluded that collusion in the pricing of textbooks led to substantially higher prices being paid for textbooks than would have been the case had the procurement process not been corrupted.

5.4 Costs and financing:The project became effective September 27, 1995 and was costed at that time at US$130.4 million. The original loan amount of US$ 130.4 million was reduced to $51.6 million after a cancellation of $36.5 million due to a debt reduction effort during the financial crisis and $44.4 million of undisbursed funds resulting from failure to complete the final rounds of procurement.

6. Sustainability

6.1 Rationale for sustainability rating:The sustainability rating is modest. The new National Education System Law (No. 20/2003) has devolved procurement of textbooks to the school level. The government is developing procurement guidelines, but has yet to actually put in place a system for monitoring the availability of textbooks and proper checks and balances on procurement. The latter might include asking schools to seek bids in sealed envelopes, public bid openings, and publication of results. Isolated schools might find some protection from local marketing monopolies if publishers were encouraged to print a "suggested retail price" on the books.

Other aspects of the project are more likely to be sustained. The Book Center continues its activities in the quality evaluation of textbooks, just as PusTekKom is continuing its reading promotion activities. Workshops for in-service training of teachers are also continuing. Overseas training as well as monitoring and evaluation in the provinces are unlikely to continue, now that the Bank financing is no longer available.

6.2 Transition arrangement to regular operations:With the transfer of the responsibility for procuring textbooks to the school level, the national Project Management Unit is being dismantled and staff absorbed elsewhere in the Department of Education. Staff of the provincial Project Implementation Units have, for the most part, been redeployed to other parts of the education offices in the provinces. The Book Center and PusTekKom are on-going parts of the MoNE and receive annual budgets. Some central government funds are earmarked for textbooks in each year’s budget although (as discussed above) they are unlikely to be adequate to supply all the needs.

- 9 -

7. Bank and Borrower Performance

Bank7.1 Lending:As discussed above under Quality at Entry, the Bank’s performance during project preparation and appraisal was unsatisfactory. Bank lending in this sector, as well as AAA work, covered all sub-sectors and provided the foundation for ongoing policy dialogue and guidance for the whole sector. Bank funding provided incremental additional funding for the challenges of improving quality of basic education and decentralization of the delivery of educational services. Appropriate resources were provided to field comprehensive teams for project development and numerous relevant studies were undertaken to ensure the quality of the project design. However, the one major flaw during this period was the arrangements for the procurement of textbooks. In retrospect the team would have benefited from the new guidelines for procurement of textbooks.

7.2 Supervision:The Bank’s supervision effort must be rated unsatisfactory, in light of the major procurement problems this project experienced. The Bank was notified, after the second round of procurement, that the technical evaluation was not fair and transparent and a supervision team modified the procedures in a effort to address this issue. These modifications were included in the third round of procurement. However, further complaints were received, following the third bidding. At that point, the Bank began to conduct a detailed review of the complaints and sought to determine whether the allegations were based on fact. This investigation showed that there were significant problems with textbook procurement during the life of the project. In retrospect, it would have been wise of the Bank to have moved more aggressively to investigate publishers’ complaints that the technical evaluation was not fair and transparent.

7.3 Overall Bank performance:Overall, the Bank’s performance is rated as unsatisfactory.

Borrower7.4 Preparation:The MoNE was closely involved in the preparation process and the studies carried out as part of project preparation. Borrower performance during preparation is therefore rated satisfactory.

7.5 Government implementation performance:The Department of Finance provided counterpart funds and budgets as required and facilitated the decentralization measures implemented under the project. As noted above, however, the MoNE should have moved forward more aggressively to investigate the allegations of procurement irregularities when they began to emerge. MoNE implementation performance must therefore be rated unsatisfactory.

7.6 Implementing Agency:Much of the Department of Education’s performance, including the Project Implementation Units, the Book Center, and PusTekKom, was satisfactory. Many aspects of project implementation were carried out competently. On the negative side, as noted above, the Department of Education also should have moved forward more aggressively to prevent misprocurement and deal with complaints about procurement from publishers as they arose. Its lack of concern over the matter (expressed in a letter to the Bank saying that procurement irregularities among the publishers was not the Department’s responsibility) led to the Bank’s withdrawal from the project. Thus, despite the positive factors, the Implementing Agency’s performance must be rated unsatisfactory.

7.7 Overall Borrower performance:

- 10 -

Overall, the Borrower’s performance is rated as unsatisfactory.

8. Lessons Learned

The Bank and Government should have been doubly careful to ensure that procurement procedures, lboth the technical evaluation of textbooks and the bidding thereafter, were fair, open, and transparent. If the Bank is involved in a future textbook procurement in Indonesia, it should insist that the technical criteria and weights for textbook evaluation be specified to the extent possible and be widely advertised. Care should be taken to ensure that there are no criteria, other than the ability to write correctly in Indonesian, which effectively eliminate interested foreign suppliers. It also seems advisable that the bidding process thereafter be divided into (i) acquisition of copyrights; and (ii) competitive bidding (with the documents in English as well as Indonesian) for printing. The Bank's new textbook procurement guidelines are a useful guide. If schools use shopping for procuring textbooks they should: (i) shop with bids in sealed envelopes and have public bid opening and (ii) make available to the public the results of the shopping process. Isolated schools might find some protection from local marketing monopolies if publishers were encouraged to print a suggested retail price on the books.

Universal school-level procedures for the initial distribution of textbooks and mechanisms to monitor land ensure that they are returned at the end of the school year should be developed and widely disseminated at the beginning of the project. These mechanisms should include (i) titles and numbers of copies of books distributed to teachers, (ii) lists of books distributed to each student, (iii) records of whether the students return the books, (iv) transparent penalties for failure to return the books (normally, the student or the parents should pay for the missing book), and (v) widespread information-sharing on these measures at the beginning of the school year, so that parents and students know what is expected.

Textbook projects should include carefully designed M&E systems that attempt to measure the impact lof new books on student learning and compare the effectiveness of various textbooks for achieving curricula objectives.

9. Partner Comments

(a) Borrower/implementing agency:From the Department of National Education:

With regard to the World Bank’s performance during supervision, its is felt that after late 1999, there was no one in the Resident Mission to help assist the MoNE with procurement and other problems.

With regard to the cancellation of the fourth round of procurement, it was felt that the World Bank made a one-sided decision in canceling the fourth round of procurement. It was believed that a solution had been agreed with the World Bank to buy copyrights of approved books and then have international competitive bidding for the printing only - and it was point out that standard bidding documents for the printing had been approved by the issuing of a no objection letter from the World Bank. It came as a surprise when the World Bank later informed MoNE that it could not proceed. The cancellation of the fourth round of bidding had a major negative impact on some 11 million junior-secondary-school students and was the reason the project did not achieve its development objectives.

MoNE would like to give its view of what happened at the time both sides were attempting to work out an alternative solution to the fourth round of bidding. In respect to the advice given by the Bank’s Task Manager (1999-2000) to split the (re)tendering process into two steps, i.e., the acquisition of the copyrights from the approved publishers and the competitive bidding of the mass printing and deliveries, MoNE’s

- 11 -

Project Management Unit took appropriate actions in close consultation with the Bank’s Task Manager and other Bank staff in preparing a retendering schedule and the two sets of bidding documents required.

The PMU sent draft documents entitled Provision of Manuscripts of Junior Secondary School Learning Materials under An Arrangement of Temporary Transfer of Publishing Rights/Copyrights. Considering the budget structure of the Loan, in which there was no specific category to allow funds to be used for payments of copyrights, the bidding documents prescribed in some GCC clauses a payment mechanism which would have avoided the revision of Loan categories, i.e., by introducing a method by which payments of copyrights were incorporated into the contract to be made for the mass printing and deliveries. The payment methods were set forth in a revised Section IV. GCC Data Sheet and Special Conditions of Contract, which was prepared at that time (a copy was given to the Bank’s Completion Mission).

While developing the bidding documents for the printing and deliveries of books, the PMU was in intensive consultation with the Bank’s Task Manager and other Bank staff, which resulted, inter alia, in (a) the alteration of the bidding proceedings, from a condition of prequalification prior to bidding to postqualification within the bidding process, (b) the package amount allowed for the bidders to bid in relation to their capacities, and (c) the breaking down of the workload into a number of packages by means of certain considerations. The most important thing to clarify here is that the PMU’s adjusted plan stuck to the agreement made during the consultations and supervision that the bidding should bluntly follow ICB rules. A revised Section II. ITB Data Sheet, which concerned the ICB/LCB question, was prepared at the time (a copy was given to the Bank’s Completion Mission).

The bidding documents which the Bank had approved were not the ones that were supposed to be used for the planned alternative bidding. The Bank objected to the documents used in the initial biddings, despite its earlier no objection. The first three rounds of bidding had run exactly according to the procedures reflected in the initial bidding documents. Although the final draft of the revised bidding documents for printing and delivery was completed, it was not eventually submitted to the Bank, because in such a situation the PMU could not possible proceed with the bidding of the copyright acquisition.

With regard to sustainability, the Government of Indonesia will in the future adopt a different approach. The Book and Reading Development Project was the last project in which the national Department of Education played a major role in the procurement and distribution of textbooks and teachers guides for junior secondary schools. This responsibility was decentralized in part to the provincial level under the project. In the future, it will be progressively decentralized to the districts (District School Committees) and then to the schools (School Boards) themselves. The provision of books for primary schools will be mainly the responsibility of the districts, which receive funds from the national Department of Internal Affairs. At the national level, the Book Center will continue to carry out its quality-control function of evaluating new textbooks submitted by publishers for review. The Department of Education team believes that future sustainability of textbook provision in Indonesia will depend on this cost-sharing: grants from the national government, funding from provincial and district governments, and funding from the schools themselves.

With regard to collusion, collusion among suppliers during bidding is not good for any country in the world. Nevertheless, the prices of the books procured under the three successful rounds of bidding were incredibly cheap: on the order of US$ 0.35 per book. Indonesia got a lot for its money, and the World Bank should be pleased that its money bought so much.

With regard to replacement of books supplied under the project, the Department of Education will not be involved in attempts to replace books which were initially supplied under the project but which have

- 12 -

since worn out or otherwise gone missing. As agreed during appraisal, publishers are expected to reprint the books which were supplied, and then the schools themselves are responsible for buying replacements. If replacement books are not found in local markets, schools have the option of writing directly to publishers and buying the books from the publishers inventories. Under other ADB- and World-Bank-assisted projects in Indonesia, block grants are made to schools for this purpose.

With regard to the evaluation of textbooks by the National Technical Evaluation Committee (NTEC), the NTEC did indeed have a team of 11 professionals (university professors and prominent experts as well as one person from the Department of Religion and one person representing private schools), but these 11 did not themselves review textbooks submitted by publishers. That job was done by technical evaluation teams, usually 9 people for each series of textbooks drawn from among senior teachers and expertise from universities and Teacher Training Institutes (IKIPs). The main team of 11 people then reviewed the results of the technical teams.

The Department of Education’s team feels that it is not true that the NTEC evaluation process was not transparent. Evaluation criteria had been given to bidders through bidding documents; however, since the publishers submitted available books already in the market for evaluation instead of preparing new books to meet the Department of Education’s requirements, the scoring system was not revealed to publishers. The Department of Education’s team feels that the evaluation was done in as fair a manner as possible, meaning that all bidders were treated in the same way. The team also pointed out that when the project began, there was not time for the publishers to go back to the authors to prepare new books. The first round of procurement began almost immediately, and therefore the publishers could but submit books already available on the market for review.

To clarify this point further, the team would like to point out that up to now, it is broadly accepted that in the handling of the evaluation of a bid, accountability governs rather than transparency. To some extent transparency takes its role as an evaluation mechanism to be applied, and this criterion is always found in normal bidding documents of any kind, but the exposition of a detailed scoring system has so far seldom been practiced.

Concerning accountability, an effort was made by the NTEC to assure how they had evaluated accountably the series of the books submitted by the bidders of the fourth round of bidding. Some time after the results of the book evaluation were released, the NTEC invited relevant bidders of all the unsuccessful series and their writers to a workshop. There the NTEC explained in details, and opened discussions on the reasons their series had failed. To our experience the authors all could accept the explanations and could agree with the NTEC on the weaknesses found in their series.

With regard to bid evaluation, and in response to question asked by the Completion Mission, the Department of Education notes that the Bank’s consultant had helped in preparing and reviewing bidding documents but was never involved in the bid evaluation process.

(b) Cofinanciers:None.

(c) Other partners (NGOs/private sector):

10. Additional Information

- 13 -

Annex 1. Key Performance Indicators/Log Frame Matrix

Monitoring IndicatorsPart A: Textbook Availability

Target Actual as of 5/31/20001. Primary school textbook production and distribution

197.5 million copies of 100 l

textbook and teachers guide titles for 6 subjects for 6 grades available in 6 years, FY1993/94-FY1998/99, at 1:1 book pupil ratioTextbooks and teachers guides l

actively used in class

243.7 million copies (120 l

percent of target) of 70 titles (70 percent of target) distributed to 27 provinces as of FY1998/99, at 1:1 book-pupil ratio

26 teacher training institutes l

independently reports as of February 2000 books generally received before school opening, 85 percent in good condition, and used in class

2. Junior Secondary School Textbook Selection

Textbooks for 8 subjects, l

teachers guides for 11 subjects evaluated and ready for procurement

Textbooks for 8 subjects (100 l

percent of target), teachers guides for 10 subjects (90 percent of target) evaluated

3. Junior Secondary School textbook production and distribution

101.5 million textbooks and 1.9 l

million teachers guides available in school

Books used actively in classl

60.3 million textbooks (50 l

percent of target) and 1.4 million teachers guides (74 percent of target) delivered. Round 4 procurement (35.2 million textbooks and 1.1 million teachers guides) was canceled.Monitoring reports books are in l

school and used heavily4. Book Center staff development to assume new role of technical support in textbook evaluation

Comparative study program: 27 l

staff months

Local degree program: 6 staff, l

96 staff-months total

Other local training (creative l

writing, proofreading): 30 staff monthsBook information system: 1 l

directory, 16 other items

0.8 staff month usedl

5 staff (83 percent of target) l

awareded 165 staff-months (170 percent of target) for graduate degree18 staff-months: 460 staff days l

(60 percent of target)

Publishing director (100 l

percent of target), 12 publications (75 percent of target) produced and distributed

Part B: Reading Promotion Targets Actual as of 5/31/20001. Training of writers, editors, publishing staff

Publishing skill in the industry l

developed: 18,900 staff days total

16,134 staff days (85 percent of l

target)

- 14 -

Workshopsl

(a) Copyright (400 staff days) (b) Children’s books (500 s/d) (c) Writing (5,400 s/d) (d) Editing (3000 s/d) (e) Illustration (3000 s/d) (f) Book design (3000 s/d)

Workshopsl

(a) 242 staff days (60%) (b) 301 staff days(60 %) (c) 5577 staff days (103%) (d) 3384 staff days (113%) (e) 2520 staff days (84 %) (f) 720 staff days (24%)

2. Promotion and distribution of promotional materials

Reading promoted regularly in the media

130 radio spots, storiesl

104 TV advertising spotsl

52 TV stories broadcastl

1,000,000 postersl

124 radio spots, stories l

produced (95 percent of target), aired on RRI

48 TV spot fillers produced (46 l

percent of total, aired on RCTI, SCTV,Anteve, TPI, Indosiar

60 TV stories produced (115 l

percent of target) aired on RCTI, TPI

1,260,000 posters and l

calendars printed and distributed (126 percent of target) to schools and public over 3 years.

Part C: Teacher Training Target Actual as of 05/31/20001. Production and distribution of teacher training materials

10 audio and 10 video l

programs on textbook use actively used by teachers in class

9 audiotape program (90 l

percent of target); 9 videotape programs (90 percent of target) produced and distributed to 3750 teacher training sites

The above indicators represent the project achievement prior to the cancelation of the fourth round of textbooks procurement.

- 15 -

Annex 2. Project Costs and Financing

Project Cost by Component (in US$ million equivalent)AppraisalEstimate

Actual/Latest Estimate

Percentage of Appraisal

Component US$ million US$ million1. Improving Textbook Availability and Quality SD Textbooks 182.40 SLTP Textbooks 107.30 Approval Process 1.10 Book Center 3.602. Reading Promotion 5.503. Teacher Education 1.204. Monitoring and Evaluation 1.205. Project Management Units 12.60

Total Baseline Cost 314.90 0.00 Physical Contingencies 11.90 Price Contingencies 28.40

Total Project Costs 355.20 0.00Total Financing Required 355.20 0.00

Note: Actual project costs were not provided based on Project Components as indicated in Table 4.l in the Staff Appraisal Report (SAR). Actual project costs were collected based on the SAR Table 4.3 Financing Plan and these figures are reported below.

Project Costs by Procurement Arrangements (Appraisal Estimate) (US$ million equivalent)

Expenditure Category ICBProcurement

NCB Method

1

Other2 N.B.F. Total Cost

1. Teaching Materials:SD Textbooks/Teachers’ guides 0.00 0.00 0.00 182.4 182.4

(0.00) (0.00) (0.00) (0.00) (0.00)SLTP Textbook/Teacher guides 108.4 25.0 10.0 143.4

(81.7) (18.8) (7.5) (108.0)

2. Educational Promotional Material

0.00 3.5 6.0/c 9.5

(2.8) (4.8) (7.6)3. Overseas training 0.00 0.00 0.30 0.00 0.30

(0.30) (0.30)4. Local Training/Workshops 0.00 0.00 1.9 0.00 1.9

(0.00) (0.00) (1.50) (0.00) (1.50)5. Consultant Services 0.00 0.00 8.8 0.00 8.8

(0.00) (0.00) (8.4) (0.00) (8.4)5. Project Management /Incremental Operating Costs

0.00(0.00)

0.00(0.00)

8.90(6.70)

0.00(0.00)

8.90(6.70)

- 16 -

Total 108.4 28.5 35.9 182.4 355.2(81.7) (21.6) (29.2) (0.00) (132.5)

Project Costs by Procurement Arrangements (Appraisal Estimate) (US$ million equivalent)

Expenditure Category ICBProcurement

NCB Method

1

Other2 N.B.F. Total Cost

1. Teaching Materials:SD Textbooks/Teachers’ guides 0.00 0.00 0.00 200.00 200.00

(0.00) (0.00) (0.00) (0.00) (0.00)SLTP Textbook/Teacher guides 59.00 0.00 0.00 59.00

(44.20) (0.00) (0.00) (44.20)

2. Educational Promotional Material

0.00 3.20 0.00 3.20

(2.60) (0.00) (2.60)3. Overseas training 0.00 0.00 0.08 0.00 0.08

(0.08) (0.08)4. Local Training/Workshops 0.00 0.00 0.50 0.00 0.50

(0.00) (0.00) (0.40) (0.00) (0.40)5. Consultant Services 0.00 0.00 3.10 0.00 3.10

(0.00) (0.00) (2.90) (0.00) (2.90)5. Project Management /Incremental Operating Costs

0.00(0.00)

0.00(0.00)

2.30(1.40)

0.00(0.00)

2.30(1.40)

Total 59.00 3.20 5.98 200.00 268.18(44.20) (2.60) (4.78) (0.00) (51.58)

Note: The Project Management Unit did not provide project costs based on procurement arrangements. Therefore, the figures shown are estimates and based on the Bank Integrated Controller's System and information from the project files.

Project Financing by Component (in US$ million equivalent)

Component Appraisal Estimate Actual/Latest EstimatePercentage of Appraisal

Bank Govt. CoF. Bank Govt. CoF. Bank Govt. CoF.1. Teaching Materials SD Textbooks 182.40 200.00 109.6 SLTP Textbooks 108.00 35.40 44.20 14.80 40.9 41.8 Educational Promotional Materials

7.60 1.90 2.60 0.60 34.2 31.6

2. Overseas Training 0.30 0.00 0.10 0.00 33.3 0.03. Local Training and Workshops

1.50 0.40 0.40 0.10 26.7 25.0

4. Consultnat services Provinces 6.80 0.00 1.90 0.00 27.9 0.0 Central MOEC 1.60 0.40 1.00 0.20 62.5 50.0

- 17 -

5. Project Management 4.40 0.00 0.60 0.00 13.6 0.06. Incremental Operating Costs

4.50 2.20 0.80 0.90 17.8 40.9

Note: The Project Management Unit did not provide overall project costs for the Government. Therefore, the figures shown are estimates and based on the Bank Integrated Controller's System and information from the project files. There were four cancellations during the project implementation period totalling US$80,889,225.09. They were as follows:Cancelled as of August 26, 1998: US$ 31.5 millionCancelled as of December 31, 1999: US$ 5.0 millionCancelled as of December 5, 2001: US$ 40.8 million (precisely $ 40,799,192.77)Cancelled as of October 18, 2002: US$ 3.6 million (precisely $ 3,590,032.32)

- 18 -

Annex 3. Economic Costs and Benefits

Not Applicable

- 19 -

Annex 4. Bank Inputs

(a) Missions:Stage of Project Cycle Performance Rating No. of Persons and Specialty

(e.g. 2 Economists, 1 FMS, etc.)Month/Year Count Specialty

ImplementationProgress

DevelopmentObjective

Identification/Preparation7/92 StudyPre-Identification

6 2 Operations Officers, 1 Reading Specialist,1 Education Researcher, 1 Book Industry Specialist, 1 Textbook Specialist

S S

11/92 Study Cont. 6 2 Operations Officers, 1 Reading Specialist, 1 Education Researcher, 1 Book Industry Specialist, 1 Textbook Specialists

S S

2/93 Identification 2 2 Operations Officers S S11/93 5 2 Operations Officers, 1

Deregulation Specialist, 1 Textbook Organization Specialist, 1 School Book Policy Specialist

S S

Appraisal/Negotiation2/95 3 2 Operations Officers, 1

Monitoring & Evaluation Specialist

S

4/95Pre-Negotiations

4 1 Operations Officer, 1 Education Specialist, 1 Operations Analyst, 1 Textbook Publishing Specialist

S

4/95Negotiations

4 1 Operations Officer, 1 Operations Analyst, 1 Lawyer, 1 Disbursement Specialist

S

Supervision

09/22/1995 4 1 Operations Officer; 1 Procurement Officer; 1 Publishing Specialist; 1 Disbursement Specialist

HS HS

05/10/1996 3 1 Operations Officer; 1 Procurement Specialist; 1 Textbook Publishing Specialist

S S

11/12/1996 3 1 Operations Officer; 1 Procurement Specialist, 1 Textbook Publishing Specialist

S S

07/30/1997 2 2 Operations Officers S S05/07/1998 2 2 Operations Officers S S11/6/1998 Mid-term

3 2 Operations Officers, 1 Textbook Publishing Specialist

S S

03/12/1999 3 2 Operations Specialist; 1 Textbook Specialist

U U

11/8/1999 2 1 Sr. Education Specialist; 1 S S

- 20 -

Operations Officer6/5/2000 3 1 Education Specialist, 2

Procurement SpecialistS S

9/22/2000 4 1 Education Specialist, 1 Sr. Procurement Specialist, 1 Procurement Specialist, 1 Implementation Specialist

S U

06/28/2002 3 1 Human Resources Specialist, 1 Implementation Specialist, 1 Program Assistant

U U

ICR6/2002 2 1 Sr. Education Specialist, 1

Procurement SpecialistsU U

Information provided comes from project files.

(b) Staff:

Stage of Project Cycle Actual/Latest EstimateNo. Staff weeks US$ ('000)

Identification/Preparation 190.0 611Appraisal/Negotiation 30.0 96Supervision 110.0 356ICR 20.0 60Total 350.0 1,123

- 21 -

Annex 5. Ratings for Achievement of Objectives/Outputs of Components(H=High, SU=Substantial, M=Modest, N=Negligible, NA=Not Applicable)

RatingMacro policies H SU M N NASector Policies H SU M N NAPhysical H SU M N NAFinancial H SU M N NAInstitutional Development H SU M N NAEnvironmental H SU M N NA

SocialPoverty Reduction H SU M N NAGender H SU M N NAOther (Please specify) H SU M N NA

Private sector development H SU M N NAPublic sector management H SU M N NAOther (Please specify) H SU M N NA

- 22 -

Annex 6. Ratings of Bank and Borrower Performance

(HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HU=Highly Unsatisfactory)

6.1 Bank performance Rating

Lending HS S U HUSupervision HS S U HUOverall HS S U HU

6.2 Borrower performance Rating

Preparation HS S U HUGovernment implementation performance HS S U HUImplementation agency performance HS S U HUOverall HS S U HU

- 23 -

Annex 7. List of Supporting Documents

Project Information Document (PID)Project Appraisal Document (PAD)Grant AgreementMemorandum of the PresidentMinutes of NegotiationsProject Status Reports (PSRs)Mission Aide-memoiresBack-to-Office ReportsProject Financial ReportsProject Correspondences

- 24 -

- 25 -