Document of The World Bank

Report No: 26216

IMPLEMENTATION COMPLETION REPORT(TF-20383; TF-21206; TF-20716; TF-21746; IDA-30500)

ON A

CREDIT

IN THE AMOUNT OF US$250 MILLION

TO THE

ISLAMIC REPUBLIC OF PAKISTAN

FOR A

SECOND SOCIAL ACTION PROGRAM PROJECT

June 25, 2003

HUMAN DEVELOPMENT SECTOR UNITSOUTH ASIA REGION

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS

(Exchange Rate Effective January 1998)

Currency Unit = PK Rupees (Rs) Rs. 1.00 = US$ 0.022US$ 1.00 = Rs. 44.0

At completion (June 30, 2002) US$ 1.00 = Rs. 62.54FISCAL YEARJuly 1 June 30

ABBREVIATIONS AND ACRONYMSADB Asian Development BankAG Auditor GeneralAJK Azad Jammu and KashmirAOPs Annual Operational PlansBHUs Basic Health UnitsCAS Country Assistance StrategyCBOs Community Based OrganizationsCFAA Country Financial Accountability AssessmentDACs Departmental Account CommitteesDAOs District Account OfficersDCA Development Credit AgreementDDOs Drawing and Disbursement OfficersDERA Drought Emergency Relief AssistanceDfID Department for International DevelopmentDHQs District Headquarter HospitalsDO Development ObjectivesDOTs Directly Observed Treatment – Short-CourseEC European CommissionEMIS Education Management Information SystemEPI Expanded Program of ImmunizationESR Education Sector ReformFATA Federally Administered Tribal AreasFDs Finance DepartmentsFM Financial ManagementFY Fiscal YearGDP Gross Domestic ProductGER Gross Enrollment RateGoP Government of PakistanGoS Government of SindhHMIS Health Management Information SystemICR Implementation Completion ReportICT Islamabad Capital TerritoryIDA International Development AssociationIMF International Monetary FundIMR Infant Mortality RateI-PRSP Interim Poverty Reduction Strategy PaperIUDs Intrauterine Device

KPP Khushal Pakistan ProgramLACI Loan Administration Change InitiativeLGO Local Government OrdinanceLHVs Lady Health VisitorsLHW Lady Health WorkersM&E Monitoring and EvaluationMCH Maternal and Child HealthMMR Maternal Mortality RateMOUs Memorandum of UnderstandingMSU Multi-Donor Support UnitMTR Mid-Term ReviewNA Northern AreasNAB National Accountability BureauNCB National Competitive BiddingNER Net Enrollment RateNFC National Finance CommissionNGO Non-Governmental OrganizationNLG Netherlands GuildersNWFP North West Frontier ProvinceO&M Operations and MaintenanceORS Oral Rehydration SaltP&D Planning and DevelopmentPA Project AgreementPAD Project Appraisal DocumentPC1 Planning Commission Proforma No. 1 (Project Document)PDP Participatory Development ProgramPFAAs Provincial Financial Accountability AssessmentsPFC Provincial Finance CommissionPHED Public Health and Engineering DepartmentPIFRA Project to Improve Financial Reporting and AuditingPIHS Pakistan Integrated Household SurveyPSR Project Status ReportPSU Primary Sampling UnitPTA Parent Teacher AssociationPWP Population Welfare ProgramQMRs Quarterly Monitoring ReportsRHCs Rural Health Centers RWSS Rural Water Supply & SanitationSACs Structural Adjustment CreditsSAP Social Action ProgramSAPP 1 Social Action Program Project 1SAPP II Second Social Action Program ProjectSBP State Bank of PakistanSC School CouncilSMC School Management CommitteeSOEs Statement of ExpendituresSWAPS Sector Wide ApproachesTA Technical AssistanceTA/DA Travel Allowance/Daily Allowance

TB Tuberculosis TGM Technical Group MeetingTHQ Tehsil Headquarter HospitalsTMA Tehsil Municipal AdministrationTPV Third Party ValidationTT Tetanus ToxoideVBFPWs Village Based Family Planning WorkersWUA Water User Associations

Vice President: Mieko NishimizuCountry Manager/Director: John W. Wall

Sector Manager/Director: Michelle Riboud Task Team Leader/Task Manager: Tahseen Sayed

PAKISTANSECOND SOCIAL ACTION PROGRAM PROJECT

CONTENTS

Page No.1. Project Data 12. Principal Performance Ratings 13. Assessment of Development Objective and Design, and of Quality at Entry 24. Achievement of Objective and Outputs 65. Major Factors Affecting Implementation and Outcome 166. Sustainability 217. Bank and Borrower Performance 228. Lessons Learned 279. Partner Comments 3010. Additional Information 33Annex 1. Key Performance Indicators/Log Frame Matrix 34Annex 2. Project Costs and Financing 42Annex 3. Economic Costs and Benefits 44Annex 4. Bank Inputs 45Annex 5. Ratings for Achievement of Objectives/Outputs of Components 49Annex 6. Ratings of Bank and Borrower Performance 50Annex 7. List of Supporting Documents 51Annex 8. Beneficiary Survey Results 53Annex 9. Stakeholder Workshop Results 54

Project ID: P037835 Project Name: SOCIAL ACTION PRG IITeam Leader: Tahseen Sayed TL Unit: SASHDICR Type: Intensive Learning Model (ILM) of ICR Report Date: June 25, 2003

1. Project Data

Name: SOCIAL ACTION PRG II L/C/TF Number: TF-20383; TF-21206; TF-20716; TF-21746; IDA-30500

Country/Department: PAKISTAN Region: South Asia Regional Office

Sector/subsector: Primary education (59%); Health (19%); General water/sanitation/flood protection sector (9%); Sub-national government administration (8%); Central government administration (5%)

Theme: Gender (P); Education for all (P); Health system performance (P); Other social protection and risk management (P)

KEY DATESOriginal Revised/Actual

PCD: 11/17/1995 Effective: 05/28/1998Appraisal: 08/29/1997 MTR: 10/01/2000Approval: 03/24/1998 Closing: 06/30/2003 06/30/2002

Borrower/Implementing Agency: GOP/Planning & Development Division; Provincial Line DepartmentsOther Partners: Asian Development Bank (ADB); Department for International Development

(DfID); European Commission (EC); and Netherlands Government

STAFF Current At AppraisalVice President: Mieko Nishimizu Mieko NishimizuCountry Director: John W. Wall Sadiq AhmedSector Manager: Michelle Riboud Ralph HarbisonTeam Leader at ICR: Tahseen Sayed Ian P. MorrisICR Primary Author: Shahid Hafeez Kardar and Task

Team

2. Principal Performance Ratings

(HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HL=Highly Likely, L=Likely, UN=Unlikely, HUN=Highly Unlikely, HU=Highly Unsatisfactory, H=High, SU=Substantial, M=Modest, N=Negligible)

Outcome: U

Sustainability: L

Institutional Development Impact: M

Bank Performance: U

Borrower Performance: U

QAG (if available) ICRQuality at Entry: U

Project at Risk at Any Time: Yes

3. Assessment of Development Objective and Design, and of Quality at Entry

3.1 Original Objective:The Second Social Action Program (SAPP-II) was designed to continue support to the Government of Pakistan’s (GoP) Social Action Program (SAP) which was started in 1992. It also built upon the successes and lessons learnt during the multi-donor supported SAPP-I (1993 – 1997). Under SAPP-II, IDA contributed $213 million, while other donors contributed US$40 million, against the GoP contribution of US$5 billion. SAPP-II was designed truly as a project to support the wider government program for social sectors. At SAPP-I’s inception, Pakistan’s level of social development was low because of uncertain political commitment to improve human development indicators, inadequate levels and composition of funding, problems of gender disparities of access to basic social services, poor quality of service delivery, weak institutional and management capacity, and issues of governance that influenced the siting of schools and staff recruitment. To overcome these factors, the GoP initiated a broad-based social sector reform program through the Social Action Program in 1992, supported by donors through the Social Action Program Project (SAPP) I.

The justification for IDA assistance was to support GoP's reform program, to harmonize donors and reduce the burden of multiple projects on Government. The Bank's current support for Sector Wide Approaches (SWAPs) is based on the same justification. A learning ICR was conducted as the lessons of SAPP-II should be instructive for the design of SWAPs in other countries.

The specific objectives of the multi-donor supported SAPP-II were to:

a) build government capacity for planning, monitoring and implementing social service programsb) increase transparency and improve governance c) increase government expenditures on basic social servicesd) promote sustainabilitye) encourage NGO and private sector participation.

Assessment of Objectives:The development objectives of the project were clear, important and appropriate to address the key constraints to delivery of social services and to improve outcomes in the social sectors. The reforms focused on institutional deficiencies that held back improvements in human development outcomes, and on the policy framework required to enhance efficiency and effectiveness in the delivery of services. These objectives were also in line with the declared priorities of Government and consistent with IDA’s Country Assistance Strategy (CAS), which emphasized the need to improve social indicators as a means to achieve poverty reduction. Moreover, the human development strategy articulated in GoP’s Interim-Poverty Reduction Strategy Paper (I-PRSP) incorporates lessons learned from the SAP experience.

These objectives were pursued under the backdrop of a very complex program that aimed to bring about change in the culture and way of thinking in the entire public service delivery system. SAP sought to bring about institutional reform in a system where internal and external accountability was poor, and which included over half a million demotivated and poorly trained teachers and basic health workers, around ten to fifteen thousand demotivated and inadequately trained education and health department officials and managers, and about thirty thousand poorly trained accounting clerks, financial officials and managers handling several million SAP-related transactions manually. By contributing about 10% of the social sector expenditures, the Project was not only attempting to influence the quality, but also the fiduciary

- 2 -

integrity of the remaining 90% expenditures. All of these objectives were aimed at improving service delivery for almost several million families and their children across the four provinces of Pakistan. The task of bringing about change and institutional reforms was gigantic to start with, and was made more challenging in an overall environment which lacked capacity and accountability mechanisms. It was being implemented during Pakistan's 'lost decade' of the 1990s.

3.2 Revised Objective: Based on the inadequate progress in achieving project objectives as noted during the Mid-Term Review (MTR), and based on a shift in Government's social service delivery strategy and decentralization of responsibilities from provincial to newly-formed local governments, agreement was reached with Government to restructure the project. With the shift in service delivery strategy to local governments, the original project design was no longer suitable for provincial level program components. After project restructuring, while the overall Development Objectives (DOs) remained the same, project support was restricted to federal health, education, and population welfare programs, which had showed relatively better outcomes and were not affected by the decentralization reform measures. The objectives were revised to include Drought Emergency Relief Assistance (DERA). A portion of the undisbursed Credit was reallocated to finance essential drought-related imports with the objective of restoring and improving the productive capacity and incomes of drought-affected households, and to strengthen Government's capacity to manage emergencies.

3.3 Original Components:The original components of the $250 million SAPP-II project included:

Component Category Cost US$M (Including Contingencies)

% of Total

Program Support (27 Subprograms)

Policy, Physical, Institution Building

$218.7 m 88%

Participatory Development Program

Policy and Other $15.9 m 6%

Coordination, M&E Project Management $9.7 m 4%TA & Institutional Capacity Development

Institution Building $5.7 m 2%

Under the Program Support Component, a sub-program was defined as a SAP activity in a Province or Federally Administered Area. Since there were three SAP sectors and four provinces – there were a total of 12 provincial sub-programs. Similarly, there were 12 sub-programs in Federal Areas and three sub-programs at the federal level, giving a total of 27 sub-programs that were to be funded under SAPP-II.

The objective of the Participatory Development Program (PDP) was to strengthen the participation of NGOs and community-based organizations (CBOs) in the delivery of social services.

The Coordination and M&E component was designed to promote monitoring and evaluation (M&E) capacity at the federal and provincial levels through the strengthening of the line departments, and to support program monitoring through the Planning & Development (P&D) Departments, independent third party monitoring (through the Auditor-General of Pakistan), and impact monitoring through the Pakistan Integrated Household Survey (PIHS) undertaken by the Federal Bureau of Statistics.

The Technical Assistance and Institutional Capacity Development component was designed to support

- 3 -

capacity development in addition to those associated with the M&E component, particularly to provide technical assistance (TA) and training to federal and provincial implementing line departments. A Multi-Donor Support Unit (MSU) was established under SAPP-I to provide technical assistance (TA) to Government. MSU, funded by several donors, was continued during SAPP-II. Additional bilateral donor funds were also available to support the TA and capacity development needs of Government.

3.4 Revised Components:Following project restructuring in June 2001, the Program Support component was restricted to the federal education, health and population welfare programs, while assistance to the provincial and federal area sector programs was withdrawn. PDP was also discontinued owing to the ineffective implementation of this component. Due to the flexibility of the disbursement framework laid out in the Development Credit Agreement (DCA), requiring annual notification of allocations, the PDP and provincial sub-program components were not deleted in the DCA. Based on the restructuring agreement with Government, these components were not financed after project restructuring. The M&E and technical assistance components were kept intact after restructuring, although with reduced allocations. At GoP's request, a new component for Drought Emergency Relief Assistance (DERA) was added. With this reallocation, the balance in the IDA Credit and the trust funds was available to finance the restructured project in FY02. At the request of Government, IDA agreed to close the project in June 2002.

3.5 Quality at Entry:Given the lessons learned from SAPP-I, the policy and institutional reform agenda and initiatives proposed to address the issues were appropriate, well-directed and consistent with the objectives of the project. However, the breadth of the program, the number of actions, innovations and new mechanisms and institutional arrangements proposed for implementation during the project period made the project design complex and formidable. From a standard project mode, the shift to implementing a program was challenging. To this extent, it was unfortunate that the concerns expressed by the provincial governments at the conclusion of SAPP-I about the complexity of a multi-sector, multi-province project were not adequately factored in the design of SAPP-II.

One of the lessons learned from SAPP-I was that ownership was tenuously held at the provincial level because provincial governments were not active participants in the project design and negotiation processes. To address this during preparation of SAPP-II, workshops at the provincial and national levels were held involving a large number civil society representatives, implementation staff and government officials. However, generating political commitment and ownership of a reform program requires consistent and continuing engagement with stakeholders in program preparation, design and implementation. Subsequent events showed that more follow-up actions were required to create awareness and build ownership to consolidate gains from the consultations and dialogue that took place prior to finalization of the design of SAPP-II.

The project was over-designed from the point of view of provincial resources, and did not sufficiently account for limited provincial capacities and time required to bring about far-reaching institutional reform and behavioral change for large-scale public institutions (i.e. in over 100,000 schools and for over half a million teachers). Given the time frame for project execution, the scope, geographic dispersion, proposed interventions, targets for operationalizing initiatives and expected outcomes with regard to institutional and governance issues were unrealistic. This was largely because of the over-estimation of the financial and institutional capacity of provincial governments to implement the reform program and the expectation that the project would provide the required capacity building support. The reforms had to be implemented through existing government institutions and procedures, and were dependent on the capabilities and jurisdictions of the various executing agencies, which possessed limited capacity. Moreover, the fiscal

- 4 -

stress experienced by the provincial governments following implementation of the 1997 National Finance Commission (NFC) Award compounded their difficulties in meeting expenditure targets under SAPP-II. The NFC is a constituted institution established every five years to determine the pool of tax and non-tax revenues, and to determine the federal and provincial shares from this divisible pool.

It was also unrealistic to expect to reverse decades of neglect of indicators of social development, re-prioritize allocations in budgets characterized by rigidities and induce institutional changes within the time-frame of the project. The large number of sub-programs did not inevitably make the project unwieldy. It was the additional requirements of five-year rolling expenditure plans; financing and procurement plans for each sub-program; process and institutional changes; new policies, instruments and innovations for service delivery improvements; and expenditure reimbursement mechanisms that made the task daunting. As a consequence, the project was unnecessarily complicated for the provincial governments with hardly any experience or capacity to meet these challenges within a tight implementation schedule. Although SAPP-II was propagating and hoping to institutionalize a bottom-up approach, the mechanism of an umbrella project tended to centralize decision-making.

Governance issues that undermined the achievement of objectives related to difficulties encountered in introducing accountability mechanisms in isolation for a few sectors without modifying the overall service delivery structure. Similarly, decentralization of powers could not be instituted in the social sectors alone in an otherwise highly centralized system of administration, in which lower level government functionaries were reluctant to exercise powers already delegated to them.

The priority emphasis placed on SAP sub-sectors meant that for provinces which were already resource constrained additional budgetary resources for these sectors could only be made available at the expense of other sub-sectors. This created intra-sectoral imbalances, e.g., between elementary and higher levels of education, which would accentuate the issue of access at the secondary level.

The performance indicators for assessing accomplishment of objectives were reasonable and related to program goals. However, there was an overload of indicators to evaluate performance and to chart the progress of the mechanisms designed to achieve the project objectives. Consequently, the linkage between project inputs and performance indicators and between objectives and outcomes was most unclear.

The design required preparation of a multitude of financial reports (including Quarterly Monitoring Reports [QMRs], Statements of Expenditures [SOEs], and quarterly financial summaries a semi-annual financial forecasts) and program financial statements. Of these, only the QMRs and the SOEs eventually were prepared. Government manually maintained the accounting system and had limited human capacity for generating such statements within the time-frame for submission. Financial statements could not be prepared as envisaged in the Memorandum of Understanding (MoU) on Financial Management and, in the end, SOEs were considered as financial statements. The manual tracking and documenting of several million manual transactions to IDA standards was humanly impossible.

SAPP-II was appraised and became effective prior to the introduction of the Bank's Loan Administration Change Initiative (LACI) guidelines. Hence, no formal financial management assessment was conducted at the appraisal stage. While an MoU on financial management was agreed, which provided a schedule and format for the dissemination of regular financial information, it could not be followed due to a lack of client ownership and capacity at the provincial levels.

A reliable system for generating accurate financial information was not in place throughout SAPP-II, largely because the design of the expenditure accounting and reporting system had not factored in the need

- 5 -

for separate SAP and non-SAP account heads. Consequently, field staff found it difficult, both conceptually and procedurally, to segregate eligible SAP expenditures. As expenditure heads for development and non-development/recurrent budgets are not the same, it made financial reporting less intelligible and consistent. As a result, the accounting system failed to provide the needed reliable data to ensure timely reimbursements and to facilitate efficient service delivery and effective monitoring and evaluation. It unduly constrained the ways and means positions of provincial governments.

Whereas difficulties were encountered by the provinces in the functioning of the expenditure disbursement mechanism, additional complications were introduced, including the requirement that SOEs be verified by the Office of the Accountant General, a condition that was out of tune with institutional capacities at the provincial and lower levels. Moreover, the incentive for line departments to prepare SOEs was weak in light of the fact that allocations had already been made, and reimbursements against eligible SOEs would flow to provincial exchequers.

Differing GoP and IDA procurement procedures led to a host of unresolved issues. The design was weak in that it was not possible to have a SAP-specific financial management and procurement system, but rather it would be an integral part of the overall system. Although the vast majority of procurement contracts were below the threshold of US$200,000 for prior review, as mentioned in the DCA, ex-post reviews carried out on a sample basis were still required. These requirements and all relevant information on IDA procurement procedures could not feasibly have been communicated to all staff dealing with government elementary schools and basic health units, as well as to the large number of executing agencies implementing the education and health programs at the district level and below. Government procedures, as specifically applicable to the program, along with other specific requirements of donors (e.g. preparation of annual financial statements, statement of expenditures and Third Party Validation exercises), should have been collated in the form of a manual/standard operating procedures, agreed upon by all the stakeholders and properly disseminated at the start of the program to all levels of staff responsible for implementation of the program.

Procurement training workshops were held, although late in project implementation. At the design stage, IDA should have conducted a detailed dialogue on NCB procedures to reach agreement on a set of NCB procedures acceptable to all. When SAPP-II was appraised, these references were not included in the legal documents. Due to this lack of prior agreement, the Auditor General's Third Party Validation (TPV) reports continued to cite non-compliance with IDA procedures and, in certain cases, with those of Government. While procurement reviews were included in the third party review of governance issues, as a part of the Auditor General's annual TPV reports, it was also important to include specific procurement ex-post reviews in the project design. This was necessary in view of the large numbers of small procurement actions below IDA's prior review threshold. These ex-post reviews should have been agreed to, incorporated in the project design and funded by the project itself.

4. Achievement of Objective and Outputs

4.1 Outcome/achievement of objective:The achievement of the objectives of the project, including an assessment of the current status, are described and rated below:

(i) Building government capacity for planning, implementation and monitoring of social service programs The achievement of this objective is rated Unsatisfactory for reasons given below:

- 6 -

SAPP-II aimed to enhance government capacity for planning and monitoring by requiring provincial line departments to prepare Annual Operational Plans (AOPs), Quarterly Monitoring Reports (QMRs) and estimated associated budgetary requirements. Planning sections of the line departments prepared AOPs and the QMRs. These institutional arrangements for planning and monitoring were supported by instruments such as the Education Management Information System (EMIS) and the Health Management Information System (HMIS).

Planning capacity showed some improvement during the SAP years, as evidenced by the ability of the line departments to prepare AOPs with limited support from outside consultants. Whereas SAPP-II introduced discipline through the QMRs, accountability structures tend to become more diffused in a program approach unlike a project in which targets are clearly defined and responsibilities can be fixed. Moreover, little headway was made in the organizational restructuring and streamlining of management systems which were required for service delivery improvements. The experience of the monitoring activities was mixed because: (a) monitoring was largely reduced to a reporting exercise, rather than an action-oriented one for the purposes of identifying timely corrective measures; and (b) the monitoring systems mostly generated quantitative data and were not sensitive to the qualitative aspects of service delivery, which was the key issue under SAPP-II.

Generally, audit reports were not submitted six months after closure of the financial year as required by the Development Credit Agreement (DCA), and these were mostly submitted during the grace period. This started to improve later in the project. The understaffed system was unable to manually prepare and submit audit reports, which had been reviewed by the line department's Developmental Account Committees [DACs], within six months of the end of each financial year.

Finally, Government implementation and monitoring capacity at all levels was constrained by critical gaps in leadership and continuity of leadership, particularly in the line departments. The absence of strong and stable departmental teams made it more difficult to implement the sector strategies and to deliver improved services. However, during the last two years of the project, effort was made to provide more stable leadership to line departments. Given such progress during the SAP years, the troubling situation over the last two years in the area of government capacity has only worsened, now exacerbated by the enormous capacity building needs at the district level.

(ii) Increasing transparency and improving governance:The achievement of this objective is rated Satisfactory for reasons given below:

There was some progress on governance issues. While some areas were not adequately addressed during the course of SAP, some elements of governance improvement measures were undertaken, were adopted widely, and have become institutionalized at various levels of government.

Government-donor coordination was instrumental in the distinct improvement of site selection for facilities and in merit-based recruitments of staff, these areas continued to be vulnerable to intervention. This was further compounded by frequent staff transfers.

The Third Party Validation (TPV) exercises conducted during SAPP-II were useful to track progress of governance reforms, particularly in the line ministries. While there was mixed acceptance of the TPV survey results by departmental officials, TPV as a concept was appreciated and third party validation began to be accepted as an appropriate instrument for certifying certain outputs. Line departments are now often using this method to verify inputs and progress. Further, in the last couple of years provincial governments have put in place measures to improve staff accountability, including systematic efforts to

- 7 -

monitor staff absenteeism. As a result, some provinces have been successful in reducing absenteeism in the education sector. The effectiveness of supervision has remained weak owing to: (i) inadequate allocations for travel allowances and per diems, and (ii) lack of administrative control of supervisors over the field staff whose performance they were required to supervise. Overall, however, measures to improve governance and transparency have been widely adopted.

(iii) Increasing government expenditures on basic social services and promoting fiscal sustainabilityThe achievement of this objective is rated Unsatisfactory for reasons given below:

GoP agreed to increase public expenditures on the social sectors from 2% of GDP to at least 2.4% during the SAPP-II period, with a government contribution of between 1.8% to 1.9% of GDP per year, with the donors providing the remaining 0.5% to 0.6%. Additionally, the protection of spending for SAP sectors was a structural benchmark under the IMF Enhanced Structural Adjustment Facility and the Extended Fund Facility for 1997-2000.

As noted in Section 5.1, during SAPP-II implementation, Pakistan experienced severe fiscal stresses which required expenditure compression to stay within IMF-guided limits for budget deficits. Nonetheless, the social sectors fared better than what might have been expected in light of the squeeze on expenditures. Defence spending declined from 5.2% of GDP in 1996-97 to 4.6% in 2000-01, compared with a fall in education-related expenditures from 1.8% of GDP to 1.7% of GDP over the same period and a leveling off of health and population spending at 0.6% of GDP. Health and population spending could be maintained throughout this period, in part, because a significant proportion of expenditure was funded by federally-run programs which were less susceptible to resource flow volatility than provincial governments following the 1997 NFC Award. As provincial fiscal crises worsened, allocations and expenditures on SAP sectors declined during the last two to three years of SAPP-II. As a result, total allocations for SAP sectors declined from 2.3% of GDP in 1997/98 to only 1.8% of GDP, with a smaller decline in expenditure from 1.7% of GDP to 1.5%.

(Percent of GDP)

1997/98 1998/99 1999/00 2000/01 2001/02Allocations Total 2.3 2.3 2.1 1.9 1.8 Foreign Aid 0.3 0.4 0.3 0.1 0.1 Govt. Contribution 2.0 1.9 1.9 1.8 1.7Expenditure Total 1.7 1.6 1.4 1.4 1.5 Foreign Aid 0.1 0.1 0.1 0.1 0.1 Govt. Contribution 1.6 1.5 1.3 1.3 1.4

SAP Allocations and Expenditures

Funds for recurrent spending (both salary and non-salary) were not commensurate with the number of facilities, particularly schools, that had been established during SAPP-I. For example, very few teachers were recruited or redeployed to make schools operational and line departments were unable to take on these recurrent obligations to ensure the financial sustainability of quality-related interventions. Although allocations for non-salary items were increased during SAPP-II, releases were usually delayed and often not in line with the allocations because of resource constraints. In fact, according to an agreement with donors, funds for approved sub-program activities were to be released to implementation agencies no later than the fifteenth day of the beginning of each quarter. Despite this, success in securing timely releases

- 8 -

continued to be poor. To achieve expenditure level targets agreed with donors, governments often hastened to spend funds released late in the fiscal year, yielding numerous audit objections.

In retrospect, projections to determine provincial capacities to sustain recurrent costs were either missing or overly optimistic. Several large health and education investment projects came to a close during SAPP-II. Provinces had to find a greater share of recurrent costs than originally envisaged, making the transition from project financing to the regular recurrent budget more difficult. Overall, social sector budgets were protected during SAP, despite tight fiscal constraints. In the last couple of years, allocations for education and health have increased with pro-poor programs under PRSP and with provincial government reform programs.

(iv) Encouraging NGO and private sector participation in the delivery of social servicesThe achievement of this objective is rated Marginally Satisfactory for reasons given below:

While the Participatory Development Program (PDP) was the major vehicle for promoting the role of civil society for improving the delivery and quality of social services, other processes were undertaken to encourage private sector and community participation in social service delivery. Over 100,000 School Management Committees (SMCs)/Parent-Teacher Associations (PTAs) were established for primary schools and Water User Associations (WUAs) were formed, indicating a trend toward increased community involvement at the facility level.

The PDP component proved to be a non-viable mechanism to foster community and NGO participation, mainly due to design problems. PDP was an initiative imposed by donors with little support at the provincial level, stemming largely from concerns about providing grants to NGOs from borrowed funds. Implementation of PDP was also hampered by delays in the selection process. In most cases, the selection process was delayed for long periods of time at the provincial level, often reaching the national level two years following the initial grant request. Complaints about the lack of transparency in the selection process added to delays at the national level, leading to a virtual stalemate in implementation of this component.

Although SMCs/PTAs and WUAs were established, progress in achieving their stated objectives was limited. This was partly because communities were not seen as key partners in the process, and partly because of an under-estimation of the time required to foster, consolidate and institutionalize community participation and to scale-up successful strategies and approaches for community mobilization. Despite the lack of progress in institutionalizing NGO and community partnerships within line departments, some expansion of partnerships with beneficiary communities did take place. These included training of PTAs, the formation of community water users’ groups and recruitment of community mobilization staff in the RWSS sector.

As a result of the intense focus of SAP on emphasizing community involvement at the grassroots level, the importance of community participation for improving service delivery is now gaining institutional acceptance. Continuing support to SMCs by provincial governments is evidence of a realization of the need for community involvement in the delivery of education services. Many government policies and programs have emerged that emphasize public-private partnership and community involvement, including the Education Sector Reform (ESR), Tawana Pakistan (a national girls nutrition program), the Khushal Pakistan Program (KPP), and micro-finance sector work involving community organizations. SAP's emphasis on NGO and community involvement has been widely accepted by Government over the last couple of years as the standard practice for most development programs. While much remains to be done to strengthen and institutionalize outreach programs through NGOs and communities, acceptance of partnership with NGOs and communities has grown. It was unfortunate, however, that donors had begun

- 9 -

to withdraw from the program by the time provincial governments began to accept the importance of community involvement for managing assets from public sector investments. While PDP did not work as planned, SAP resulted in a policy shift towards community participation and NGOs.

Overall Summary:For ordinary citizens expanded investments and active donor involvement did not fully translate into more and better quality service delivery. An important reason underlying these weak outcomes was a failure to functionalize all the facilities established during SAPP-I as a result of continued recruitment bans (resulting in around 20% of health outlets being without a female health worker, and hundreds of closed schools due to lack of teacher appointments), thus defeating the purpose of creating additional service outlets.

Outcomes were also poor owing to underutilization of service delivery outlets because of inappropriately selected sites, poor maintenance of installed infrastructure, and an inadequate number and skill mix of staff at the service centres. This was compounded by: (i) compression of expenditures to adjust for resource shortfalls, (ii) lack of timely provision of complementary non-salary inputs, as well as equipment and consumables critical to service delivery, and (iii) continued weaknesses in accountability mechanisms. Nonetheless, the upgrading and repair of physical infrastructure in the education sector improved the overall teaching and learning environment and contributed to better access.

Other factors that affected progress on project objectives included frequent transfers of key personnel and poor progress on governance related issues and persistent absenteeism of service providers. Limited financial and human resource capabilities in provincial governments also slowed reform progress, and did not attract the degree of attention required to attain project outcomes, particularly for a program this large and complex.

While there was mixed progress with some satisfactory ratings for the achievement of objectives, as outcomes did not improve to a satisfactory level, overall performance of achievement of development objectives is rated Unsatisfactory.

Despite the uneven progress on outcomes, for the first time in Pakistan’s history SAP provided a framework that brought together policy makers, managers and implementation staff of the social sectors and pushed to the forefront of public discourse the need to accelerate social development. To that end, SAPP-II had notable success in changing mindsets and in creating awareness about past neglect of social indicators, especially in the sub-sectors of elementary education and primary healthcare. Expenditures on the social sectors were accorded some degree of protection, even during periods of severe financial crisis, with a focus on the operationalization of facilities rather than merely the establishment of new infrastructure.

Other than SAPP-II’s accomplishment in changing attitudes and focusing attention on the social sectors, its positive impacts also include enhanced emphasis on improving service delivery through staff training and checks on service provider absenteeism. Heightened emphasis on the provision of quality services created an environment in which the structural framework for the local governments installed in 2001 was designed. SAP's emphasis on the need to improve service delivery at the implementation level created a rationale for the devolution of powers to local governments.

A realization also developed that the provision of additional funds is not a sufficient alone for ensuring the implementation of a policy reform agenda. Institutional restructuring and strengthening are required to buttress it. In particular, during the second year of the project, the findings of the Annual Audit Report and identification of ghost schools and teachers in a survey conducted by the Government of Punjab alerted

- 10 -

both decision makers and donors to the continuing failure to enhance expenditure efficiencies and ensure sustainable improvements in matters related to governance, accountability and transparency of operations. To address this, SAPP-II initiated contract-based, facility-specific recruitment of teachers and health sector personnel, as well as policy to induct female teachers for primary schooling.

In addition, initiatives like scholarships for girls and the provision of free textbooks for primary classes as incentives for parents to send their children to school are now being funded by provincial governments from their own resources. Another important contribution of SAPP-II was the establishment of PTAs and SMCs through which communities were involved in the functioning of government-run primary/elementary schools. Although the role of SMCs in the management of schools is somewhat peripheral in nature, their formation and wider acceptance of a community role in the running of schools appears to be a sustainable intervention under SAPP-II. Provincial governments have both continued to support these committees financially as well as empowered them to utilize these funds.

4.2 Outputs by components:Performance in outcomes by sector is discussed below.

SAP/SAPP is generally perceived to have failed in achieving its objectives in view of the poor outcomes and continued poor delivery of publicly provided social services.

Education:Outcomes for Education are rated Unsatisfactory for reasons given below:

Primary enrollment levels based on PIHS data remained relatively flat from 1998-99 to 2001-02, rising by only one percentage point, from 71% to 72%, with the gross enrollment rate (GER) for girls constant at 61%, while for boys increasing from 80% to 83%. There was, however, a narrowing of the urban-rural gap, with rural GER growing from 63% to 66%, and urban GER declining from 94% to 91%.

The net enrollment rate (NER) at the primary level for both rural areas and girls increased by one percentage point, but fell by the same margin for urban areas and boys, thus keeping the national NER at its 1998-99 level of 42%. Similarly, NER for the middle level of education remained constant at 16%, the same as in 1998/99. Subsequent analysis done by GoP shows NER to be 52%. For the purpose of this report, original data provided by the Federal Bureau of Statistics is being used.

SAPP-II introduced a number of good initiatives to improve the quality of and access to elementary education, especially for the most disadvantaged. Some important measures that were undertaken to achieve these objectives included the introduction of contract-based, facility-specific hiring of teachers, adoption of a merit-based system of recruitment to improve teacher accountability, and teacher training to promote quality education. Policy reform to adopt co-education and induct female teachers for primary schooling during this period aimed to expand female access to education. Incentives for parents to send their children to school, including free textbooks for primary students and scholarships for girls enrolled in grades six to eight were also available in some areas to address demand side aspects of low participation, poverty, and high opportunity costs.

The concepts and strategies launched by SAP have become a part of the policy debate on education, with at least two provinces continuing to fund free textbooks and stipends to students from their own resources. These interventions are increasingly accepted as promising instruments for enhancing enrollment and retention of children, especially girls. While the potential of these initiatives is increasingly recognized, provincial governments are grappling with the mechanisms and institutional arrangements for their effective

- 11 -

implementation.

Health: Outcomes for Health are rated Marginally Satisfactory for reasons given below:

There was a marked improvement in core health indicators during the SAPP-II period. The infant mortality rate (IMR) declined from 89 per 1000 births in 1998-99 to 82 per 1000 births in 2001-02, with a large urban-rural gap of 23 percentage points, reduced from 27 points in 1995/96. The percentage of children 12-23 months who had been fully immunized improved slowly, increasing from 49% to 53% with a small male/female gap. Immunization coverage of children increased in all provinces, except Balochistan, where coverage fell from 34% in 1998/99 to 24% in 2000/01.

Antenatal care utilization grew modestly, increasing from 31% in 1998/99 to 35% in 2001/02, with a persistent urban-rural differential (63%/26%). An increase in the use of antenatal care were observed across all provinces. However, there was no change in post-natal care consultations. The proportion of women who received at least one Tetanus Toxoid (TT) vaccination increased from 39% in 1998/99 to 46% in 2001/02. However, only 38% of pregnant women received two doses of TT vaccine during their last pregnancy. The proportion of women assisted by skilled birth attendants during their last pregnancy increased modestly from 19% in 1998/99 to 24% in 2001/02. The 2003 Health Management Information System (HMIS) report indicates a modest increase in the utilization of first level care facilities from an average of 21 patients to 28 patients per facility per day. The 2001/02 Pakistan Integrated Household Survey (PIHS) assessment of first level care health facilities reported all Rural Health Centers (RHCs) (n=17), and 92% of the (87 out of 94) Basic Health Units (BHUs) had female health staff.

As part of the programmatic policies, there was an increase in resource availability for the Lady Health Workers (LHWs) program, immunization, HIV/AIDS control and TB Control. The increased allocations were used to expand coverage for the LHWs program, ensuring regular availability of vaccines, expanded health education activities of the HIV/AIDS program, and initiation of the TB control program using the Directly Observed Treatment Short-Course (DOTS) strategy. There was also expansion of family planning services through the health sector, although this was constrained by a lack of regular availability of contraceptives. Improvements in the areas of malaria control and nutrition were limited.

Programmatic expansion resulted in improved program outputs. LHWs program coverage was expanded from 25% to 50% of the population, with 62% of rural household having access to basic primary health care and family planning services through LHWs. Evaluation of the LHWs program indicates that it has had a significant impact in areas with improved access to basic services. Health outcomes, such as the proportion of mothers who receive Tetanus Toxoide (TT) vaccinations, births attended by skilled birth attendants and contraceptive use were considerably better in areas served by LHWs. Impact assessment of the information education campaign of the HIV/AIDS program in 2001, in comparison to 1996 data, indicates a major improvement in people’s knowledge of HIV transmission and prevention measures, with and an increase from 4% to 75%. Although the TB DOTS program was launched late in SAPP-II, population coverage rose to 45% from 8%.

These advances indicate a shift away from tertiary health care in favor of preventive, primary and secondary health care. The policy shift towards preventive health care is still being pursued in the new GOP’s 2001 Health Policy, which continues to focus on strengthening preventive health services, including immunization, communicable diseases, such as malaria, TB, and HIV/AIDS, maternal and child health,

- 12 -

nutrition and family planning, and improving the quality of health care at the tehsil and district headquarter hospitals.

Health outcomes and health care utilization continue to improve slowly. Improvement in indicators for maternal health care was slow because change in the quality of related services was, at best, marginal. Whatever positive results there have been for maternal care, can be partly attributed to an improved availability of female health care providers, including LHWs. Despite higher investments, however improvement in the availability of services by skilled birth attendants was nominal. It is misleading to attribute the entire improvement in health indicators to the effectiveness of sub-program interventions, as these indicators are also strongly influenced by factors such as educational levels, fertility rates, availability of clean drinking water, etc.

Rural Water Supply and Sanitation (RWSS):Outcomes for RWSS are rated as Unsatisfactory for reasons given below:

The rural water supply sector did not undergo any significant change over the period 1998-99 to 2001-02. There was no increase in the percentage of households relying on piped water inside the house, although there was a decrease in reliance on open sources of drinking water, with the use of hand and motor pumps having increased from 65% to 70%.

There was hardly any change in sanitation practices in rural areas. The percentage of households relying on covered and underground drains remained at a low 1%, while there was only modest improvement in access to open drains, from 31% in 1998-99 to 33% in 2001-02. The availability of toilet facilities also improved marginally. This facility was available to 37% of the rural population in 1998-99, which increased to 41% in 2001-02.

SAPP-II improved consistency in the policy environment for the RWSS sector. The adoption of a uniform policy at the outset ensured that all investments in rural water supply and sanitation, regardless of the source of financing, were based on community organized schemes -- a fundamental departure from the supply-driven approach prior to SAP. A moratorium continued to apply to the installation of new schemes not designed using these principles, with the exception of the KPP schemes started in 2000. The effective translation of a uniform policy into better access to clean potable water in rural areas was unclear.

There have been prevalent concerns that many water supply and drainage schemes ostensibly transferred to communities, particularly mechanized drainage schemes, would eventually be abandoned and fall into disrepair. The inability of provincial governments to rehabilitate schemes to deliver service levels acceptable to communities impeded their transfer. In some cases of successful transfers, WUAs were formed on the basis of official directives in the absence of substantive preparation and genuine participation by communities. Neither situation bodes well for the future sustainability of functioning water supply schemes.

Population Welfare:Outcomes for Population Welfare are rated as Satisfactory for reasons given below:

The downward trend in fertility rates noted after SAPP-I was maintained during SAPP-II, with a decline from 5.4 in 1996/97 to 4.8 in 2000/01. There was a gradual increase in awareness and use of family planning methods during SAPP-II. Contraceptive use among married women increased from 93% to 96%. The number of women currently using any family planning method increased from 24% in 1996/97 to 28% in 2000/01. Contraceptive use has been increasing, but it still remains very low compared to other

- 13 -

countries in the region. Knowledge of family planning also improved in rural areas, from 91% to 95%. Availability and access was improved through diversification of family planning sources through health facilities (about 50% of health facilities provide family planning services), expansion of community-based workers, and expansion of social marketing thorough private outlets and NGOs. The availability of contraceptives in the population welfare centers was maintained. However, availability of adequate supply levels in health facilities deteriorated as the Ministries of Population Welfare and Health failed to resolve policy differences on the pricing of contraceptives and procedural requirements to issuing sales receipts to replenish supplies.

There was little progress in implementing a broader reproductive health approach, despite development of a National Reproductive Health agenda, through a joint effort by the Ministry of Health and Population Welfare.

The most effective interventions were arguably those which brought about improvements in knowledge dissemination to promote behavior change, with endorsement to permit method-specific messages through mass media (except for those pertaining to condoms use). Increased availability and access to family planning services through the LHW and Village Based Family Planning Workers (VBFPWs) programs, and expanding social marketing interventions made services and supplies more accessible overall.

Participatory Development ProgramWhile NGO and community participation in social sector service delivery was not increased effectively through PDP implementation, the emphasis on increasing the involvement of NGOs and communities has been accepted as standard in the policy frameworks (ESR, PRSP, etc.) of most major social sector programs (see 4.1 above).

Coordination and Monitoring & EvaluationThe strengthening of line departments and P&D departments, independent third party monitoring through the Office of the Auditor General, and impact monitoring through PIHS were key elements of the monitoring & evaluation component. The TPV as an instrument to monitor governance issues was found to be extremely valuable. The PIHS was very useful in monitoring of output and outcome data, and also served as a major source of information and evaluation by donors and government at MTR. Line departments' capacity for monitoring and evaluation was not developed extensively, although planning tools (Annual Operational Plans) were introduced and made a regular part of the SAPP-II requirement. Monitoring through quarterly monitoring reports was carried out regularly during SAPP-II. These mechanisms were not institutionalized, however, and the building of capacities to plan, implement and monitor remains as a major challenge for most line departments.

TA & Institutional Capacity DevelopmentTechnical assistance was designed to build the capacity of line and P&D departments, mainly through the Imprest Accounts mechanism. This mechanism did not work, owing to delayed approvals of internal government procedures for utilization of Imprest Accounts. These accounts became operational towards the end of the Project. As a result, funds for TA went largely unutilized. Additional TA was provided through the Multi-Donor Support Unit (MSU), which managed to provide assistance to line departments and P&D departments on specific sectoral issues. The MSU was found by line and P&D departments to be quite useful, although new capacities were not always successfully institutionalized.

4.3 Net Present Value/Economic rate of return:Not applicable

- 14 -

4.4 Financial rate of return:Not applicable

4.5 Institutional development impact:The federal and provincial SAP Secretariats were responsible for the coordination of SAP activities linked to planning missions, handling administration, financing issues and monitoring. With the help of the MSU, the personnel of these units were trained to carry out their duties, although capacity building gains were frequently lost to staff transfers. The monitoring of actual program implementation by various line departments was weak.

The capacity of the new organizations such as teaching institutions, district health management teams, SMCs and WUAs, as well as the human capital for developing policy and for planning new instruments and program interventions (particularly in health sector) were strengthened through a variety of parallel initiatives funded through sector investment projects. However, some of the benefits of training to realize project objectives were lost with the re-deployment and transfer of trained staff.

Limited progress was made in strengthening capacity and creating minimum critical mass in terms of the number of personnel, their competence and skill composition, and in supporting the infrastructure required to manage systems. The necessary financial management staff were recruited only after significant delays. At MTR, there was realization of the need to review procedures for the use of Imprest Accounts to enable timely and flexible use of funds for capacity building and monitoring and evaluation. While planning, monitoring and evaluation capacities at the provincial level improved somewhat (as demonstrated in the preparation of AOPs and, in some cases, the formulation of five-year rolling expenditure plans), they remained particularly weak at the district level.

The continued lack of institutional capacity, which was a major handicap in the implementation of SAPP-II, was partly because projects for capacity building and technical assistance under SAPP-II were not processed by the provinces in time, and only a fraction were approved. This unduly delayed the capacity development initiative under SAPP-II, which accounts for the fact that the majority of capacity building activities took place towards the end of the project. This affected, in particular, the availability of trained staff at the district level. For example, training for Drawing and Disbursing Officers (DDOs) in financial management was carried out in the concluding phase of the project rather than at the beginning, as it should have been, resulting in a less-than-anticipated impact on improving staff's skill and experience. Part of the problem in building capacity was that, other than policy reforms to improve service delivery, some management and institutional changes also required additional funding which provincial governments could not provide.

Building ownership for reforms, and dissemination of SAP objectives and strategies to a wider government and non-government audience should have been an important part of the program. As these were not part of the mainstream program, there were negligible efforts to communicate the program's objectives and strategies. The design of the program might have benefited from the incorporation of a communications strategy, particularly for building ownership of the reforms prior to implementation. Such a communications strategy would have helped to raise awareness and reduce the effects of institutional memory loss stemming from constant staff turn over in key SAP-related departments. One donor agreed to support development of a SAP communications strategy and to support awareness campaigns. However, this was both a late and abortive entry which never took off as it could not be integrated into the mainstream program.

The rating for institutional development impact is Modest. While there were problems in building

- 15 -

institutional capacities, particularly for planning and monitoring, SAPP-II gave rise to the foundations for policy reforms and ownership of them. Community and NGO participation, devolved social service delivery, and improvements in governance and transparency are three critical areas for social sector programs which have become increasingly institutionalized over the past couple of years.

5. Major Factors Affecting Implementation and Outcome

5.1 Factors outside the control of government or implementing agency:A major constraint to the achievement of SAPP-II objectives, particularly with respect to governance and program management issues was that the environment and the overall political and macro economic situation that existed at the program design stage changed continuously during implementation. This period was characterized by political instability, economic crisis, a severe drought, austerity measures stemming from a resource squeeze and accountability drives launched in 1999/2000. The project became effective shortly after Pakistan’s nuclear test in May 1998, which resulted in sanctions against the country. Provinces were faced with a severe financial crunch owing to a shortfall in revenue receipts (70% of the difference between actual and projected revenues due to a decline in the tax ratio, and 30% due to a lower than expected GDP) and lower than expected fiscal transfers from the divisible pool. Under the 1997 NFC Award, resource flows to the provinces during the SAPP-II period were Rs.192 billion less than what they would have been had the 1990 NFC formula been retained. The provincial governments also encountered difficulties in financing expenditures upfront due to the GoP's fiscal deficit targets.

In addition, the proportion of expenditures eligible for reimbursement were agreed annually, thus creating problems for provincial finance departments to understand the financing arrangements. They were unable to factor in estimates of potential inflows on the recurrent side of the budget from donor reimbursements as the ratios were agreed and finalized much later in the year. However, provincial line departments admitted that the project had served as a useful instrument for protecting expenditures on the social sectors, even in such adverse circumstances, and that without donor support allocations for SAP sectors may have been significantly lower.

There was a funding hiatus between the last disbursement under SAPP-I and the first disbursement under SAPP-II. Credit flows from social sector projects were tapering off and the volume of retroactive financing that was available as a replacement was inadequate. This gap, and accompanying uncertainty, demotivated the finance departments, especially since they were hard pressed to provide upfront funding for on-going activities, and meet the fiscal deficit targets set by GoP. One of the reasons for a lack of provincial interest in SAPP-II was provinces' belief that donor funding was inadequate to address the real resource requirements. The donor reimbursement ratios, ranging between 6% to 8% were viewed as modest, especially in view of the conditionalities that were attached to the funding. However, while the level of external financing was perceived to have fallen short of expectation, provincial governments never spent what they had committed to, particularly because they were in a fiscal bind and were unable to finance expected levels of social sector expenditures.

Progress in achieving the desired outcomes in the social sectors was adversely affected by socio-cultural factors whose influence cannot be reversed, but rather requires continuous and sustainable interventions over a long period of time. For instance, although awareness of contraceptives among married women in the age group 15 to 49 increased sharply from 38% in 1991-92 to 96% in 2001-02, there was a lower proportional increase in actual use of such methods and instruments, from 24% to only 28% (Pakistan Reproductive Health and Family Planning Survey). The gap between knowledge and practice/use continues to be wide because of traditional socio-cultural resistance to family planning, and inadequate number of outlets and female workers, and lack of female education. However, the utilization of family planning

- 16 -

services improved in urban centres because of the demand created through social marketing efforts.

5.2 Factors generally subject to government control:Government ownership of SAPP-II and its commitment to the objectives was built slowly, but remained uncertain and generally weak throughout the duration of the project. In addition, formal donor supervision, oversight and coordination arrangements contributed to the development of a perception that SAPP-II was donor-driven, which further undermined efforts directed at creating domestic ownership of the program. Resultantly, SAPP-II was often perceived as being donor driven with uneven awareness of its objectives and proposed prescriptions at the provincial and lower tiers of government.

Whereas the importance of sensitizing the political leadership was apparent, there was a lack of a concerted effort to strengthen and expand this awareness. Although policy dialogue played an instrumental part in raising awareness of the need to attend to the low social indicators, it was largely confined to the bureaucracy. Similarly, awareness of the program at the field level was restricted to the provision of funding, with implementation staff having virtually no knowledge of the details of the program and its objectives. Nevertheless, ownership improved in the last two years of the project with the establishment of the Social Sector Committee for Cabinet under the stewardship of the Federal Minister for Education, with the participation of provincial social sector ministers. This resulted in social sector issues being brought to the forefront of policy discussions, although commitment to the reform agenda remained stronger at the federal level. The ESR was framed under this context of ownership of the reforms.

5.3 Factors generally subject to implementing agency control:The key factors that had a substantial adverse impact on implementation of SAPP-II and the functionalization of service outlets constructed during SAPP-I were not sector-specific, but persistent and common across sectors and geographical divisions of the country. Although the Government agreed that recruitment bans would not extend to SAP sectors, in practice these restrictions were applied indiscriminately, with the social sectors not exempt from their reach.

A lack of continuity of key staff negatively affected the level of commitment, the quality of leadership, institutional memory and understanding of the program, making it difficult to hold anyone accountable, thus disrupting reform efforts and compromising project implementation. The inability to ensure meaningful community participation in service delivery, continuing governance issues, such as lack of credible and effective arrangements to check staff absenteeism, and ineffective supervision and monitoring systems impeded sustainable improvements in service delivery.

5.4 Costs and financing:The project appraisal document estimated the five-year cost of the program at over US$10 billion. Out of this US$8 billion was estimated as Government financing and US$610 million (including ADB) was committed through SAPP-II donor financing. In addition, approximately US$1.4 billion was forecast as additional donor support, including disbursements from ongoing donor assisted projects, new project assistance, and additional mobilization of donor funds. By the end of the program period, Government expenditure on SAP was about 61% of appraisal estimates, and SAPP donor financing amounted to approximately 66% of the committed amount. The financing gap was also due to a lack of new large donor project assistance, estimated in the total program costs during appraisal, compounded by the fact that several large ongoing donor assisted projects had closed by 2000.

SAPP-II project financing mechanism-SOEs, financial management issues, disbursements and procurement:

There were two modes of funding, the SOE based reimbursement mechanism and Imprest Accounts for

- 17 -

capacity building and monitoring activities.

Donors provided finances retroactively through reimbursement of expenditures on activities that complied with approved operational plans (there were hundreds of thousand transactions annually). Funding mechanisms were designed to provide additional resources and ease such disbursements, thereby avoiding problems of lack of counterpart funds encountered under project funding.

Financial management of the program was weak due to a lack of proper understanding, non-compliance and delays in the implementation of the program arrangements for financial management, and the organization of the financial management system around weak internal control mechanisms. The design envisioned the use of adequately skilled government staff and was based on Governments’ mainstream systems for accounting, reporting and auditing. However, staff were either appointed late (and then subjected to frequent transfers) or not appointed at all and often lacked an appropriate skill set. A lack of proper accounting staff at the Provincial levels resulted in the absence of any analysis of financial management issues, or the development and implementation of corrective actions. Consequently, the requirements of the program such as preparation of annual financial statements, and SOEs were not properly disseminated.

The set of sub-programs that would qualify for reimbursement were agreed each year, and the understanding reached could not be fully communicated to lower levels on a timely basis. Rules, procedures and instructions for using Imprest Accounts had not been prepared in advance and therefore these accounts become functional only in the latter part of the project.

Additional specific points relating to financial management include the following:

Without the core issue of improved and timely recording of expenditures having been addressed, procedural mechanisms for claiming reimbursements through SOEs complicated by a requirement that only the Accountant General’s Office verify expenditures and expectation that departmental staff, with limited exposure to financial systems, decompose them into salary and non-salary components.

The main problems with SOEs related to: (a) the requirement that line departments reconcile SOEs with the records of the Accountant General’s office; (b) frequent revisions in their formats; and (c) difficulties in obtaining monthly expenditure statements from hundreds of District Account Officers (DAOs). This resulted in staff responsible for project implementation and technical managers spending a disproportionate amount of time preparing SOEs, ensuring timely reconciliations and settling audit queries. Financial management and capacity building through the recruitment of staff was severely delayed and took place only toward the end of the program.

An unfortunate outcome of SAPP-II was that it became a victim of the audit procedures built into the expenditure reimbursement mechanism. Part of the problem was that the audits were conducted in a traditional manner. Staff were bound by prescribed classifications for the categorization of audit observations. Resultantly, procedural lapses and misclassifications of expenditures were, at times, categorized as irregularities and misappropriations. These audit report issues made administrators extremely cautious in incurring expenditures, especially on new initiatives and on non-salary components.

The audit report submitted by the Auditor General (AG) of Pakistan for FY 1997/98 and FY 1998/99 identified a very large number of audit observations pointing towards major irregularities and misappropriations. In the view of Government, these reports were hastily submitted to donors without conducting proper Departmental Account Committee (DAC) meetings. The audit procedures are that draft

- 18 -

audit reports are discussed with line departments, and finalized on the basis of the review. Draft reports were sent to donors. Due to the findings of these reports, and as a result of private complaints received by the Bank, the Bank’s Department of Institutional Integrity conducted an Integrity Assurance Mission to Pakistan during April/May 2001. There was an apparent disconnect between the AG's office estimation of ineligible expenditures, and the Mission's estimation. The Mission placed these observations well above the 1% estimated by the Government, the Mission’s findings and recommendations were communicated to the Government. The Government responded to the findings at two levels. First, it started system-wide governance and financial management reforms, and second, has taken the necessary legal, disciplinary and other actions to address specific identified cases. The Government has indicated that the quality of the audit reports was affected by a lack of required audit follow-up with Departmental Account Committees (DACs), and inability of the audit staff to differentiate between a departure from rules and procedures and proven fraud cases.

SAPP was the first Bank project in Pakistan which financed a percentage of all government program expenditures in the basic social services. Therefore, the issues which emerged during implementation had wide raging impact, and actions have been initiated at various levels by both Bank and GOP to address the systemic financial management issues which emerged during SAPP-II implementation.

Actions by the Bank:From 1999 onwards, the Bank included a full time financial management specialist in the SAPP team, and a separate mission on financial management issues was conducted in September 2000 which presented specific recommendations to line departments and GoP for addressing systemic issues as well as those related to expenditure reporting and verification.

The Bank also twice withheld disbursements (in 2001 and 2002) pending satisfactory and credible resolution of financial management and audit issues. Disbursements were subsequently authorized based on submission of verified Statements of Expenditure by Provincial Accountant Generals, submission of SOE audit opinions and unqualified audit reports. Disbursements were made based on expenditures reported by line departments and the Accountant General, and where there was a discrepancy between the two figures, the Bank used the lower expenditure number for disbursements. To ensure that disbursements were not financing ineligible expenditures, the Bank initiated recoveries by deducting the identified ineligible expenditures from the overall expenditures submitted for reimbursements, and recoveries were made for the full program period.

As weak public financial management is endemic throughout the public sector in Pakistan. The Bank supported GOP’s reforms under the Project to Improve Financial Reporting and Auditing (PIFRA) to strengthen accounting and audit functions, and build public sector capacity. In addition, a Country Financial Accountability Assessment (CFAA) was conducted, and two provincial PFAAs (Provincial Financial Accountability Assessments) are underway. The Bank has also negotiated major financial management reforms as a part of the two Provincial Structural Adjustment Credits (SACs) in Sindh and NWFP, and two SACs for the GOP.

Actions by Borrower:The Ministry of Finance initiated an intensive effort with the AG’s office and all concerned SAP agencies to conduct the pending DAC meetings to discuss the audit reports and take remedial actions. The Accountant General’s office was specifically asked to verify all expenditures submitted through SOEs, with Auditor General confirming the authenticity of expenditures through submission of SOE audit opinion. The Auditor General established two task forces for reviewing audit observations pertaining to SAP and other foreign-aided-projects. Detailed reviews were conducted on each audit observation along with

- 19 -

supporting documentation. The subsequent annual audit reports contained less significant audit observations pertaining to misappropriation and embezzlement. The findings of the audit reports were also brought to the attention of the external accountability agency, the National Accountability Bureau (NAB), which was established by the military government soon after it assumed power in October 1999. Army monitoring teams were established to specifically take action again misuse of funds and authority. NAB conducted detailed investigations based on the information provided by the AG. Specific examples of action taken include initiation of disciplinary cases against more than 4000 teachers, launching investigations against at least fifty high profile cases, and conviction of three ministers who were disqualified from holding public offices due to cases of illegal teacher appointments and misappropriation of resources under a donor-assisted program.

More recently, a National Anti-Corruption Strategy has been developed with the assistance of the Department for International Development (DfID). This strategy has been approved by the President and Prime Minister. While these actions have been taken, significant work is still needed to build implementing line departments’ capacities, and to institute strong internal controls at all levels of government, and especially in the district governments which are now directly responsible for social service delivery.

Some systemic financial management improvements which have come about after the SAPP-II experience include: (i) regular reconciliation, consolidation and timely reporting of expenditure data, to expedite appropriate financial reconciliations and setting of audit queries; (ii) timely follow up on audit observations; (iii) setting up of internal audit units in each of the departments, as recommended by the AG, as well as the Bank; (iv) appointment of properly skilled accounting staff in the line departments; and (v) development of an FM manual to help provide important guidelines and procedures, including rules, procedures and instructions for using Imprest Accounts.

Procurement:Inadequate procurement capacity of implementing agencies to diligently follow IDA’s procurement procedures affected the overall quality and timeliness of procurement under the project. The Bank conducted numerous procurement training workshops in which many project staff participated. Nonetheless, the dispersed nature of the project and the large number of implementing agencies and staff made wider interactions difficult. Additionally, the numbers involved several hundreds of procurements. Capacity building through engagement of procurement consultants in each province was conceived in the Project Appraisal Document (PAD), but did not materialize due to administrative delays and bottlenecks. Most procurement, including all civil works, involved small contracts over widely dispersed geographical areas, and was below the Bank's prior review threshold. Large procurement operations were confined to pharmaceuticals and contraceptives which were procured through UN agencies, and large consultant contracts were to be based on the Bank's policy for selection of consultants.

In the procurement of civil works, in particular, generic deviations were observed between Government’s procurement procedures and practices, as followed by the implementing agencies, and those of IDA. These included the Government’s procedure to restrict bidding only to contractors pre-registered with government departments, and the use of a composite schedule of rates instead of item rates for inviting bids and awarding contracts. While improvements for National Competitive Bidding (NCB) procedures were specified in the PAD, these were not explicitly stated in the Development Credit Agreement (DCA) as, at the time of SAPP-II appraisal, it was not the practice to make such references in IDA legal documents. These improvements were legally incorporated at a later stage in the project, during Project restructuring in 2001. In more recent IDA-financed operations, procurement capacity strengthening measures, including the hiring of procurement consultants where required, are being explicitly agreed upfront during project

- 20 -

preparation. The required NCB improvements are now being clearly specified in the legal agreements to eliminate any ambiguities, and to facilitate remedial actions by the IDA in event of serious deviations.

5. Major Factors Affecting Implementation and Outcome

5.1 Factors outside the control of government or implementing agency:A major constraint to the achievement of SAPP-II objectives, particularly with respect to governance and program management issues was that the environment and the overall political and macro economic situation that existed at the program design stage changed continuously during implementation. This period was characterized by political instability, economic crisis, a severe drought, austerity measures stemming from a resource squeeze and accountability drives launched in 1999/2000. The project became effective shortly after Pakistan’s nuclear test in May 1998, which resulted in sanctions against the country. Provinces were faced with a severe financial crunch owing to a shortfall in revenue receipts (70% of the difference between actual and projected revenues due to a decline in the tax ratio, and 30% due to a lower than expected GDP) and lower than expected fiscal transfers from the divisible pool. Under the 1997 NFC Award, resource flows to the provinces during the SAPP-II period were Rs.192 billion less than what they would have been had the 1990 NFC formula been retained. The provincial governments also encountered difficulties in financing expenditures upfront due to the GoP's fiscal deficit targets.

In addition, the proportion of expenditures eligible for reimbursement were agreed annually, thus creating problems for provincial finance departments to understand the financing arrangements. They were unable to factor in estimates of potential inflows on the recurrent side of the budget from donor reimbursements as the ratios were agreed and finalized much later in the year. However, provincial line departments admitted that the project had served as a useful instrument for protecting expenditures on the social sectors, even in such adverse circumstances, and that without donor support allocations for SAP sectors may have been significantly lower.

There was a funding hiatus between the last disbursement under SAPP-I and the first disbursement under SAPP-II. Credit flows from social sector projects were tapering off and the volume of retroactive financing that was available as a replacement was inadequate. This gap, and accompanying uncertainty, demotivated the finance departments, especially since they were hard pressed to provide upfront funding for on-going activities, and meet the fiscal deficit targets set by GOP. One of the reasons for a lack of provincial interest in SAPP-II was provinces' belief that donor funding was inadequate to address the real resource requirements. The donor reimbursement ratios, ranging between 6% to 8% were viewed as modest, especially in view of the conditionalities that were attached to the funding. However, while the level of external financing was perceived to have fallen short of expectation, provincial governments never spent what they had committed to, particularly because they were in a fiscal bind and were unable to finance expected levels of social sector expenditures.

Progress in achieving the desired outcomes in the social sectors was adversely affected by socio-cultural factors whose influence cannot be reversed, but rather requires continuous and sustainable interventions over a long period of time. For instance, although awareness of contraceptives among married women in the age group 15 to 49 increased sharply from 38% in 1991-92 to 96% in 2001-02, there was a lower proportional increase in actual use of such methods and instruments, from 24% to only 28% (Pakistan Reproductive Health and Family Planning Survey). The gap between knowledge and practice/use continues to be wide because of traditional socio-cultural resistance to family planning, and inadequate number of outlets and female workers, and lack of female education. However, the utilization of family planning services improved in urban centres because of the demand created through social marketing efforts.

- 21 -

5.2 Factors generally subject to government control:Government ownership of SAPP-II and its commitment to the objectives was built slowly, but remained uncertain and generally weak throughout the duration of the project. In addition, formal donor supervision, oversight and coordination arrangements contributed to the development of a perception that SAPP-II was donor-driven, which further undermined efforts directed at creating domestic ownership of the program. Resultantly, SAPP-II was often perceived as being donor driven with uneven awareness of its objectives and proposed prescriptions at the provincial and lower tiers of government.

Whereas the importance of sensitizing the political leadership was apparent, there was a lack of a concerted effort to strengthen and expand this awareness. Although policy dialogue played an instrumental part in raising awareness of the need to attend to the low social indicators, it was largely confined to the bureaucracy. Similarly, awareness of the program at the field level was restricted to the provision of funding, with implementation staff having virtually no knowledge of the details of the program and its objectives. Nevertheless, ownership improved in the last two years of the project with the establishment of the Social Sector Committee for Cabinet under the stewardship of the Federal Minister for Education, with the participation of provincial social sector ministers. This resulted in social sector issues being brought to the forefront of policy discussions, although commitment to the reform agenda remained stronger at the federal level. The ESR was framed under this context of ownership of the reforms.

5.3 Factors generally subject to implementing agency control:The key factors that had a substantial adverse impact on implementation of SAPP-II and the functionalization of service outlets constructed during SAPP-I were not sector-specific, but persistent and common across sectors and geographical divisions of the country. Although the Government agreed that recruitment bans would not extend to SAP sectors, in practice these restrictions were applied indiscriminately, with the social sectors not exempt from their reach.

A lack of continuity of key staff negatively affected the level of commitment, the quality of leadership, institutional memory and understanding of the program, making it difficult to hold anyone accountable, thus disrupting reform efforts and compromising project implementation. The inability to ensure meaningful community participation in service delivery, continuing governance issues, such as lack of credible and effective arrangements to check staff absenteeism, and ineffective supervision and monitoring systems impeded sustainable improvements in service delivery.

5.4 Costs and financing:The project appraisal document estimated the five-year cost of the program at over US$10 billion. Out of this US$8 billion was estimated as Government financing and US$610 million (including ADB) was committed through SAPP-II donor financing. In addition, approximately US$1.4 billion was forecast as additional donor support, including disbursements from ongoing donor assisted projects, new project assistance, and additional mobilization of donor funds. By the end of the program period, Government expenditure on SAP was about 61% of appraisal estimates, and SAPP donor financing amounted to approximately 66% of the committed amount. The financing gap was also due to a lack of new large donor project assistance, estimated in the total program costs during appraisal, compounded by the fact that several large ongoing donor assisted projects had closed by 2000.

SAPP-II project financing mechanism-SOEs, financial management issues, disbursements and procurement:

There were two modes of funding, the SOE based reimbursement mechanism and Imprest Accounts for capacity building and monitoring activities.

- 22 -

Donors provided finances retroactively through reimbursement of expenditures on activities that complied with approved operational plans (there were hundreds of thousand transactions annually). Funding mechanisms were designed to provide additional resources and ease such disbursements, thereby avoiding problems of lack of counterpart funds encountered under project funding.

Financial management of the program was weak due to a lack of proper understanding, non-compliance and delays in the implementation of the program arrangements for financial management, and the organization of the financial management system around weak internal control mechanisms. The design envisioned the use of adequately skilled government staff and was based on Governments’ mainstream systems for accounting, reporting and auditing. However, staff were either appointed late (and then subjected to frequent transfers) or not appointed at all and often lacked an appropriate skill set. A lack of proper accounting staff at the Provincial levels resulted in the absence of any analysis of financial management issues, or the development and implementation of corrective actions. Consequently, the requirements of the program such as preparation of annual financial statements, and SOEs were not properly disseminated.

The set of sub-programs that would qualify for reimbursement were agreed each year, and the understanding reached could not be fully communicated to lower levels on a timely basis. Rules, procedures and instructions for using Imprest Accounts had not been prepared in advance and therefore these accounts become functional only in the latter part of the project.

Additional specific points relating to financial management include the following:

Without the core issue of improved and timely recording of expenditures having been addressed, procedural mechanisms for claiming reimbursements through SOEs complicated by a requirement that only the Accountant General’s Office verify expenditures and expectation that departmental staff, with limited exposure to financial systems, decompose them into salary and non-salary components.

The main problems with SOEs related to: (a) the requirement that line departments reconcile SOEs with the records of the Accountant General’s office; (b) frequent revisions in their formats; and (c) difficulties in obtaining monthly expenditure statements from hundreds of District Account Officers (DAOs). This resulted in staff responsible for project implementation and technical managers spending a disproportionate amount of time preparing SOEs, ensuring timely reconciliations and settling audit queries. Financial management and capacity building through the recruitment of staff was severely delayed and took place only toward the end of the program.

An unfortunate outcome of SAPP-II was that it became a victim of the audit procedures built into the expenditure reimbursement mechanism. Part of the problem was that the audits were conducted in a traditional manner. Staff were bound by prescribed classifications for the categorization of audit observations. Resultantly, procedural lapses and misclassifications of expenditures were, at times, categorized as irregularities and misappropriations. These audit report issues made administrators extremely cautious in incurring expenditures, especially on new initiatives and on non-salary components.

The audit report submitted by the Auditor General (AG) of Pakistan for FY 1997/98 and FY 1998/99 identified a very large number of audit observations pointing towards major irregularities and misappropriations. In the view of Government, these reports were hastily submitted to donors without conducting proper Departmental Account Committee (DAC) meetings. The audit procedures are that draft audit reports are discussed with line departments, and finalized on the basis of the review. Draft reports were sent to donors. Due to the findings of these reports, and as a result of private complaints received by

- 23 -

the Bank, the Bank’s Department of Institutional Integrity conducted an Integrity Assurance Mission to Pakistan during April/May 2001. There was an apparent disconnect between the AG's office estimation of ineligible expenditures, and the Mission's estimation. The Mission placed these observations well above the 1% estimated by the Government, the Mission’s findings and recommendations were communicated to the Government. The Government responded to the findings at two levels. First, it started system-wide governance and financial management reforms, and second, has taken the necessary legal, disciplinary and other actions to address specific identified cases. The Government has indicated that the quality of the audit reports was affected by a lack of required audit follow-up with Departmental Account Committees (DACs), and inability of the audit staff to differentiate between a departure from rules and procedures and proven fraud cases.

SAPP was the first Bank project in Pakistan which financed a percentage of all government program expenditures in the basic social services. Therefore, the issues which emerged during implementation had wide raging impact, and actions have been initiated at various levels by both Bank and GOP to address the systemic financial management issues which emerged during SAPP-II implementation.

Actions by the Bank:From 1999 onwards, the Bank included a full time financial management specialist in the SAPP team, and a separate mission on financial management issues was conducted in September 2000 which presented specific recommendations to line departments and GoP for addressing systemic issues as well as those related to expenditure reporting and verification.

The Bank also twice withheld disbursements (in 2001 and 2002) pending satisfactory and credible resolution of financial management and audit issues. Disbursements were subsequently authorized based on submission of verified Statements of Expenditure by Provincial Accountant Generals, submission of SOE audit opinions and unqualified audit reports. Disbursements were made based on expenditures reported by line departments and the Accountant General, and where there was a discrepancy between the two figures, the Bank used the lower expenditure number for disbursements. To ensure that disbursements were not financing ineligible expenditures, the Bank initiated recoveries by deducting the identified ineligible expenditures from the overall expenditures submitted for reimbursements, and recoveries were made for the full program period.

As weak public financial management is endemic throughout the public sector in Pakistan. The Bank supported GOP’s reforms under the Project to Improve Financial Reporting and Auditing (PIFRA) to strengthen accounting and audit functions, and build public sector capacity. In addition, a Country Financial Accountability Assessment (CFAA) was conducted, and two provincial PFAAs (Provincial Financial Accountability Assessments) are underway. The Bank has also negotiated major financial management reforms as a part of the two Provincial Structural Adjustment Credits (SACs) in Sindh and NWFP, and two SACs for the GOP.

Actions by Borrower:The Ministry of Finance initiated an intensive effort with the AG’s office and all concerned SAP agencies to conduct the pending DAC meetings to discuss the audit reports and take remedial actions. The Accountant General’s office was specifically asked to verify all expenditures submitted through SOEs, with Auditor General confirming the authenticity of expenditures through submission of SOE audit opinion. The Auditor General established two task forces for reviewing audit observations pertaining to SAP and other foreign-aided-projects. Detailed reviews were conducted on each audit observation along with supporting documentation. The subsequent annual audit reports contained less significant audit observations pertaining to misappropriation and embezzlement.

- 24 -

The findings of the audit reports were also brought to the attention of the external accountability agency, the National Accountability Bureau (NAB), which was established by the military government soon after it assumed power in October 1999. Army monitoring teams were established to specifically take action again misuse of funds and authority. NAB conducted detailed investigations based on the information provided by the AG. Specific examples of action taken include initiation of disciplinary cases against more than 4000 teachers, launching investigations against at least fifty high profile cases, and conviction of three ministers who were disqualified from holding public offices due to cases of illegal teacher appointments and misappropriation of resources under a donor-assisted program.

More recently, a National Anti-Corruption Strategy has been developed with the assistance of the Department for International Development (DfID). This strategy has been approved by the President and Prime Minister. While these actions have been taken, significant work is still needed to build implementing line departments’ capacities, and to institute strong internal controls at all levels of government, and especially in the district governments which are now directly responsible for social service delivery.

Some systemic financial management improvements which have come about after the SAPP-II experience include: (i) regular reconciliation, consolidation and timely reporting of expenditure data, to expedite appropriate financial reconciliations and setting of audit queries; (ii) timely follow up on audit observations; (iii) setting up of internal audit units in each of the departments, as recommended by the AG, as well as the Bank; (iv) appointment of properly skilled accounting staff in the line departments; and (v) development of an FM manual to help provide important guidelines and procedures, including rules, procedures and instructions for using Imprest Accounts.

Procurement:Inadequate procurement capacity of implementing agencies to diligently follow IDA’s procurement procedures affected the overall quality and timeliness of procurement under the project. The Bank conducted numerous procurement training workshops in which many project staff participated. Nonetheless, the dispersed nature of the project and the large number of implementing agencies and staff made wider interactions difficult. Additionally, the numbers involved several hundreds of procurements. Capacity building through engagement of procurement consultants in each province was conceived in the Project Appraisal Document (PAD), but did not materialize due to administrative delays and bottlenecks. Most procurement, including all civil works, involved small contracts over widely dispersed geographical areas, and was below the Bank's prior review threshold. Large procurement operations were confined to pharmaceuticals and contraceptives which were procured through UN agencies, and large consultant contracts were to be based on the Bank's policy for selection of consultants.

In the procurement of civil works, in particular, generic deviations were observed between Government’s procurement procedures and practices, as followed by the implementing agencies, and those of IDA. These included the Government’s procedure to restrict bidding only to contractors pre-registered with government departments, and the use of a composite schedule of rates instead of item rates for inviting bids and awarding contracts. While improvements for National Competitive Bidding (NCB) procedures were specified in the PAD, these were not explicitly stated in the Development Credit Agreement (DCA) as, at the time of SAPP-II appraisal, it was not the practice to make such references in IDA legal documents. These improvements were legally incorporated at a later stage in the project, during Project restructuring in 2001. In more recent IDA-financed operations, procurement capacity strengthening measures, including the hiring of procurement consultants where required, are being explicitly agreed upfront during project preparation. The required NCB improvements are now being clearly specified in the legal agreements to eliminate any ambiguities, and to facilitate remedial actions by the IDA in event of serious deviations.

- 25 -

6. Sustainability

6.1 Rationale for sustainability rating:Sustainability is rated as Likely for reasons noted below:

SAPP-II resulted in a few key policy shifts that have formed the backbone of several government initiatives in the last couple of years. While outcomes during SAPP-II were lower than expected and implementation less than satisfactory, a large number of initiatives and reforms - and ownership of these reforms - have become entrenched because of the emphasis that SAP placed on these issues. The continued emphasis SAPP-II placed on improvement of service delivery at the implementation level through decentralization influenced and provided the rationale for devolution of powers to the local level. The focus on community participation and NGO involvement in service delivery that SAPP-II continuously highlighted has become an integral part of several important public programs and policies, including the PRSP, ESR, and the Khushal Pakistan Program, as well as several micro-finance initiatives that have emerged. Many of the changes supported through SAPP-II could be considered as irreversible as is evident by the fact that most initiatives commenced under SAP are being promoted by national policy frameworks such as the PRSP and ESR, and are being pursued by provincial governments. After SAP, PRSP provides the strategic framework for social sectors, and is strongly anchored on social sector programs and related pro-poor expenditures. At the national level, the ESR, while taking a sector wide approach, continues to focus on the SAP policies, including priority for girls' education, role of communities, and building public private partnerships for service delivery. In the health sector, the GOP's Health Policy 2001 maintains the policy shift in favor of preventive health care. Further, the reforms that were introduced under SAP have now been taken on in the provincial reform programs, with emphasis on protecting investment for health and education, supported through provincial SACs (in Sindh and NWFP) and a possible education sector adjustment program in Punjab. After the closure of SAPP, these provincial governments have significantly increased their education budgets, and are also fully financing programs aimed at increasing girls' enrollments. The unsatisfactory rating on achievement of outcomes is based on past performance, whereas the likelihood of sustainability is forward-looking.

SAPP-II was a five-year program but by the third year, implementation had become uneven and problems as mentioned above were emerging. Donors began to retreat, and the program was not continued till its full completion. Even over a five year period, it would not have been realistic to expect notable improvements in social sector outcomes. Improvements in the social sectors take time and any kind of measurable impact requires a long-term process of investment and reform.

SAPP-II did contribute towards the institutionalization of the reforms and policy shifts mentioned above, with the cautionary note that it is too early to give a judgment on the continuity of the financial support for, and institutionalization of, such reforms at the local government level. This is partly because institutional reform tends to be a slow and long process, and instituting governance reforms is particularly difficult in societies struggling to change ingrained attitudes and practices and revamp institutional structures that have acquired a high degree of permanence over the years. In the end, success and sustainability will hinge critically on the devolution being able to improve accountability systems of service delivery and address issues pertaining to adequacy of allocations, access and quality.

Sustainability of SAPP-II has to be rated against the backdrop of these reform programs, the progressive

- 26 -

policy shifts and frameworks, and ownership by government of these reforms. Therefore, sustainability is rated Likely, even though the development outcomes are rated as unsatisfactory -- since the latter reflects the absence of tangible improvements in outcomes by the end of the SAPP implementation period. If the current reforms are continued in letter and in spirit, one should see improvements in human development outcomes in the medium term.

6.2 Transition arrangement to regular operations:Since SAPP-II supported GOP's entire basic social service delivery program, there were no transition arrangements to regular operations as required in investment projects. The issue is being addressed here in relation to the key policy dimensions which are being continued even after closure of SAPP-II. GOP's PRSP is strongly anchored in social sectors and focuses on increased the social sector expenditures. The political commitment at the highest levels is also evidenced in the incorporation of SAP objectives and policies in the ESR, and in Provincial Reform Programs.

7. Bank and Borrower Performance

Bank7.1 Lending:Learning from the experience of SAPP-I, Bank preparation intensely focused on developing stakeholder ownership and participation during project preparation. Lending development was technically sound, and the preparation team not only engaged with the provincial governments, but also conducted several stakeholder workshops involving district and local officials of the line departments. Workshops were also conducted with civil society representatives to develop the Participatory Development Program (PDP). However, no full-time financial management specialist was involved during the preparation stage. The subsequent number of issues in implementation of the MoU governing financial management may have been reduced if there had been more extensive technical focus and dialogue on financial management issues during project preparation. The number of people in preparation missions, sometimes more than forty IDA and other donor mission members, overloaded government's capacity, and wrap-up meetings could not be used for conducting policy dialogue and reaching policy agreements.

The preparation for SAPP-II commenced in 1996, with the expectation that SAPP-II would be in place by 1997, soon after closure of SAPP-I, in order that it would retroactively finance any gap period. However, SAPP-II became effective only in May 1998, and there was a full one-year gap in donor financing. Although retroactive financing was included in SAPP-II, there was a considerable funding hiatus and the full financing gap could not be surmounted. A second gap emerged as a result of different preparation timetables of two major financiers, the Asian Development Bank (ADB) and International Development Association (IDA), where ADB’s SAPP-II project was made effective in late 1996, while the IDA project became effective in 1998. This complicated financing arrangements after the closure of SAPP-I, and taxed Government's capacities due to separate preparation schedules of the two multilateral agencies.

7.2 Supervision:Typically, three supervision missions per year were undertaken to supervise SAPP-II: (a) to review the draft Annual Operational Plans (AOPs) for the following fiscal year in time for agreements to be incorporated into the budget preparation process; (b) to take decisions on the eligibility of the Annual Operational Plans, and to resolve any outstanding program or budgetary issues; and (c) mid-year to review implementation progress. Supervision activities were generally well-planned and comprehensive in terms of identifying issues to be resolved and actions to be initiated. Supervision missions left behind daunting array of actions that provincial governments, suffering from limited capacity and frequent changes in leadership, were expected to undertake.

- 27 -

From the start of the project, supervision missions repeatedly flagged implementation issues, including a lack of a shared vision on SAP; poor governance; weak implementation capacity; inadequate allocations for non-salary components; delays in the release of funds; staffing delays; rapid turnover in leadership positions; non-merit based criteria for staff appointments and deployment; staff absenteeism; failure to appoint qualified counterparts; lack of progress on capacity building and PDP initiatives; and weaknesses in financial management and procurement procedures. Reviews tended to focus more on budget issues and tracking processes and inputs, than on quality aspects. Nevertheless, issues relating to service delivery(with the expectation that these would eventually influence outcomes) started to be raised, particularly in the latter part of SAPP-II.

The PIHS, HMIS and EMIS were the main sources of data for the health and education indicators. The PIHS served as the main instrument for evaluating the impact of SAP, while the results of TPV exercises and financial audits were used to track progress on governance and financial management issues. The PIHS was conducted in alternate years during the SAPP-II period (1998-99 and 2001-02), as originally planned, and its results were well utilized for analysis, especially at the time of the Mid-Term Review (MTR). The utilization of HMIS and EMIS data for monitoring purposes was at best nominal. Moreover, since the program and annual target indicators were not specified, it was not clear how the achievement of goals was being assessed. In the absence of specified targets, even regular monitoring of indicators would have only had limited success in evaluating program performance and determining the extent to which sector performance was adequate to ensure continued disbursements.

Supervision missions were often large and unwieldy, with each donor (irrespective of the level of their financial contribution to the program) represented. Often micro, project-specific or narrowly-based sectoral issues were raised, many of which were viewed by Government counterparts as superficial in relation to the larger program objectives. The Aide Memoires left with government were enormous, partly owing to the wide scope of the program, and failed to prioritize issues, actions and targets. The un-prioritized list of detailed actions for follow up became formidable because of the sheer number of tasks to be performed, and the difficulty of line departments to absorb and respond in light of their limited capacities.

In the absence of desired level of political commitment to the program, donors appeared to have assumed greater operational and a leadership roles for resolving issues, rather than demanding a clearer demonstration of commitment from Government. Additionally, interactions with Government focused on tactical rather than strategic issues and discussions with government tended to gravitate around sector specific issues in part due to the multi-sectoral nature of the project. These factors unintentionally shifted focus away from policy reforms and cross-sectoral issues, critical for sustainable social delivery improvements, to tracking expenditures in SAP sectors, reviewing procurement procedures, preparing AOPs, and evaluating the effectiveness of instruments to achieve the expected outcomes.

Financial management monitoring started to be addressed by supervision missions as of May 1999 with the involvement of financial management staff. This enhanced supervision team's understanding of IDA's financial management and reporting requirements. Supervision teams did not include an expert on institutional issues at any time during the project, although the centerpiece of SAPP-II was institutional and governance reforms for improving the efficiency and effectiveness of social service delivery.

7.3 Overall Bank performance:Whereas SAPP-II was conceived as a policy instrument for program lending it was implemented in project mode and supervised by Bank staff as a project.

- 28 -

Most government officials regarded the role of the donors as helpful and constructive. Major issues identified by provincial functionaries on the role of the donors pertained to: (a) large and frequent missions; (b) demanding procedures and reporting requirements; and (c) rigid enforcement of procurement and financial management procedures that resulted in some issues remaining unresolved. One concern of Government was that donors did not point out the severity of the problems that they were encountering until too late - virtually at the time they decided to pull out. This perception is not misplaced as up to the time of the MTR, IDA had not indicated that it would take formal remedial or other actions if improvements in the identified areas continued to remain unresolved even though disbursements were suspended in a few areas/sectors showing poor progress. At MTR GoP launched a major initiative to establish local governments under the Devolution Program, thereby shifting service delivery responsibility to them. Due to this change of strategy for the delivery of social services from provinces to districts, poor outcomes in the provincial programs and lack of clarity on new institutional arrangements led to IDA's decision to fund only the federal education, population and health programs, and halt the financing of provincial programs. The rationale for this was the heavy dependence of these programs on external funds and their relatively better progress and outcomes and implementation. This decision violated the spirit of SAPP-II to operate in the provinces.

When it had become evident that owing to political and macro-economic uncertainties that Government would not be able to allocate adequate funding for non-salary expenditures, IDA did not revise its strategy. Rather, it continued to try to obtain commitments on expenditures, which was unrealistic, given the financial constraints of provinces. While allocations were declining and progress on outcomes on a number of strategic interventions was uncertain, IDA continued to disburse funds, despite SAPP-II having qualified as a problem project. IDA neither changed its strategy nor considered sanctions involving the suspension of financial support until specified agreed outcomes had been achieved. In retrospect, sanctions should have commenced earlier than FY01. During the early years of the project, IDA was reluctant to exercise its leverage by withholding disbursements. During the latter years of the project however, IDA withheld disbursements because of non-compliance with financial management covenants.

There was little evidence to support the project ratings in the Project Status Reports (PSRs). PSRs continued to give satisfactory ratings on compliance with legal covenants (with the exception of financial covenants which were given unsatisfactory ratings), although recruitment bans continued to be applied to the social sectors and progress in securing timely releases continued to be poor. This divergence was not explained satisfactorily, although the PSR ratings did become more realistic during the latter part of the project.

Donor Coordination:Coordination of donor efforts was carried out by IDA. At the request of GoP, donor coordination commenced from the SAPP-I period during the early nineties, and continued throughout SAPP-II.

During implementation, while core coordination was restricted to SAP financiers, there were efforts especially at the end of missions to present mission findings to other donor partners. Within the SAPP donor group, IDA, the ADB and bilateral donors worked as cohesive teams. While there was internal coordination, this sometimes did not appear as such to Government. During the early period of SAPP-II, each province had large donor-assisted investments projects, including projects supported by the two multilateral agencies. Often dialogue under the projects did not link up effectively with SAPP-II, and counterparts were unable to distinguish between the program and projects.

While donor coordination ensured policy consistency and complemented monitoring and supervision efforts, coordination became complicated because each donor, irrespective of the size of its financial

- 29 -

assistance, wanted to be an active participant in the project and its supervision. As a result, it often became difficult to maintain a common, cohesive, donor stance. For instance, there was lack of clarity on compliance criteria for expenditures eligible for reimbursement; IDA would classify some expenditures as non-compliant which ADB would consider as eligible.

Multi-donor supervisory missions repeatedly raised issues related to service delivery, they placed focus more on budget issues, tracking processes, input mechanisms and instruments to attain ends, than on issues of quality and outcomes. IDA continued to reimburse expenditures even after SAPP-II was identified as a problem project, allocations to the social sectors were declining, and progress on a number of strategic interventions was deemed unsatisfactory. IDA neither realigned its strategy nor did it consider the use of sanctions involving the suspension of financial support until specified outcomes had been achieved. The realignment of strategy occurred late (i.e. after MTR) when the project was restructured. During the same period, IDA also withheld disbursements on two occasions due to non-compliance on Financial Covenants. Overall, Bank performance has been rated unsatisfactory, although a ranking of marginally satisfactory may be appropriate.

Borrower7.4 Preparation:During the latter period of SAPP-I, Government requested continued donor support through a follow up operation. High-level committees were established by Government to develop ownership for the program, including Chief Minister’s Committees in the provinces. During the SAPP-I period, the program was reviewed at the Cabinet level under the Social Sector Advisor to the Prime Minister. Line agencies began an intensive process for preparing sector strategies and accompanying rolling sector financing plans, with support from donors and MSU. The first program development mission was conducted in May 1996. By the time the pre-appraisal mission took place in December 1996, the government had changed. This affected the pace of preparation due to staffing changes and cabinet changes at the federal level. Despite these political and accompanying staffing changes, the momentum for preparation was maintained and sector departments continued not only with preparation of their annual operational plans, but also developed sector strategies. During this preparation period, the MoU for conducting Third Party Validation Surveys (TPV) on governance issues was signed. Borrower performance during the preparation phase was satisfactory.

7.5 Government implementation performance:The coordinating agency at the federal and provincial levels were the Federal Planning and Development Division (P&D) and Provincial Planning and Development Departments respectively. A key gap in these arrangements was the lack of direct role of federal and provincial finance ministries. In addition, the Federal Bureau of Statistics (FBS) was responsible for conducting the Pakistan Integrated Household Survey (PIHS), and the Auditor General for conducting the annual Third Party Validation Surveys (TPV) on governance issues, as well as submission of annual audit reports.

The federal and provincial P&D departments were responsible for coordinating line departments, and for monitoring implementation. Federal and provincial steering committees were responsible for overall program review and monitoring, based on information provided in the QMRs. QMRs became routine reports with many significant data gaps and therefore could not be used as effective monitoring tools. The P&D departments' own capacities could not be strengthened to effectively manage this large program, as the capacity development program envisioned under the project never got fully underway due to procedural and approval delays. A critical aspect of the program, efficient financial management and tracking of expenditures, could not take off due to a lack of qualified accounting staff in these departments.

- 30 -

External tools to monitor governance and impact were produced in a timely manner. During the first two years, TPV reports were not fully acknowledged by line departments, and it was only toward the end of the program that they were reviewed seriously, for the purpose of taking corrective actions. While SAPP-specific TPV surveys have ended, different provinces have agreed to conduct these surveys using their own resources, as a tool to monitor the impact of their programs. An important success of the program was institutionalization of the PIHS, started during SAPP-I. PIHS was effectively used to monitor program impact during the MTR, when performance was assessed in relation to progress on outcome indicators. Even after closure of the project, PIHS is being financed by Government, and is continuing with changes incorporated in view of the requirements of PRSP monitoring indicators.

SAP required stability in the overall governance environment. This was not the case due to frequent political change. There were periods, especially from 2000 onwards, when the federal government commenced serious implementation reviews through the Social Sector Cabinet Committee (SSCC), with active engagement of both federal and provincial government representatives. Subsequent decisions to restructure SAPP were taken through these deliberations. Overall, implementation performance remained uneven during different periods of implementation, ranging from marginally satisfactory to unsatisfactory.

7.6 Implementing Agency:In the provinces and federal areas, SAP was implemented through the departments of education, health, public health and engineering (for RWSS), and population covering all elementary education, basic health, population and RWSS programs. These departments historically managed these programs and the sheer size and scope of the regular program was not new to them. The new aspects were the sector-wide reforms which were initiated by SAP which stressed the system. Donor requirements (e.g. preparation of annual operational plans, quarterly monitoring reports, SOEs, etc.) additionally stretched their limited capacities. While support for capacity building was included in SAPP-II, it never got fully off the ground due to delays in establishing Imprest Accounts, and later, delays in obtaining approvals for the capacity building programs. During the latter part of the program, the energies of the line department staff were diverted towards resolution of audit related issues. These staff lacked financial management expertise, yielding further capacity constraints and diverting attention away from their primary tasks.

Throughout SAP, donors interacted with provincial line departments, yet with limited or no interaction with district officials responsible for day-to-day implementation. District officials were rarely briefed on SAP, consequently, they did not clearly understand its objectives. While planning capacities improved through the preparation of annual operational plans, monitoring within departments and at a more macro level through the coordinating Planning & Development Departments remained uneven. This resulted in more extensive donor oversight through missions, and donors were often perceived as micro managing the program.

Implementation progress varied during different phases of implementation, and was largely dependent on the leadership within each department and on the teams engaged in monitoring implementation. During different periods, the performance of a sub-program improved significantly, and was considered satisfactory due to stable and able leadership. Yet, the same sub-program might regress when such leadership was lacking. Frequent transfers of line department secretaries, especially in the education sector, negatively effected program implementation.

Overall, implementing agency performance is rated unsatisfactory, although a more realistic rating would be marginally satisfactory as there were periods when, as noted above, implementation performance, especially in the larger sub-programs, was satisfactory.

- 31 -

7.7 Overall Borrower performance:Provincial governments were not fully attuned to a program approach as most of their previous interactions with donors had been on more specific investment projects. This, compounded by the fact that line departments function as compartmentalized entities, contributed to SAPP-II not being perceived as a program. Partly because of this, inter-departmental and inter-agency coordination remained a problem throughout the project and led to implementation delays.

The program was not viewed holistically. Sectors were viewed independent of one another and interventions were not introduced as parts of an overall, comprehensive strategy using an integrated approach. Failure to ensure integration of SAP sectors in a geographical area meant that cross-sectoral factors and linkages between departments were not sufficiently accounted for. The coordinating agency in the planning and supervision of SAP was the Planning and Development Department, which lacked adequate information on expected resource flows. The Finance Department, being better informed on resource availability, should have been more engaged board during the project design and implementation stages.

Weak procurement capacity in implementing agencies for following IDA’s procurement procedures affected the quality and timeliness of procurement. Provinces, however failed to engage procurement consultants to help implementing agencies carry out project procurement more effectively in accordance with IDA procedures. The Auditor General's office was initially not fully aware of IDA procurement procedures - a weakness that compounded problems since AG representatives would highlight procedural deviations that were not deviations from IDA's procurement procedures (e.g. IDA procedures do not require that small contracts procured under National Shopping procedures be advertised), while other practices (e.g, negotiations with the lowest bidders) were accepted by Government, but were violations of IDA’s procedures. IDA subsequently provided procurement training to AG staff assigned to the Project.

Borrower performance in SAPP-II is rated unsatisfactory, although a fairer assessment would assess government performance as uneven for overall rating of marginally satisfactory. As noted earlier, Government launched a number of positive initiatives, such as contract-based, facility specific recruitment of service providers, adoption of co-education and a decision to recruit female teachers at the primary level. Provincial governments also introduced innovative approaches for increasing enrollments, such as the provision of scholarships for girls and free textbooks for elementary students. These initiatives are now being funded by the provinces’ own sources. However, it remains to be seen whether these approaches and policy decisions will be continued by district governments. Factors within the control of Government, such as recruitment bans, the delayed release of funds, and high turnover of key personnel undermined progress towards project objectives.

8. Lessons Learned

Institutional Reform & Complexity of Program1. Complex programs such as SAPP-II that aim to bring about gigantic institutional reform take a long time to implement. A longer implementation time-frame, greater perseverance and stronger donor coordination are needed to move to scale on successful ‘pilot’ interventions and innovations and to undertake fundamental institutional and governance reforms that require, almost as a prerequisite, changes in orientation and mindset, attitudes, work ethics, and even political culture. The low levels of achievement under SAPP-II provide yet another example of the failure of umbrella projects, especially those involving a multi-sector and multi-provincial operation. Such projects are scattered in their scope. It is inherently difficult to ensure a degree of focus that can be attained in designing programs and projects that are province-specific and relevant to their stage of development.

- 32 -

2. The technical and political difficulties of implementation tended to be under-estimated, which resulted in over-optimism about the pace at which success could be achieved. Failure to fully recognize that political considerations can overwhelm technical and capacity issues and can result in programs being designed and supported even when probability of success is low. It instills a more relaxed approach to implementation. This factor partly explains the continued funding under SAPP-II, despite poor progress on governance issues. If political factors are well-defined, then the process of program design, its appropriateness and the likelihood of its success will not only be transparent but will also enable a clearer crafting of roles, responsibilities and accountabilities, of both borrowers and donors.

3. The agenda should not be overloaded. When implementation capacity is weak, project designs should be simple and modest in their scope, involving fewer components and executing agencies. This extensive agenda detracted focus from the core reforms, particularly of the institutional variety, that were critical for sustainability. Perhaps this was partly owing to the narrow specialization of some mission members and their lack of experience of issues connected with institutional reforms and program/policy-based lending.

4. Excessively detailed policy conditionalities, especially when Government's track record was weak or doubts persisted concerning ownership, are unlikely to have made implementation of the program more effective. This suggests that the specific structure of conditionalities and proposed actions are less important than the underlying political commitment to the core policy and institutional reforms. Moreover, experience from other lending operations in Pakistan suggests that there is no clear, unambiguous evidence concerning the effectiveness of prior actions. Proposed prior actions may not significantly influence program/project implementation, in the absence of strong and coherent integration in the program design. Creating and sustaining ownership requires that: (a) the program be underpinned by prolonged dialogue prior to design finalization; and (b) stability and continuity of key personnel. If political factors are well understood and sufficiently accounted for in the program's design, then the program's appropriateness and likelihood of success will be greater. This will also allow for a clearer delineation of roles, responsibilities, and accountabilities for both borrower and donor(s).

5. Not only is it difficult to apply conditionalities to complex institutional issues that are generally critical for achieving longer term sustainability, institutional reforms in the social sectors are difficult to implement in isolation without addressing the issue of poor governance and without undertaking fundamental institutional reforms in the government as a whole. It is important to address and fix governance issues before large scale program expansions. In a major program such as SAP, reforming institutions and behaviors required a framework and teams with aligned visions to steer change. Government's programs after SAP, i.e., PRSP, ESR, and provincial reform programs, have benefitted from this lesson as seen in efforts to build stakeholder consensus.

6. SAP was able to introduce new alternative instruments and institutional arrangements to improve governance without straining already scarce public resources. It addressed the issue of accountability of service providers to service recipients by institutionalizing community participation in service delivery. Although achievement of this objective was only partial, the concept of community involvement remains to date in the institutional structures being designed for public service delivery. Longer-term donor commitment to support advocacy, and mobilization efforts will help strengthen community involvement.

7. The lessons learnt from SAPP-II experience in the area of financial management have been successfully applied in the case of the DERA project, and have led to a wider recognition of financial management issues in other programs.

Capacity Issues

- 33 -

8. Over optimistic assessments of Borrower capacity result in the formulation of objectives which are too ambitious. This erodes the credibility of both borrowers and donors by establishing programs with a weak design, on the one hand, and reforms which are carried out unevenly and haltingly, on the other. It takes time to build adequate institutional capacity to take successful initiatives to scale, and this needs to be adequately factored into the program design.

9. Capacity building is difficult to undertake in a multi-sector and multi-province project like SAPP-II. It requires consistent and effective leadership to implement a well-conceived and appropriately sequenced action plan, and the transfer of skills requires a minimum level of qualification and professional competence in the part of counterparts.

10. Implementing agencies' financial management capacity should be comprehensively assessed during project design and a baseline level of capacity should be in place prior to start of programs and projects. Additionally, Government staff responsible for incurring expenditures (e.g. Drawing and Disbursing Officers) should be provided regular training and other capacity building support.

11. Procurement capacity strengthening measures should be explicitly agreed during appraisal and put in place as early ads possible in the project preparation phase.Measures for improving National Competitive Bidding procedures should also be agreed upfront and incorporated in the legal agreement. There is also a need to harmonize procurement procedures and use common bidding documents to avoid borrowers being overly burdened by having to satisfy the requirements of different sets of procedures of donors and government. The Bank has already taken steps in this regard.

Coordination, Management and Monitoring Issues

12. Since additional budgetary allocations were critical for service delivery improvements and considering the importance placed on financial management and sustainability issues, much more coordination and dialogue is required with Ministry of Finance and the provincial finance departments.

13. Continuity in supervision mission teams with an appropriate skill mix and adequate time for conducting field visits can reduce frequent and, at times, inconsistent changes in strategies and priorities and enhance capacity.

14. Instruments such as the PIHS strengthen knowledge of the sector and provide rich data for analyzing and understanding household behaviors. The regularity of household surveys and the use of the data have been a significant success of the program. Additionally, the use of third party validation has been extremely useful in verifying program inputs and for monitoring progress in governance areas.

Future Support for Social Sector Reform

15. An important lesson of SAP is that sector policies can be better supported through adjustment lending which has higher visibility, and where additional resources can be leveraged through policy changes in support of agreed outcomes. The Bank has drawn lessons from both the negative and positive aspects of the SAP experience, and is supporting national and provincial reforms through structural adjustment programs which include strong human development reforms. The Bank is also examining the possibilities of providing support to social sectors through sector adjustment operations. The financing mechanism should provide the line departments/provincial governments an incentive through the availability of additional funds or through upfront funding.

- 34 -

16. Structural adjustment is more suitable for policy and macro adjustments and governance reforms. Investment gaps in physical infrastructure and institutional capacity, both in hardware and software, may be better filled by investment projects to support field level activities. Enhancing the effectiveness of Structural Adjustment Credits (SACs) requires complementary sectoral and project lending to strengthen capacities (say for policy formulation, planning and monitoring and evaluation) critical for improving sectoral outcomes.

17. To avoid intra-sectoral imbalances future sectoral lending could be a hybrid of program and project lending that is predicated upon an integrated overall investment strategy for the sector, and an action plan that incorporates an appropriate sequencing scheme for the different sub-sectors, as opposed to the sub-sector approach of the SAPPs.

18. Respecting the autonomy of the new local governments, a Social Development Fund could be established at the provincial level that district governments can draw upon by satisfying criteria that promote national or provincial policies and goals pertaining to the social sectors. Such an institutional mechanism could be a resource pool, funding from which would be distributed either as tied grants or incentive based performance grants.

9. Partner Comments

(a) Borrower/implementing agency:The ratings by the Borrower on SAPP-II are summarized below:

Performance RatingOutcomes Un-satisfactory/modestlSustainability LikelylInstitutional Development Impact ModestlBank Performance Satisfactory with qualificationslBorrower Performance Satisfactory with qualificationsl

Background:The Government of Pakistan’s Social Action Program (SAP), started in 1992, was supported by donors in two phases through Social Action Projects I and II (SAPP-I and SAPP-II). The program showed improvements in both outcomes and expenditures during the first phase. During the second phase, from 1997 onwards, the program was supported through the multi-donor SAPP-II. Government tried to maintain its expenditure levels, although there were shortfalls after 1998 when the country faced an acute financial crisis. In addition, expected additional donor financing could not materialize. There were also shortfalls out of the committed donor funds under the multi-donor SAPP-II arrangements. Despite resource constraints, provincial governments tried their utmost to maintain expenditure levels for SAP sectors, often at the expense of other sectors which, during the later years of SAPP-II, became difficult for the provinces in view of multi-sector demands. However, implementing agencies maintained their focus on the priority sectors, especially education.

Assessment of Development Objectives:There were modest improvements in outcomes in the health sector, while improvements in the population outcomes were better. However, the education sector, where the bulk of expenditures took place, did not show commensurate improvements in outcomes, especially enrolments, although the gender gap in primary education enrolments did decline during this period. Despite these modest improvements in outcomes in the

- 35 -

social sectors, there was an overall positive impact of the program.

POSITIVE IMPACTSFor the first time in Pakistan’s history, SAP and SAPP-II brought together policy makers, managers land implementation staff from the social sectors. It brought the importance of social sectors to the forefront. It was successful in changing mind-sets to a significant extent and in creating a good degree of awareness of the importance of investing in the social sectors.The multi-sector framework enabled all stakeholders to examine social sectors in a holistic manner.lStrong emphasis was placed on the need to improve service delivery through staff training and to curb labsenteeism, despite the fact that this could not be converted into actions at the appropriate levels.It helped in conceptualizing as high priority the need to develop institutions and restructure systems as lopposed to the provision of additional funds.Contract-based recruitment of teachers and health sector personnel, and benefits of mixed schools and lcommunity schools were established as good initiatives.Issues of governance were highlighted and absenteeism was reported by surveys conducted by lprovincial governments with the assistance of Army Monitoring Teams, and corrective measures were taken to rid the system of these negative attributes.Scholarship and free textbooks were provided for elementary classes as an incentive for parents to send ltheir children to school. Provincial governments, even after the withdrawal of donors, are following the incentive policies developed under this program, using their own resources. Furthermore in Sindh, this program has been extended to urban areas. Teacher-training courses were developed during SAPP-II to improve education quality. These are still lbeing implemented by the provincial governments.PTA's and SMC's performance is rated fairly high. Provincial governments have not only supported lthese committees financially, but have also empowered committees, for example funds for school level procurement continue to be provided at the school committees level, setting new positive trends. Provincial departments feel that SAP played a pivotal role in protecting funds from cuts in an lenvironment when the Finance Division had sufficient justification for imposing cuts, such as during the financial crisis and severe drought 1998-02.Most importantly, expenditures for on the social sectors were accorded some degree of protection in lspite of severe financial crises, political instability, external shocks and serious macroeconomic problems. Enhanced emphasis on improving service delivery through staff training and checks on absenteeism of lservice providers were started. Corrective actions taken by provincial governments, and which continue to be taken even after the lproject has ended, on the report of the Auditor General of Pakistan in matters related to governance accountability and transparency of operations. Policy decisions to adopt co-education and induct teachers for primary level schooling.l

Due to the rigidity of the program it took some time at the start to gauge the ground realities, which were later made more apparent through reviews. Dialogues were useful in making required corrections.

Major Factors affecting implementation and outcomes(a) The ICR highlights the following general factors:

Lower outcomes were also due to inadequate management, inappropriate selection of sites, poor maintenance of infrastructure, inadequacy of skilled staff, frequent transfer of key personnel, uneven location of service outlets, non-merit based recruitment of staff, absenteeism of service providers, lack of political ownership at the local level and limited financial and human resource capabilities in provincial

- 36 -

governments. The monitoring system did not go beyond reporting exercises and was not sufficiently results oriented. Corrective measures were not taken adequately and in time. It generated quantitative data, but did not highlight accuracy and qualitative aspects. Since funding was not upfront, the flow of financial resources was neither smooth nor need-based, and delays occurred in the delivery of funds at points where procurement was to take place or salaries paid. Finally, documentation needs were founded upon on a weak local level system.

However, responsibility of unsatisfactory performance (SAPP-II) is a joint one by donors and GOP. Clearer identification of capacity constraints needed to be made upfront. Low levels of donor support for SAPP, at around 6-8% of expenditures, was perhaps disproportionate to GOP’s financial support to the program. This provided inadequate incentives. Among many reasons for unsatisfactory performance of SAPP-II given in the ICR, one thing is missing, i.e. insufficient evaluation carried out to derive lessons from SAPP-I due to a lack of time. Some of the consultants hired to do this either did not perform adequately or were recruited too late due to bureaucratic problems. This should have been noted in the report.

(b) Donor Related Factors:Three supervision missions were undertaken to supervise SAPP-II. The objectives were to review the draft Operational Plan for the fiscal year, issues related budget, eligibility of operational plans, and to resolve program and budgetary issues. In the middle of the year, supervision missions were conducted to review implementation status. The focus of the missions was on expenditures.

- Supervision missions were well planned and comprehensive in so far as identifying the range of issues but were ineffective in addressing the key strategic ones. Because of inadequacies of provincial governments, frequent changes in leadership and governance issues resulted in making the supervision mission' task more difficult and a fresh start had to be made frequently.

- Issues to address the trends in outcomes, such as learning assessment and movement on institutional and governance reforms, remained largely in the background. This resulted in delayed formulation and enforcement of financial management system, which was not only complicated, but also changed continuously due to a lack of capacity to run it efficiently at the operational level.

- Supervision missions worked at the provincial level, and did not conduct adequate assessment of service delivery improvement, or direct interaction with district and field level staff.

- Moreover, partly owing to the wide scope and dispersed nature of country-wide service facilities, Aide Memoires were comprehensive, but not sufficiently internalized by implementing partners. This led to confusion in setting priorities for addressing issues, and agreeing on doable actions and targets.

- IDA did not put adequate emphasis on the implementation of capacity building activities at the field level. Resultantly, implementation of them took place at the tail end of the project period;

- The perception was that the MSU played a supervisory and oversight function.

SUSTAINABILITY:Several policy areas which emerged during the SAP period, and which also were part of SAPP-II, are being continued after the closure of SAP/P. After SAP, the PRSP provides a strategic framework for development of the social sectors, with a strong social sector focus as expenditures on basic social services are considered to be pro poor. The PRSP is also anchored on achievement of the Millennium Development Goals.

In addition, the concerned governments are continuing important polices started during SAP. These include a focus on girls primary and elementary education; special incentives such as free textbooks and

- 37 -

scholarships for girls; sustained increases in social sector spending over the past two years, especially in education; involvement of communities in the education sector through parent teacher associations and school committees; continuation, with government financing, of important health programs such as LHW, EPI, AIDS; reforms to improve financial management; continuation of quality programs such as teacher training programs; commencement of a national education assessment program; and continuation of third party assessments by provincial governments. An important part of the SAP reforms was decentralization, which could not be implemented as envisioned. However, a similar agenda is now being advanced by the Government under the devolution reform process, with the establishment of local governments and the transfer of basic social service delivery responsibility to district governments. In view of all these efforts, it is expected that service delivery and outcomes will improve over the coming years. In view of these considerations, sustainability is considered likely.

LESSONS LEARNT:The agenda should not be overloaded. Project designs should be simpler and more limited in scope, lwith fewer components and implementing agencies.Political factors should be well defined, as should the identification and assessment of risks at the time lof program formulation.The time frame for implementation of the program should be longer to account for the fact that lSAPP-II introduced into the system a number of new innovations, mechanisms and the institutional arrangements for provincial governments, which had limited capabilities in an environment of uncertain political commitments.Institutional reform cannot be achieved simply by applying conditionalities without addressing the core lissues, i.e. governance and fundamental institutional reform.Capacity building requires consistent and effective leadership to implement a well conceived and lappropriately sequence action plan.A social development fund could be established as an alternative institutional mechanism at the lprovincial level so that district governments can accomplish their goals more easily. Funding would be distributed as tied grants or incentive-based performance grants.SAPP-II was an umbrella project with multi-sector, multi-province and multi-donor coverage and linputs. To ensure a greater degree of focus, the project should have been province specific and relevant to the situation of a province with more emphasis on outcomes.The provincial finance departments should have been fully on board, especially for recurrent budgets.lProcurement capacities should be strengthened at the project preparation phase. NCB requires limprovements and there is a need to harmonize the procurement procedures and the use of common bidding documents.Training is a key input in all cases and at all levels to make better use of financial resources. It is the lperson behind the delivery of service that counts, particularly in education where the teacher is the key determinant of quality.

(b) Cofinanciers:The European Commission (EC) and the Department for International Development (DfID) state that the ICR comprehensively covers their observations. Both EC and DfID agree to co-own the report.

(c) Other partners (NGOs/private sector):Consultations were conducted in the context of the preparation of the Country Assistance Strategy (CAS) and the Poverty Assessment Report, and by partner donors such as DfID, and the outcome of these deliberations has been incorporated in the ICR.

10. Additional Information

- 38 -

N/A

- 39 -

Annex 1. Key Performance Indicators/Log Frame Matrix

Outcome / Impact Indicators:

Indicator/Matrix

Projected in last PSR1

Actual/Latest Estimate

Total enrollment government schools (1-5) -- male, female, total

53 42

Combined government and non- government rate (1-5) -- male, female, total

71 72

Gross middle (6-8) enrollment rates 40 41Children aged 10-18 years that left school before completing primary level

15 15

Infant mortality rate 89 82

1. End of Project2. From PIHS 2001-2002Output Indicators:

Indicator/Matrix

Projected in last PSR1

Actual/Latest Estimate

EducationNumber of Primary SchoolsNumber of Middle SchoolsEnrollment in Primary SchoolsEnrollment in Middle SchoolsNumber of Teachers in Primary SchoolsNumber of Teachers in Middle Schools

127,70912,9849,555,9292,524,558373,00096,000

HealthNumber of Lady Health WorkersNumber of Lady Health VisitorsNumber of Basic Health UnitsNumber of Rural Health Centers

73,4485,8455,230541

Population WelfareNumber of Family Welfare CentersRHS A Centers

3,21925

1 End of project

Annex 1: Key Performance Indicators/Log Frame Matrix

- 40 -

Table 1: Net Enrollment Rates at the Primary Level Net Primary Enrollment Rates (%)

1998-99 2001-02

Male Female Overall Male Female Overall Urban 54 57 56 57 58 57 Rural 44 35 40 44 38 41

Punjab

Overall 47 40 44 47 43 45 Urban 63 57 60 56 50 53 Rural 37 21 29 41 25 33

Sindh

Overall 47 35 41 46 34 40 Urban 59 49 54 59 51 55 Rural 45 27 37 47 31 39

NWFP

Overall 47 30 39 48 33 41 Urban 58 51 54 55 41 49 Rural 42 25 34 36 21 29

Balochistan

Overall 44 28 36 39 24 32 Urban 58 56 57 57 54 56 Rural 43 30 37 43 33 38

Pakistan

Overall 47 37 42 46 38 42 Source: PIHS 2001-02 1. Enrollment in katchi is excluded 2. Original figures have been used for 2001-02

Table 2: Net Enrollment Rates at the Middle Level Net Middle Enrollment Rates (%)

1998-99 2001-02

Male Female Overall Male Female Overall Urban 27 26 26 24 32 28 Rural 18 12 16 16 12 14

Punjab

Overall 21 16 19 18 18 18 Urban 29 27 28 22 27 24 Rural 14 3 9 12 4 8

Sindh

Overall 20 13 17 15 12 14 Urban 24 17 21 25 21 23 Rural 15 5 10 15 6 11

NWFP

Overall 16 7 11 16 8 12 Urban 23 13 18 19 13 16 Rural 9 5 7 10 2 6

Balochistan

Overall 11 6 9 11 4 8 Urban 27 25 26 23 29 26 Rural 16 9 13 15 8 12

Pakistan

Overall 19 13 16 17 14 16 Source: PIHS 2001-02 1. Original figures have been used for 1998-99 and 2001-02

- 41 -

Table 3: Access to Primary Schools in Rural Areas (2001-02) Percentage of rural households with school within their community

Type of School In the PSU

Within 0-2 Km

Within 2-5 Km

Within 6-10 Km

Within 11-20 Km

Over 20 Km

Govt. Primary school for girls 67 7 12 5 5 3 Govt. Primary school for boys

85 6 5 2 1 1

Pvt. Primary school for girls 7 6 12 15 16 44 Pvt. Primary school for boys 7 6 13 15 15 44 Pvt. Primary school (co-edu.) 30 6 12 15 16 21 Source: PIHS 2001-02 Table 4: School Quality Indicators

School Infrastructure % of Schools in Sample Adequate building 33 Furniture 51 Textbooks 23 Drinking water 52 Toilets 48 Electricity 32 Separate girls toilet (co-ed facilities) 16 Source: PRHS (2001) Poverty Assessment Report, World Bank

Table 5: Teacher Absenteeism Teacher attendance in schools with classes being held (%)

Schools with no

classes being held (%) Total teachers Male teachers Female teachers

Punjab 3.5 75.3 77.6 72.6 Sindh 28.6 77.5 87.3 11.1 NWFP 22.7 90.8 94.3 80.6 Balochistan 25.0 85.0 90.6 36.4 Pakistan 16.5 81.2 86.2 68.5 Source: PRHS (2001) Poverty Assessment Report, World Bank

Table 6: Main Reasons for Never Attending School (2001-02) Boys of Age 10-18 Girls of Age 10-18 Main Reasons for Never

Attending School Urban Rural Pakistan Urban Rural Pakistan Too expensive 53 36 40 33 25 26 Child not willing 22 30 29 11 6 7 Too far away 2 10 8 3 15 14 Had to help with work 5 5 5 1 2 2 Parents/elders disapproved 1 3 3 36 36 36 Had to help at home 1 2 2 3 5 5 Education not useful 1 1 1 1 1 1 Other 15 12 13 12 10 10 Source: PIHS 2001-02

- 42 -

1

Table 7: Main Reasons for Leaving School (PIHS, 2001-02) Boys of Age 10-18 Girls of Age 10-18 Main Reasons for Leaving

School Urban Rural Pakistan Urban Rural Pakistan Child not willing 39 49 46 28 24 25 Too expensive 35 26 29 24 17 19 Had to help with work 8 6 7 2 2 2 Had to help at home 1 4 3 7 8 8 Parents/elders disapproved 4 2 3 16 18 17 Education completed 1 0 0 1 1 1 Too far away 0 2 2 1 7 6 Other 11 11 11 20 22 21 Source: PIHS 2001-02

Table 8: Percentage of Children Aged 12-23 Months that have been Fully Immunized (based on record and recall)Children Aged 12-23 Months (At Least One Immunization % )

1998-99 2001-02 Male Female Both Male Female Both

URBAN AREAS

64 63 64 70 71 70

Punjab 62 68 64 72 80 76 Sindh 66 54 60 66 63 64 NWFP 82 74 77 81 57 70 Balochistan 51 52 51 34 37 36 RURAL AREAS

47 42 45 48 45 46

Punjab 56 47 52 52 50 51 Sindh 31 24 27 39 26 33 NWFP 49 54 51 52 57 55 Balochistan 35 29 32 22 21 22 OVERALL 52 47 49 53 52 53 Punjab 57 52 55 57 58 57 Sindh 42 35 38 49 40 45 NWFP 52 56 54 56 57 57 Balochistan 36 32 34 24 24 24

Source: PIHS 2001-02

Table 9: Infant Mortality (Per 1000 Live Births)1998-99

(1995-97 Period)2001-02

(1997-99 Period)All Pakistan 89 82Urban 73 65Rural 95 88Male 90 84Female 89 81By Mother’s Education*

No Education 90 89Class 1-4** 56 80Class 5-9 61Class 10 or more 49

Source: PIHS 2001-02

- 43 -

* Figures for 1998-99 have been taken from Poverty Assessment Report (Table 3.10)** For 1998-99, the category “class 1-4” contains percentage for “some education” in Table 3.10 Poverty Assessment Report

Table 10: Female Health Care (Related to Childbirth) – 2001-02 1998-99 2001-02 Pre-natal Consultation (%) All Pakistan 31 35 Rural 22 26 Urban 60 63 Delivery Occurred at Home (%) All Pakistan 82 78 Rural 89 86 Urban 61 55 Delivery Unassisted by Trained Personnel (%)* All Pakistan 40 37 Rural 43 44 Urban 25 20 Post-natal Consultation (%) All Pakistan 9 9 Rural 6 6 Urban 17 16 Source: PIHS 2001-02 *This category includes: Family member/relative, neighbor, TBA, LHW and other

Table 11: Selected Characteristics of Government Health Facilities (2001-02) Facility Characteristic of Health Facility

Dispensary RHC BHU All Three % of facilities with no female staff 53 0 7 21

Average number of patients during past week 14 82 43 38

% of facilities with electricity 47 88 83 72 % of facilities with stock of: Oral re-hydration salts 69 82 76 75 Contraceptives 20 47 59 46 Anti malaria 80 88 85 84 Vaccines 29 94 71 60 Source: PIHS 2001-02

- 44 -

Table 12: Access to Health Facilities in Rural Areas (Community Questionnaire) Percentage of rural households with facilities present in their village Facility Punjab Sindh NWFP Balochistan Pakistan

Government hospital 3 1 3 0 2 Government dispensary 11 16 11 19 12 Private hospital 5 3 1 1 4 Private dispensary 13 16 11 3 13 Private practitioner 21 22 30 6 22 Rural health centre 4 5 13 4 6 Basic health unit 18 11 33 12 18 MCH center 4 3 11 0 5 Family welfare centre 9 3 10 0 8 FP mobile service unit 18 1 9 0 12 Village FP worker 29 16 27 1 24 Community health worker 6 4 4 0 5 Immunization camp 16 3 21 2 13 Lady health worker 80 36 44 21 62 Trained Dai 34 33 38 10 33 Source: PIHS 2001-02

Table 13: Knowledge and Use of Contraception (Married Women, Age 15-49)

Knowledge of Contraceptives (%)

Ever Use Contraceptives (%)

Currently Using Contraceptives (%)

1998-99 2001-02 1998-99 2001-02 1998-99 2001-02 All Pakistan 93 96 23 26 17 19 Rural 91 95 17 21 12 14 Urban 99 99 39 40 29 31 Source: PIHS 2001-02

Table 14: Mean Number of Children Ever Born (Married Women, Aged 15-49) Mean No. of Children Born

1998-99 2001-02

All Pakistan 4.2 4.1 Rural 4.2 4.1 Urban 4.1 4.0 Source: PIHS 2001-02

- 45 -

Table 15: Water and Sanitation (Pakistan-Overall) 1998-99 2001-02 Drinking Water: Piped indoor 22 22

Tap outside house 4 3 Hand/motor pump* 57 61

Open source (river, pond) 18 14 Drains: Underground 14 14 Covered 2 2 Open 34 35

None 50 49 Toilet:

Yes 54 57 No 46 43

Source: PIHS 2001-02 1. Piped indoor-Tap in house 2. Open source-Dug well, River/Canal/Stream and other 3. Hand pump/M. pump includes handpumps both inside and outside, motor pump and tubewell outside the house

Table 16: Water Supply and Sanitation (Pakistan-Rural) 1998-99 2001-02 Drinking Water: Piped indoor 9 8

Tap outside house 3 2 Hand/motor pump* 65 70

Open source (river, pond) 23 20 Drains: Underground 1 1 Covered 1 1 Open 31 33

None 68 66 Toilet: Yes 37 41

No 63 59 Source: PIHS 2001-02 1. Piped indoor-Tap in house 2. Open source-Dug well, River/Canal/Stream and other 3. Hand pump/M. pump includes handpumps both inside and outside, motor pump and tubewell outside the house

Table 17: TPV Results Compliance Percentage

Recruitment Procurement Absenteeism Site Selection Round I March 1998 72% 59% 50% 49%

Round II March 1999 95% 54% 40% 59%

Round III March 2000 62% 50% 30% 81%

Round IV March 2001 64% 36% 31% 62%

- 46 -

Fig.1: Results of TPV Exercises

72

95

6264 59

54 50

36

5040

3031

4959

81

62

0102030405060708090

100

Per

cen

t

Recruitment Procurement AbsenteesimMonitoring

System

SiteSelection

Results of the Last Four Rounds of TPV

1998

1999

2000

2001

- 47 -

Annex 2. Project Costs and Financing

Project Cost by Component (in US$ million equivalent)Project Cost by Component Appraisal

EstimateUS$ million

Actual/Latest Estimate

US$ million

Percentage of Appraisal

Program Support (27 Sub Programs)Participatory Development ProgramCoordination, M&ETA & Institutional Capacity Development

8427.647.128.816.9

51710

4.11.1

61%0%

14%6%

Total Baseline Cost 8520.4 5176.2 60.75%

- 48 -

.

Project Financing by Component (in US$ million equivalent)

Component Appraisal Estimate Actual/Latest EstimatePercentage of Appraisal

Bank Govt. CoF. Bank Govt. CoF. Bank Govt. CoF.Program Support (27 Sub Programs)

218.70 8045.80 163.10 213.00 4918.00 40.00 97.4 61.1 24.5

Participatory Development Program

15.90 0.00 31.20 0.00 0.00 0.00 0.0 0.0 0.0

Coordination, M&E 9.70 0.00 19.10 1.00 3.10 0.00 10.3 0.0 0.0TA & Institutional Capacity Development

5.70 0.00 11.20 0.20 0.90 0.00 3.5 0.0 0.0

DERA 0.00 0.00 0.00 29.80 0.00 0.00 0.0 0.0 0.0

- 49 -

Annex 3. Economic Costs and Benefits

NA

- 50 -

Annex 4. Bank Inputs

(a) Missions:Stage of Project Cycle Performance Rating No. of Persons and Specialty

(e.g. 2 Economists, 1 FMS, etc.)Month/Year Count Specialty

ImplementationProgress

DevelopmentObjective

Identification/PreparationMay 1996 6 SOC.SECTOR TEAM LEADER

(1); EDUCATION SPECIALIST (1) HEALTH SPECIALIST (1); POPULATION SPECIALIST (1); SOCIAL SECTOR SPECIALIST (1); PROCUREMENT SPEC. (1)

June 1996 6 SOC.SECTOR TEAM LEADER (1); EDUCATION SPECIALIST (1) HEALTH SPECIALIST (1); POPULATION SPECIALIST (1); SOCIAL SECTOR SPECIALIST (1); PROCUREMENT SPEC. (1)

November 1996 7 SOC.SECTOR TEAM LEADER (1); EDUCATION SPECIALIST (1) HEALTH SPECIALIST (1); POPULATION SPECIALIST (1); SOCIAL SECTOR SPECIALIST (1); PROCUREMENT SPEC. (1); FIN. MGT SPECIALIST (1)

December 1996 7 SOC.SECTOR TEAM LEADER (1); EDUCATION SPECIALIST (1) HEALTH SPECIALIST (1); POPULATION SPECIALIST (1); SOCIAL SECTOR SPECIALIST (1); PROCUREMENT SPEC. (1); FIN. MGT SPECIALIST (1)

Appraisal/NegotiationJanuary 1997 7 SOC.SECTOR TEAM

LEADER (1); EDUCATION SPECIALIST (1) HEALTH SPECIALIST (1); POPULATION SPECIALIST (1); SOCIAL SECTOR SPECIALIST (1); PROCUREMENT SPEC. (1); FIN. MGT SPECIALIST (1)

10/11/1997 7 SOC.SECTOR TEAM LEADER (1); EDUCATION SPECIALIST

- 51 -

(1) HEALTH SPECIALIST (1); POPULATION SPECIALIST (1); SOCIAL SECTOR SPECIALIST (1); PROCUREMENT SPEC. (1); FIN. MGT SPECIALIST (1)

Supervision08/04/1998

12/09/1998

05/17/1999 9 SOC. SEC. TEAM LEADER (1); EDUCATION SPECIALIST (1); HEALTH SPECIALIST (1); POPULATION SPECIALIST (1); SR. SOCIAL SECTOR SPEC (1); HR SPECIALIST (1); SOC. SECTOR SPECIALIST (1); PRINCIPAL ECONOMIST (1); FINANCIAL MGT. CONSULT (1)

S S

08/15/1999 5 SOCIAL SCTR TEAM LDR (1); SR SOC. SECTOR SPEC. (1); SR. EDUCATION SPEC. (1); PRINCIPAL ECONOMIST (1); FINANCIAL MGT SPEC. (1)

S U

02/22/2000 10 SOCIAL SECTOR TEAM LDR (1); SR SOCIAL SECTOR SPEC. (1); SR EDUCATION SPEC. (1); SR. ECONOMIST (1); OPERATIONS OFFICER (1); FIN. MANAGEMENT SPEC. (1); PRINCIPAL HD SPECIALIST (1); PRINCIPAL ECONOMIST (1); COMMUNITY DEVELOPMENT SPEC. (1); PROGRAM ASST. (1); SR. WATER & SANITATION SPECIALIST (1)

05/23/2000 10 SOCIAL SECTOR TEAM LDR (1); SR SOCIAL SECTOR SPEC. (1); SR EDUCATION SPEC. (1); SR. ECONOMIST (1); OPERATIONS OFFICER

U U

- 52 -

(1); FIN. MANAGEMENT SPEC. (1); PRINCIPAL HD SPECIALIST (1); PRINCIPAL ECONOMIST (1); COMMUNITY DEVELOPMENT SPEC. (1); PROGRAM ASST. (1); SR. WATER & SANITATION SPECIALIST (1)

10/23/2000 Mid Term Review

12 TASK TEAM LEADER/SR. SOCIAL SECTOR SPEC. (1); LEAD SPECIALIST (1); OPERATIONS OFFICER (1); SR. ECONOMIST (1); SR. EDUCATION SPECIALIST (2); SR. HEALTH SPECIALIST (1); REPRODUCTIVE HEALTH SPECIALIST (1); PRINCIPAL ECONOMIST (1); FINANCIAL MANAGEMENT SPECIALIST (1); SR. WATER & SANITATION SPECIALIST (1); SR. COMMUNITY DEVELOPMENT SPECIALIST (1)

U U

03/15/2001 7 TEAM LEADER/SR. SOCIAL SECTOR SPEC. (1); HEALTH SPECIALIST (2); SR. EDUCATION SPECIALIST (2); OPERATIONS OFFICER (1); FINANCIAL MANAGEMENT SPEC. (1)

S U

09/15/2001 8 TASK LEADER (1); SR. HEALTH/REPROD.SPEC (1); SR. EDUCATION SPEC. (1); SR. PROCUREMENT SPEC. (1); LEAD, HD (1); SR. HEALTH SPEC. (1); SR. FIN. MGT. SPEC. (1); DISBURSEMENT ANALYST (1)

S U

02/12/2002 7 SR. EDUC. SPECIALIST (3); SR. HEALTH SPECIALIST (1); CONSULTANT (1); LEAD OPERAT. OFFICER (1); LEAD HUMAN DEV. SPEC. (1)

S U

- 53 -

ICR01/30/2003 8 SR. EDU. SPECIALIST (2);

SR. HEALTH SPECIALIST (1); SR. ECONOMIST (1); SR. FIN. MG. SPEC (1); CONSULTANT (2); SR. WATER & SANITATION SPECIALIST (1)

(b) Staff:

Stage of Project Cycle Actual/Latest EstimateNo. Staff weeks US$ ('000)

Identification/Preparation 267.0 759.5Appraisal/Negotiation 116 343Supervision 343.0 2079.6ICR 20 100Total 746 3282.1

- 54 -

Annex 5. Ratings for Achievement of Objectives/Outputs of Components(H=High, SU=Substantial, M=Modest, N=Negligible, NA=Not Applicable)

RatingMacro policies H SU M N NASector Policies H SU M N NAPhysical H SU M N NAFinancial H SU M N NAInstitutional Development H SU M N NAEnvironmental H SU M N NA

SocialPoverty Reduction H SU M N NAGender H SU M N NAOther (Please specify) H SU M N NA

Private sector development H SU M N NAPublic sector management H SU M N NAOther (Please specify) H SU M N NA

- 55 -

Annex 6. Ratings of Bank and Borrower Performance

(HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HU=Highly Unsatisfactory)

6.1 Bank performance Rating

Lending HS S U HUSupervision HS S U HUOverall HS S U HU

6.2 Borrower performance Rating

Preparation HS S U HUGovernment implementation performance HS S U HUImplementation agency performance HS S U HUOverall HS S U HU

- 56 -

Annex 7. List of Supporting Documents

A review of Selected RWSS Transferred to, Operated and Maintained by the Communities (October 1998)Aide Memoires : SAPP-II Preparation Mission (1996 - 1998)Aide Memoires : SAPP-II Supervision Missions (1999 – 2002)Country Assistance Strategy, 2001 (World Bank Document)Current Demographic Situation Sector Problems and Issues and The Way Forward – Population Sector in Pakistan (October 2001)Devolution in Health Progress and Prospects – Local Government Ordinance 2001 (October 2001)Education Sector Reforms – Action Plan, Ministry of Education, Government of Pakistan (2001)Health Policy, Ministry of Health, Government of Pakistan (2001)Household Income Expenditure Surveys, Government of PakistanIncluding the Poor in the PRSP process by Consulting the Poor at the Grass Roots - Rural Support Programme Network (RSPN), supported by DFID (2003)Interim Poverty Reduction Strategy Paper, Government of Pakistan (2001)Memorandum and Recommendation of the President for the Sindh Structural Adjustment Credit, World Bank (2002)Memorandum and Recommendation of the President for the NWFP Structural Adjustment Credit, World Bank (2002)Pakistan Integrated Household Survey (PIHS), Federal Bureau of StatisticsPakistan Poverty Assessment Report, World Bank ReportPakistan Rural Household Survey (PRHS), Government of PakistanProject Appraisal Document, Second Social Action Program Project, World Bank Document (1998)Project Status Reports, World Bank Reports of Federal SAP Secretariat, Planning & Development Division, Government of Pakistan (2000 - 2001)Reports on Provincial Stakeholder Workshops (Punjab, Sindh, NWFP)Report on Workshop with Senior Government Officials at the Federal LevelSeminar Report on: Pakistan National Symposium on International Experiences with Decentralization and Education, World Bank (Quetta, June 2001)SAP Management and SAP Institutional Arrangements (November 2000)SAPP-II Mid-Term Review – Performance Indicators 1991-1999 (Final Reports - all provinces and areas) - Oxford Policy ManagementThird Party Validation: Report on Governance Issues, Government of Pakistan (March 2001)

Multi-Donor Support Unit (MSU) Reports:Analysis of the Last Four Rounds: Third Party Validation (TPV) Report: Governance Issues (March 2001)Analytical Papers – Analysis of Reimbursements under SAPP-II (May 2000)Bunyad (NGO) Report on School Management Committees, PunjabCase Studies of School Management Committees conducted for all provinces (World Bank Study)Current Status, Issues and Future Strategies – Water Supply and Sanitation in Pakistan (October 2001)Elementary Education in Pakistan – Current Status, Issues and Future Strategies (October 2001)Governance in the Social Sector in Pakistan – Analysis, Issues and Recommendations (October 2001)Report of National Technical Group Meeting (TGM): Devolution and Decentralization: Implication for the Education Sector – February 2001, IslamabadReport of the Participatory Development Program (PDP) Awareness Seminar - Sukkur (MSU Report)SAP Report on TGM on Devolution and Decentralization: Implication for the Education Sector (June 2001)SAP Sector Statistics 1992/93 – 1998/99 (April 2000)

- 57 -

Seminar on Community Participation in School Education (December 1998)Staff Turnover in SAP Implementing Department (May 2000)Strengthening School Councils in Punjab (MSU Report)Strengthening School Management Committees in Sindh (MSU Report)Technical Group Meeting (TGM) Report on Community Participation in Support of Strengthening District Health Systems (March 2001)Workshop Report on Strengthening School Management Committee in Sindh (January 2002)

- 58 -

Annex 8. Beneficiary Survey Results

During 2000 – 2003, several beneficiary assessment surveys and dialogues with the community were conducted by the Bank, the Government, civil society organizations and other donors such as DfID. These were conducted in the context of preparation of the Government’s Poverty Reduction Strategy Paper (PRSP) including the DfID supported consultations conducted by the Rural Support Network: grassroots consultations for preparation of the Bank’s Country Assistance Strategy (CAS); Bank’s Poverty Assessment Report; Case studies for assessing role and functions of School Management Committees (SMCs) etc. The findings of these surveys and consultations were used to assess the impact of SAP from the beneficiary perspective.

SAPP-II was not a stand alone investment project, and was focused on improving access and quality of the social services delivery provided through the public sector. The communities consulted highlighted the responses of the ordinary citizen and rural poor. The breakdown of governance was critically noted in these consultations where poor felt marginalized from the limited or no access to government. The communities expressed their priorities for human development, especially education and health. One major community demand was for water, especially during the time when there was a drought situation in the country – especially in Sindh and Balochistan provinces. But overall education was the number one priority, followed by health care. In almost all beneficiary consultations, the condition of government facilities was considered very weak, with many facilities cited as being non functional due to lack of staff, especially teachers and health workers. Lack of female facilities, and women’s limited access to health and pre and ante natal care was consistently raised as an issue. It was evident from these consultations that the decade of SAP (supported by donors through Social Action Program Projects I and II) did not lead to visible improvements in delivery of these basic social services pointing towards the urgent need to not only fix service delivery but also to fix the underlying systemic governance issues.

Three key aspects stand out in these diverse consultations and dialogue:

a) Interestingly, while the rural poor expressed low confidence in the quality of public service delivery they still have high expectations that Government will and should provide these services, and there is high demand for quality education and health services. This demonstrates the urgency of fixing basic service delivery, which is as important a need now as it was when SAP was put in place in the early nineties.

b) The services meant for the poor are not in fact reaching them, and there is very little, if any, recourse that the poor have to reach out to the decision makers. In this respect, there was some hope that the local governments, being closer to the people, would improve service delivery conditions.

c) The importance of community involvement, through local community groups/organizations was considered critical for achieving greater accountability in the system. In case studies conducted for SMCs, it was evident that support organizations play a significant role in helping communities to mobilize and in empowering them.

The range of beneficiary consultations and dialogue with communities is available in several reports listed in the section on documentation consulted.

- 59 -

Annex 9. Stakeholder Workshop Results

Executive Summary of Federal and Provincial Focus Group Workshops

IntroductionTo facilitate the preparation of a Learning ICR for SAPP-II, focus group discussions were held with policy makers at the federal level and district officials and field-level staff (doctors, teachers, LHWs, Lady Health Visitors [LHVs] and community mobilization staff) from Punjab, NWFP and Sindh. The purpose underlying these deliberations were to assess the successes and failures of SAPP-II in achieving its objectives, review the progress in improving social service delivery, determine the sustainability of initiatives launched under SAPP-II, identify the reasons for sub-optimal performance of various sub-programs, evaluate project impact on governance and institutional reforms and elicit the views of policy makers and implementation staff on donor management and supervision of the project. This summary tries to encapsulate the direction, content and tenor of this dialogue, which provided useful insights for incorporation in the ICR.

1. Impact of SAPP-II on Trends in Social Indicators and Outcomes

Impact of SAPParticipants generally agreed that SAP had been successful in focusing government attention on designing efficient and sustainable water supply systems, and on improving the health indicators through recognition of the primacy of preventive health care in contrast with the greater emphasis on curative health services in previous health programs of the government. Participants appreciated SAP’s focus on the provision of elementary education and noted that while there had been some improvement in the delivery of educational services, overall education outcomes were not up to expectations. However, while there had been some improvement in the delivery of social services, the benefits that had accrued were not commensurate with the investments made and the funds expended on the program. The appointment of female staff (especially that of female paramedics) in health facilities was perceived to be a major reason for enhanced access.

Participants reported that during SAP the teaching environment had greatly improved as a result of the upgrading of physical infrastructure – involving electrification of schools, construction of additional classrooms and the provision of missing facilities (such as boundary walls and latrines). District level officials regarded the provision of drinking water as a basic public service and appreciated SAP’s focus on the rural water supply sector. However, some were of the opinion that adequate attention had not been given to the importance of water quality and the provision of sewerage related infrastructure which resulted in health problems related to water borne diseases.

Awareness and Ownership of SAP ObjectivesThere was a perception that ownership of SAPP-II remained weak, largely because of frequent transfers of administrators and the failure to appreciate the need for continuously building ownership through sustained dialogue. Awareness of SAPP-II objectives and scope was believed to be particularly poor at the district level

To ensure the achievement of SAP goals and enhance effective implementation, most respondents emphasized the need for an ongoing concerted effort to generate understanding of the social sector reform agenda and associated objectives at all levels of government.

Teachers and district officials supported recent initiatives like girls' scholarships and free textbooks, and reported that these interventions had been relatively successful in inducing a change in parental attitudes

- 60 -

towards education. In their opinion, parents regarded scholarships/cash rewards as the most powerful incentive for sending their children to school.

Intra-sectoral ImbalancesSome stakeholders highlighted the issue of intra-sectoral imbalances created by SAPP-II. In their view the enhanced focus on elementary education and primary health care was achieved at the expense of the higher education and tertiary health care services, which, for instance, accentuated the issues of limited access to education at the secondary level for those graduating from elementary grades.

Coordination between line departmentsIt was generally accepted that coordination between the line departments remained weak throughout SAPP-II.

Coverage of health servicesAlthough some participants were of the opinion that there had been a marked improvement in the coverage of primary health services during SAPP-II, district officials and field level staff were of the view that since a significant proportion of the health facilities were located at long distances from settlements it was difficult for some communities to gain access to health services. Hence, despite greater awareness of the importance of primary health care among the target population, the utilization of health facilities continued to be lower than what had been hoped for.

Several field personnel reported significant progress made in enhancing awareness and in the use of family planning methods. As a result, fertility rates had declined. District and field-level staff reported a substantial increase in immunization coverage and a perceptible decrease in the IMR and MMR. These positive trends in the health indicators were attributed to recruitment of female staff and LHVs at BHUs, and their improved presence at the health outlets and better, albeit moderate, availability of medicines and supplies.

Field staff maintained that Lady Health Workers had played an instrumental role in convincing expectant mothers of the need for ante-natal care and reported increases in the percentage of women receiving ante-natal care services at the health facilities. Lady doctors and LHVs were of the opinion that there was a gradual increase in the proportion of deliveries conducted by trained personnel.

2. Progress in Addressing Cross-Sectoral Issues

SustainabilityAlthough direct financial assistance to PTAs/School Management Committees/Councils from provincial governments continues to be available after devolution, district officials felt that it was too early to pass a judgement on the sustainability of the SAPP-II focus on social sectors and on enhanced allocations for non-salary inputs. Concerns were expressed on the durability of some initiatives and institutions such as the PTAs/SMCs/SCs and Water User Associations.

Most field level staff and district officials claimed that barring a minority of need-based schemes that communities would continue to maintain, the balance, particularly the mechanized drainage schemes, would eventually be abandoned. These were considered unsustainable since the communities did not have the financial wherewithal to bear the high electricity tariffs.

Some participants felt that SMCs may either be wound up or may wither away if funding was discontinued by the provincial governments, since district governments did not have adequate resources to fund SMCs.

- 61 -

Institutional capacityIt was generally accepted that line departments did not have adequate planning, implementing and monitoring capacity. This capability was particularly weak in matters related to financial management. Nor had this capability been developed on a timely and durable basis under SAPP-II. Resultantly, the same problems occurred repeatedly. Based on SAPP-II experience, these capacities were being improved.

MonitoringDistrict officials and field staff were of the view that effective monitoring required the provision of transportation facilities or travel and per diem allowances (TA/DA) for district level supervisory staff. They maintained that lack of transportation or inadequate budgetary provisions for TA/DA payments to supervisory staff had constrained their oversight functions.

Sustainability of Water Supply SchemesDistrict officials of one province maintained that notwithstanding the moratorium on new rural water supply schemes that was in operation for the most part of SAPP-II, those implemented under KPP and other programs continued to add to the stock of rural water supply systems executed without community participation. This had made it difficult to transfer the O&M responsibilities to the beneficiary communities.

Staffing

Contract-based RecruitmentParticipants agreed that the recruitment of local teachers on facility specific contracts had improved the functioning of primary schools. However, some were of the view that since pay incentives for contract employees were inadequate and their career paths and prospects uncertain, they feared that these teachers would be de-motivated, which would affect the quality of service provision.

TransfersThe frequent transfers of key officials responsible for the planning, implementation and supervision of SAPP-II adversely affected program performance through loss of consistent leadership, institutional memory and understanding of project objectives.

Ban on RecruitmentOwing to the continued ban on recruitments many posts essential for operationalizing constructed facilities remained vacant for long periods, although, under SAPP-II the social sectors were exempted from these bans. District officials were of the opinion that the lack of qualified staff in rural health facilities in particular was severely restricting access to health-care. In their view, doctors needed to be recruited on more attractive pay packages to serve in the rural areas.

DecentralizationParticipants from the federal and district governments recognized SAPP-IIs positive impact on decentralization of administrative functions to lower tiers as they regarded devolution as a more powerful institutional arrangement for improving the quality of services in the SAP sectors of education and health.

District and implementation staff claimed that Tehsil Municipal Administration (TMAs) did not have adequate technical capacity to maintain the water supply schemes handed over to them. Participants noted that the TMAs were not adhering to the uniform policy in the RWSS sector and hence expressed their concerns on post devolution sustainability of community participation in designing, implementing and

- 62 -

managing water supply schemes.

Lack of flexibility in utilization of fundsDistrict officials reported that, on the one hand, the implementation agencies had not been empowered by the provincial government to re-appropriate funds from one budget head to another and, on the other hand, lack of funds had made it difficult to carry out repairs and maintenance of the existing stock of infrastructure.

Third Party Validation (TPV) ExercisesParticipants expressed doubts about the quality of, and the manner in which, Third Party Validation exercises were conducted and claimed that the results of the validation assignments were not reported objectively.

3. Social Sector Budgets and Timely Availability of Funds

Release of FundsThe delays in the release of funds to lower-level field formations and executing agencies affected implementation. There were also inadequate budgetary allocations for repairs and maintenance and other non-salary inputs (for consumables). They also reported that the flow of development funds for the construction of new infrastructure was now better.

4. Community Participation in Delivery of Social Services

EducationParticipants generally held the view that the effectiveness of SMCs was constrained by their inability to hold teachers accountable for chronic absenteeism. Education Department staff maintained that teacher absenteeism continued to be a problem because of the lack of accountability of teachers to SMCs.

Some district officials and field level-staff noted that SMCs established during SAPP-II had played a meaningful role in improving the infrastructural facilities and in the availability of instructional materials. District officials from Sindh reported the efforts of SMCs to create ownership of educational facilities among communities and claimed that through the efforts of these committees the enrollment of girls at primary level had increased and interference in the functioning of schools had been reduced.

They considered it was unrealistic to expect community members (most of whom were illiterate) to take an active interest in the management and supervision of schools when appropriate steps had not been taken to educate them about their roles and responsibilities as members of these committees. They attributed this low turnout to lack of community interest, especially rural females, in managing schools and their inability to take time out from their pressing daily schedules.

Field staff also expressed their concern on the restrictions faced by SMCs on the utilization of funds. They were of the view that SMCs should have greater flexibility on the use of funds allocated to them. At present, these funds can be used for repairs and maintenance of school buildings and purchase of instructional materials. Some participants were of the opinion that the SMCs should be empowered to use these funds for hiring teachers on contract. HealthParticipants were of the opinion that the role of communities in monitoring delivery of health services would always be limited owing to their inability to devote time for such activities. District officials stated

- 63 -

that the health committees had not been particularly successful in supervising service delivery. The members were more interested in matters related to staff postings and transfers. Field staff also stated that wherever such committees had been formed, they had greatly facilitated their work. They reported that the health committees established with the efforts of Lady Health Workers had membership of religious leaders, school teachers, mid-wives and these comprised community members who were willing to work voluntarily to enhance awareness and promote use of primary health care services and family planning methods.

5. Participatory Development Program (PDP) and NGO-Government Collaboration in Social Service Delivery

Some policy makers were of the view that the targeted timeframe for institutionalizing the involvement of NGOs in service provision was ambitious. The effectiveness of the participatory approach was also undermined by the attitude of the provincial line departments as they viewed NGO participation in the delivery of social services as an encroachment on their functional domain. District officials also raised concerns about the lack of flexibility in the views of donors on the concept and manner of community participation through NGOs.

6. SAPP-II Project Financing Mechanism-SOEs, Financial Management Issues, Disbursements and Procurement

Statement of ExpendituresSeveral participants dwelt at length on the difficulties encountered in preparing SOEs. They complained that although donors were only meeting a small share of total government expenditure on the SAP sectors, inordinate time and effort was taken up by activities connected with the submission of SOEs for reimbursements.

Financial ReportingThe financial accounting and reporting mechanisms continued to remain weak. SAP coordinators faced difficulties in disaggregating expenditures by category (salary/non-salary) to be able to appropriately classify SAP expenditures for finalizing reimbursement claims.

ProcurementThe procurement rules of donors were not fully understood and confusion remained throughout SAPP-II. Field staff complained about donor inflexibility with respect to procurement rules, claiming government procedures were equally efficient and more cost-effective.

7. SAPP-II Management and Donor Supervision

Some participants remarked that donor supervision missions tended to focus on disbursement issues rather than on institutional reform objectives of SAPP-II. Participants criticized donor micro-management of the project and highlighted the inadequate emphasis placed by donors on the monitoring of performance indicators and on the assessment of outcomes.

Many participants were of the opinion that whereas both government and donors wanted to enhance expenditures on the social sectors they were not paying enough attention to the identification and resolution of problems constraining effective project implementation. Some district officials also reported that since funds had not been allocated for acquiring land for setting up new schools, there were several cases where notables of the village/community, who had donated the plot, interfered in the running of the school creating

- 64 -

operational difficulties.

Some district officials were of the view that donors (or consultants hired by them) ought to have monitored program execution by undertaking field visits and by regularly consulting district and field level staff. This would have enabled removal of bottlenecks, thereby ensuring timely resolution of implementation problems.

- 65 -

- 66 -