document of the world bank · 2016. 7. 14. · cte technical school council (consejo técnico...

Document of The World Bank

Report No: ICR00001629

IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-79480)

ON A

LOAN

IN THE AMOUNT OF US$220 MILLION

TO THE

UNITED MEXICAN STATES FOR A

SCHOOL-BASED MANAGEMENT PROJECT

IN SUPPORT OF THE SECOND PHASE OF THE

SCHOOL-BASED MANAGEMENT PROGRAM

January 29, 2015

Education Global Practice Latin America and the Caribbean Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

i

CURRENCY EQUIVALENTS

(Exchange Rate Effective January 26, 2015)

Currency Unit MX$1.00 = US$0.07 US$1.00 = MX$14.63

FISCAL YEAR

January 1 – December 31

ACRONYMS

AEF Federal Education Authority (Autoridad Educativa Federal) AEL Local Education Authority (Autoridad Educativa Local) AGE Support for School Management (Apoyos a la Gestión Escolar) APL Adaptable Program Loan CEMI Evaluation, Monitoring and Information Unit at the Directorate of School

Management (Coordinación de Evaluación, Monitoreo e Información de la DGAG)

CEPS School Council of Social Participation (Consejos Escolares de Participación Social)

CONAFE National Council for Education Development (Consejo Nacional de Fomento Educativo)

CONAPASE National Council of Social Participation in Education (Consejo Nacional de Participación Social en la Educación)

CONEVAL National Council for the Evaluation of Social Development Policy (Consejo Nacional de la Evaluación de la Política de Desarrollo Social)

CPS Country Partnership Strategy CTE Technical School Council (Consejo Técnico Escolar) CTPS Social Participation Technical Council (Consejo Técnico de Participación

Social) DGAG Directorate of School Management (Dirección General Adjunta de Gestión) DGDGIE General Directorate of School Management Development and Education

Innovation (Dirección General de Desarrollo de la Gestión e Innovación Educativa)

DGE General Directorate of Evaluation (Dirección General de la Evaluación) ENLACE National Evaluation of Academic Achievement in Schools (Evaluación

Nacional del Logro Académico de Centros Escolares) FEEC State Trust Funds (Fideicomisos Estatales de Escuelas de Calidad) FM Financial Management FNEC National Trust Fund (Fideicomiso Nacional de Escuelas de Calidad) FY Fiscal Year GDP Gross Domestic Product GoM Government of Mexico HDI Human Development Index

ii

IBRD International Bank for Reconstruction and Development IDA International Development Association IERR Internal Economic Rate of Return IMF International Monetary Fund INEE National Institute for the Evaluation of Education (Instituto Nacional para la

Evaluación de la Educación) INEGI National Institute of Statistics and Geography (Instituto Nacional de

Estadística y Geografía) IPP Indigenous Peoples Plan (Plan de Pueblos Indígenas) LA Loan Agreement LAC Latin America and the Caribbean M&E Monitoring and Evaluation MGEE Strategic Model of School Based Management (Modelo de Gestión Escolar

con Enfoque Estratégico) MIS Management Information System NAFIN National Financial Development Bank (Nacional Financiera, S.N.C., I.B.D) NDP National Development Plan (Plan Nacional de Desarrollo) OECD Organization for Economic Co-operation and Development PAD Project Appraisal Document PAT Annual Work Plan (Programa Anual de Trabajo) PDO Project Development Objective PEC Quality Schools Program (Programa Escuelas de Calidad) PETC Full Time Schools Program (Programa Escuelas de Tiempo Completo) PETE Strategic School Transformation Plan (Plan Estratégico de Transformación

Escolar) PISA Program for International Student Assessments PRF Project Results Framework PSE Education Sector Program (Programa Sectorial de Educación) SBM School Based Management SEB Subsecretariat of Basic Education (Subsecretaría de Educación Básica) SEP Ministry of Public Education (Secretaría de Educación Pública) SHCP Ministry of Finance and Public Credit (Secretaría de Hacienda y Crédito

Público) SIGED Mexico’s Education Management Information System (Sistema de

Información y Gestión Educativa) SIPEC Information System for the Quality Schools Program (Sistema de Información

del Programa Escuelas de Calidad) SPD Professional Teaching Service (Servicio Profesional Docente) SNTE National Teacher’s Union (Sindicato Nacional de Trabajadores de Educación)

Vice President: Jorge Familiar

Country Director: Gerardo M. Corrochano Practice Manager: Reema Nayar

Project Team Leader: Rafael E. De Hoyos Navarro ICR Team Leader Rafael E. De Hoyos Navarro

iii

UNITED MEXICAN STATES SCHOOL-BASED MANAGEMENT PROJECT

IN SUPPORT OF THE SECOND PHASE OF THE SCHOOL-BASED MANAGEMENT PROGRAM

CONTENTS Data Sheet

A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph

1. Project Context, Development Objectives and Design ............................................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................................. 4 3. Assessment of Outcomes ............................................................................................................ 9 4. Assessment of Risk to Development Outcome ......................................................................... 17 5. Assessment of Bank and Borrower Performance ..................................................................... 17 6. Lessons Learned........................................................................................................................ 19 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ........................... 20 Annex 1. Project Costs and Financing .......................................................................................... 21 Annex 2. Outputs by Component.................................................................................................. 22 Annex 3. Economic and Financial Analysis ................................................................................. 25 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............................. 27 Annex 5. Beneficiary Survey Results ........................................................................................... 29 Annex 6. Stakeholder Workshop Report and Results ................................................................... 30 Annex 7. Summary of Implementing Agency’s ICR .................................................................... 31 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................................... 41 Annex 9. List of Supporting Documents ...................................................................................... 42 MAP .............................................................................................................................................. 45

iv

A. Basic Information

Country: Mexico Project Name: School Based Management (APL II)

Project ID: P115347 L/C/TF Number(s): IBRD-79480

ICR Date: 12/15/2014 ICR Type: Core ICR

Lending Instrument: APL Borrower: GOVERNMENT OF MEXICO

Original Total Commitment:

USD 220.00M Disbursed Amount: USD 220.00M

Revised Amount: USD 220.00M

Environmental Category: C

Implementing Agencies: Ministry of Public Education (SEP)

Cofinanciers and Other External Partners: B. Key Dates

Process Date Process Original Date Revised / Actual

Date(s)

Concept Review: 05/27/2009 Effectiveness: 12/20/2010 12/20/2010

Appraisal: 02/22/2010 Restructuring(s): 04/25/2013 09/20/2013 06/26/2014

Approval: 06/17/2010 Mid-term Review: 05/17/2012 05/17/2012

Closing: 06/30/2013 06/30/2014 C. Ratings Summary C.1 Performance Rating by ICR

Outcomes: Moderately Satisfactory

Risk to Development Outcome: Low or Negligible

Bank Performance: Moderately Satisfactory

Borrower Performance: Moderately Satisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings

Quality at Entry: Moderately Unsatisfactory

Government: Satisfactory

Quality of Supervision: Moderately Satisfactory Implementing Agency/Agencies:

Moderately Satisfactory

Overall Bank Performance:

Moderately Satisfactory Overall Borrower Performance:


v

C.3 Quality at Entry and Implementation Performance IndicatorsImplementation

Performance Indicators

QAG Assessments (if any)

Rating

Potential Problem Project at any time (Yes/No):

No Quality at Entry (QEA):

None

Problem Project at any time (Yes/No):

No Quality of Supervision (QSA):

None

DO rating before Closing/Inactive status:


D. Sector and Theme Codes

Original Actual

Sector Code (as % of total Bank financing)

Pre-primary education 30 30

Primary education 60 60

Secondary education 10 10

Theme Code (as % of total Bank financing)

Education for all 30 30

Managing for development results 30 30

Participation and civic engagement 40 40 E. Bank Staff

Positions At ICR At Approval

Vice President: Jorge Familiar Pamela Cox

Country Director: Gerardo M. Corrochano Gloria M. Grandolini

Practice Manager: Reema Nayar Chingboon Lee

Project Team Leader: Rafael E. De Hoyos Navarro Ricardo Rocha Silveira

ICR Team Leader: Rafael E. De Hoyos Navarro

ICR Primary Author: Abril Ibarra Vicente Garcia

F. Results Framework Analysis Project Development Objectives (from the Loan Agreement) The objective of the Project is to strengthen PEC by increasing overall coverage and social participation in Eligible Schools while putting greater emphasis on public marginalized schools and on the indigenous population, as well as a reorientation of the direct support to schools to improve public schools’ internal efficiency and learning outcomes.

vi

Revised Project Development Objectives (as approved by original approving authority) N/A (a) PDO Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised

Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1 : Percentage of basic and special education schools participating in PEC Value (quantitative or Qualitative)

18 (original) 20.30 (revised)

22.00 23.00 23.85

Date achieved 06/30/2010 06/30/2013 06/30/3014 06/30/2014

Comments (incl. % achievement)

Target exceeded. Actual value reflects data from 2012-13 school year. Indicator revised in the April 2013 restructuring. The original wording of this indicator was “number of schools participating in PEC as percentage of the total number of basic education schools.” In addition, the original Project Results Framework (PRF) had an indicator related to the “total number of basic education schools participating in PEC which was expected to increase 25 percent by 2013 (from 39,180 in 2009 to around 50,000 in 2013).”This target was met since the total number of PEC schools was 50,624 in the school year 2013-14 (in 2009, the baseline was revised to 40,790).

Indicator 2 : Percentage of school community members who participate in the design of the PETE Value (quantitative or Qualitative)

74.20 80.00 75.00 72.70

Date achieved 06/30/2010 06/30/2013 06/30/2014 06/30/2014


Target not met. Indicator revised in the April 2013 restructuring. The original wording of this indicator was “commitment to the goals and activities of the school plan (PETE) as expressed by parental participation in PETE design and adjustment.” The revised target aimed to keep parental participation constant (with an aimed increase of 0.8 percent). Although the target was not met, the 1.5 percent decline from the baseline was not statistically significant, indicating that parental participation was maintained. Actual value reflects data from 2012-13 school year.

Indicator 3 : Percentage of PEC schools in highly marginalized and marginalized areas Value quantitative or Qualitative)

40.00 (original/preliminary)38.4 (revised/final)

50.00 55.00 51.35

Date achieved 06/30/2010 06/30/2013 06/30/2014 06/30/2014


Original target exceeded and 93 percent achieved against revised target. Chiapas and Veracruz (states with high percentage of schools located in highly marginalized and marginalized areas) were not able to participate in the last year of the Project becausethey did not provide their counterpart funding conditioning the delivery of the federal funds. However, the overall number of schools located in highly marginalized areas that receive support to implement their school improvement plans in Mexico has increased with the new targeting and approach of programs such as Programa Escuelas de Tiempo Completo (PETC) which, among others, now provide direct support to schools targeting those in marginalized and highly marginalized areas, following the PEC school based management (SBM) model. Although the disaggregated number of PETC schools located

vii



approval documents)

Formally Revised

Target Values



in highly marginalized and marginalized areas is not yet available for the school year 2013-14, the overall number of PETC schools expanded from 6,708 in 2012-13 to 15,349 in 2013-14 and up to 23,182 to date.

Indicator 4 : Percentage of Indigenous modality schools participating in PEC Value (quantitative or Qualitative)

7.0 (original) 15.40 (revised)

15.00 19.30 13.5 (original) 18.00 (revised)

Date achieved 06/30/2010 06/30/2013 06/30/2014 06/30/2014


90 percent of original target met and revised target 93 percent met. Indicator revised inthe April 2013 restructuring. The original wording of this indicator was “percentage of PEC schools with indigenous students,” which accounted for schools with one or more indigenous students (actual original reflects data from 2011-12 school year). Chiapas and Veracruz (states with a high percentage of schools located in indigenous communities) were not able to participate in the last year of the Project because they did not provide their counterpart funding conditioning the delivery of the federal funds. However, the overall number of schools located in indigenous communities that receive support to implement their school improvement plans in Mexico has increased with the new targeting and approach of programs such as PETC which, among others, now provide direct support to indigenous schools following the PEC SBM model. Although the disaggregated number of PETC schools located in highly marginalized and marginalized areas is not yet available for the school year 2013-14, the overall number of PETC schools expanded from 6,708 in 2012-13 to 15,349 in 2013-14 and up to 23,182 to date.

Indicator 5 : Percentage of school grants spent on education quality inputs Value quantitative or Qualitative)

59.70 N/A 66.86 74.54

Date achieved 06/30/2010 6/30/2014 06/30/2014 Comments (incl. % achievement)

Target exceeded. Indicator added in the April 2013 restructuring.

(b) Intermediate Outcome Indicator(s)



approval documents)

Formally Revised Target

Values



Indicator 1 : Percentage of community members that are familiar with PETE Value (quantitative or Qualitative)

76.80 82.00 84.00 83.50

Date achieved 06/30/2010 06/30/2013 06/30/2014 06/30/2014 Comments (incl. % achievement)

Original target exceeded and formally revised target 99 percent achieved. This indicator was revised in the April 2013 restructuring. The original wording for this indicator was

viii



approval documents)


Values



“Percentage of community members that know the PETE.” Actual value reflects data from 2012-13 school year.

Indicator 2 : Percentage of community members that observe that parents are informed about student performance.

Value (quantitative or Qualitative)

99.00 99.00 99.80

Date achieved 06/30/2010 06/30/2013 06/30/2014 Comments (incl. % achievement)

Target exceeded. Actual value reflects data from 2012-13 school year.

Indicator 3 : Percentage of community members that observe participatory decision-making between parents, teachers and principal about school affairs


90.20 93.00 91.00 93.50


Targets exceeded. Actual value reflects data from 2012-13 school year.

Indicator 4 : Percentage of community members that observe teachers encouraging and supporting student performance


72.10 77.00 78.00 75.20


Original target 98 percent achieved and revised target 96 percent achieved. Actual value reflects data from 2012-13 school year.

Indicator 5 : Percentage of community members that observe teachers encouraging the active participation of the students


66.90 72.00 74.00 91.50


Targets exceeded. Actual value reflects data from 2012-13 school year.

Indicator 6 : Total hours of technical assistance provided by the CNEPEC to the CGEPECs per yearValue (quantitative or Qualitative)

732 970 3,748

Date achieved 06/30/2010 06/30/2014 06/30/2014

ix



approval documents)


Values




Target exceeded. Indicator added in the April 2013 restructuring to measure technical assistance given to States in the form of capacity building to follow PEC Operating Rules and prioritize schools in indigenous and marginalized areas.

Indicator 7: Percentage of financed PEC schools that report financial information on time to PEC's National Management Information System


24 states (original) 88.40 (revised)

31 states 94.40 31 (original) 74.90 (revised)

Date achieved 06/30/2010 06/30/2013 06/30/2014 06/30/2014


Original target achieved. Revised target 85 percent met. Actual value reflects preliminary data for the 2012-13 school year. Revised in the April 2013 restructuring. The original wording of this indicator was “Percent of states that report on time the information of the Program through a certified MIS.”

Indicator 8 : Dissemination of the simplified school planning instrument Value (quantitative or Qualitative)

Unsimplified school planning instrument

Dissemination Dissemination

Date achieved 06/30/2010 06/30/2014 06/30/2014


Target met. Indicator added in the April 2013 restructuring. The dissemination of the simplified school planning instrument was implemented through different strategies, including the distribution of the new format to schools and the capacity building of relevant authorities at the state level on the use of the tool to enable them to train their schools.

Indicator 9: Dissemination of PEC's objectives, activities and results


Publication of EDUCARE magazine; design and launch of PEC volumes, bulletins and informative notes

Continued implementation of communications strategy

Continued implementation of communications strategy

Date achieved 06/30/2010 06/30/2014 06/30/2014


Target met. Indicator added in the April 2013 restructuring. The Ministry of Public Education (Secretaría de Educación Pública, SEP) implemented a comprehensive communications campaign that included, among others, the constant updating and upgrading of the program’s webpage and the production and transmission of several TV programs related to PEC.

x

G. Ratings of Project Performance in ISRs

No. Date ISR Archived

DO IP Actual Disbursements

(USD millions) 1 07/30/2010 Satisfactory Satisfactory 0.00 2 02/23/2011 Satisfactory Satisfactory 44.00 3 08/15/2011 Satisfactory Satisfactory 61.27 4 03/25/2012 Satisfactory Satisfactory 114.31 5 10/15/2012 Satisfactory Satisfactory 114.31 6 06/30/2013 Satisfactory Satisfactory 187.96 7 01/08/2014 Satisfactory Satisfactory 218.73 8 07/10/2014 Moderately Satisfactory Moderately Satisfactory 219.45

H. Restructuring (if any)

Restructuring Date(s)

Board Approved PDO

Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in USD millions

Reason for Restructuring & Key Changes Made

DO IP

04/25/2013 S S 187.96

Dropped Component 3 becausemost activities were not critical to the achievement of the PDO and were supported outside the Project. Revised Component 2 to include the impact evaluation previously under Component 3, the provision of technical assistance to states to improve their prioritization of marginalized and indigenous schools, and the carrying out of an M&E survey to collect data from community members on the SBMmodel in PEC schools, among others. The restructuring also adjusted the results framework to ensure full measurement of the achievement of the PDO. A reallocation of part of the loan proceeds to increase the number of beneficiary schools and anextension of the closing date by one year were also part of this restructuring.

09/20/2013 S S 187.96

The only change was a reallocation of all funds (US$500,000.00) from Category 3 (Consultant Services) to Category 1 (Goods, Works and Consultant's Services supported

xi

Restructuring Date(s)

Board Approved PDO

Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in USD millions

Reason for Restructuring & Key Changes Made

DO IP

under direct support to schools). This change was needed becauseSEP decided to finance the impact evaluation of PEC with its own resources. The Bank continued to oversee the quality of this impact evaluation. This reallocation also allowed assigning more direct support to schools.

06/26/2014 S S 219.45

The only change was a reallocation of US$93,331.55 from Category 2 (Incremental Project Costs) to Category 1 (Goods, Works and Consultants' Services supported under the direct support to schools). The change was necessary, given SEP's decision to finance the remaining operating costs with its own resources. The reallocation also allowed assigning more direct support to schools.

I. Disbursement Profile

1

1. Project Context, Development Objectives and Design A. Country and sector issues 1.1 Context at Appraisal 1. Improving basic educational attainment and learning outcomes for all was, and still is, central to improving Mexico’s competitiveness in the global economy and to decrease inequality. At the time of the Project’s Appraisal (May 2010), Mexico was the world’s fourteenth largest economy, yet it was ranked 60th in terms of global competitiveness, one of the lowest for an OECD country and below several other countries in the region (Global Competitiveness Report 2009-10). Mexico’s labor productivity was low compared to other OECD countries and in 2011 was even below its level in 1995. Therefore, Mexican labor was not able to compete with low cost producers in Asia and Central America. Mexico thus needed to focus more on moving from a low-cost labor-based economy to a knowledge-based economy. In addition, Mexico ranked 56th in the Human Development Index (HDI), the second lowest score among OECD countries, and had a significant percentage of its population living in moderate and extreme poverty in 2010: 51 and 19 percent, respectively. 2. The quality of education faced important challenges. Mexico’s performance in the Program for International Student Assessment (PISA) was the lowest among OECD countries. In PISA 2009 (with an average of 500 points and a standard deviation of 100), Mexico achieved a greater total of 425 points in reading, 419 points in math, and 416 points in science. By comparison, the United States, one of Mexico’s main commercial competitors, performed 69 points higher in math. Learning outcomes, as measured by the national standardized test ENLACE (Evaluación Nacional del Logro Académico en Centros Escolares), were also low, especially among the poor. Although test scores have generally increased over time for all students (including indigenous and disadvantaged students), consistent disparities remain. Students from public schools performed lower than those from private schools by 0.6 standard deviations (60 points) in ENLACE 2010 (with an average of 500 points and a standard deviation of 100), with this gap increasing for indigenous and disadvantaged students. In addition, while educational enrollments were increasing, those that remained outside of the system were mainly poor or indigenous. For instance, the percentage of net enrollment in 9th grade in 2010 was 87 percent; however, the net enrollment for the same grade was only 75.30 percent for students living in highly disadvantaged localities and 85 percent for indigenous students. 3. To address these challenges, the Alliance for Education Quality (La Alianza para la Calidad de la Educación) was created through a formal agreement between the Federal Government and the largest National Teacher’s Union (Sindicato Nacional de Trabajadores de Educación, SNTE). The Alliance sought to improve student learning in basic education via the mobilization of civil society and state governments, improved participation of all stakeholders, and the transformation of management and service delivery in education. In this context, the Quality Schools Program (Programa Escuelas de Calidad, PEC), created in 2001 and supported by the Bank since 2006, that focused mainly on the peri-urban poor became even more relevant to advance the country’s education agenda by increasing school autonomy and social participation.

2

1.2 Original Project Development Objectives (PDO) and Key Indicators 4. The Loan Agreement (LA) states that the objective of the Project is to “strengthen PEC by increasing overall coverage and social participation in Eligible Schools while putting greater emphasis on public marginalized schools and on the indigenous population, as well as a reorientation of the direct support to schools1 to improve public schools’ internal efficiency and learning outcomes.” The objective on the LA differs from the Project Appraisal Document (PAD). The main inconsistency is that the PAD makes it clear that the long term development objective of the Adaptable Program Loan (APL) is “to improve the quality of education as measured by coverage, social participation, and educational outcomes” and that educational outcomes were not Project specific objectives as implied by the PDO in the Loan Agreement. 5. The Key Outcome Indicators of the Project as stated on page 33 of the PAD were:

Number of schools participating in PEC as percentage of the total number of basic education schools (from 18 in 2009 to 22 percent in 2013);

Total number of basic education schools participating in PEC (from 39,180 in 2009 to 50,000 in 2013);

Commitment to the goals and activities of the Strategic School Transformation Plan (Plan Estratégico de Transformación Escolar, PETE) (as expressed by parental participation in PETE design or adjustment (as percentage of total parents) (from 74.2 percent in 2009 to 80 percent in 2013);

Percentage of PEC schools in highly marginalized and marginalized areas (from 40 percent in 2009 to 50 percent in 2013); and

Percentage of PEC schools with indigenous students (from 7 percent in 2009 to 15 percent in 2013).

1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 6. The PDO was not revised. In the April 2013 restructuring, three out of five key indicators were revised, one dropped (due to repetition), and one added to ensure the full measurement of the PDO achievement. The following are the revised/added key indicators:

Percent of basic and special education schools participating in PEC (from 20.3 in 2009 to 23 percent in 2013). Revised to clarify its meaning, includes an explicit reference to special education schools (also covered by PEC).

Percentage of school community members who participate in the design of the PETE (from 74.2 in 2009 to 75 percent in 2013). Revised because baseline data reflected the participation of community members, not parents.

Percentage of Indigenous modality schools participating in PEC (from 15.4 in 2009 to 19.35 percent in 2013). Revised to better reflect the Project’s impact on PEC’s reach to the

1 School Grants, originally used in the PDO, has been replaced with the term “direct support to schools” at the request of the Borrower

3

indigenous population. Under the original indicator, a PEC school was counted if it had only one indigenous student.

Percentage of direct support to schools spent on education quality inputs (from 59.7 in 2009 to 66.86 percent in 2013). Added to measure the reorientation of the direct support to schools.

1.4 Main Beneficiaries 7. The Project benefited public basic education (pre-primary, primary and lower secondary) schools participating in PEC in all 31 states and the Federal District while putting greater emphasis on marginalized areas and on the indigenous population. Principals, parents, teachers, and students in approximately 49,000 PEC schools and 7,669,759 students (annual average) benefited through the provision of direct support to schools and technical assistance to implement school improvement plans. Indirect Project beneficiaries were education authorities at the state level that received technical assistance to oversee the Project and improve their prioritization of indigenous schools and schools in marginalized areas. 1.5 Original Components (as approved) 8. Component 1: Direct Support to Schools. Provision of support to PEC, through the provision of direct support to Eligible Schools to implement school improvement plans. 9. Component 2: Monitoring and Oversight. Provision of support for: (i) the continuous operation, maintenance and upgrading of the PEC national management information system (MIS); and (ii) the accreditation and strengthening of state programs to manage information systems, to ensure that the information obtained through the referred state systems is compatible with the national MIS and provides all the technical and fiduciary information required for the satisfactory implementation of the Project in the relevant State. Provision of technical assistance to monitor and oversee the Project through: (i) the carrying out of supervision visits; (ii) the organization and carrying out of capacity building activities at the state level; and (iii) the coordination of regional and national meetings attended by PEC’s federal and state authorities. Design and implementation of an information campaign to disseminate Project objectives, activities and results. 10. Component 3: Carrying out of evaluations and assessments to create a strong analytical basis for development and improvement of PEC, including: (i) an external evaluation of PEC’s performance and preparation of an annual report; (ii) a qualitative evaluation to determine the non-quantifiable benefits accruing from PEC; (iii) Ministry of Public Education (Secretaría de Educación Pública, SEP) student learning achievement assessments using national education standards for each school year; and (iv) one impact evaluation of the Project. Carrying out of Project-related policy studies including: (i) studies focusing on improving the coordination of PEC with other SEP programs, including a study to improve the coordination of PEC with the program administered by the National Council for Educational Development (Consejo Nacional de Fomento Educativo¸ CONAFE), and a study to improve the coordination of PEC with the Oportunidades Program; and (ii) studies focusing on improving the equity and efficiency of PEC.

4

1.6 Revised Components 11. Component 2 was amended in the April 2013 restructuring to: (i) specify the provision of technical assistance to the states to improve their prioritization of indigenous schools and schools in marginalized areas and monitor their adherence to the PEC Operating Rules; (ii) add the dissemination of the simplified school planning instrument; (iii) add the carrying out of a monitoring and evaluation (M&E) survey to collect data from community members on the strategic school based management (SBM) model in PEC schools; and (iv) carry out an impact evaluation and a qualitative evaluation of the Project.

12. Component 3 was deleted. 1.7 Other significant changes 13. The April 2013 restructuring also extended the closing date of the Project by one year, from June 30, 2013 to June 30, 2014, to allow time for the completion of all Project activities. In addition, all Project restructurings (April 2013, September 2013, and June 2014) reallocated part of the Loan Proceeds from Categories 2 (Incremental Project Costs) and/or 3 (Consultant Services) to Category 1 (Goods, Works and Consultants’ Services under direct support to schools) in order to increase the number of beneficiary schools and to reflect that the Government was using its own funds to support the incremental Project costs and consultant services originally planned under the Project. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry Strengths

Project preparation took place from May 2009 to June 2010. The Project’s overall focus on PEC, which aims to increase school autonomy and social participation, as well as the Project’s PDO, which placed greater emphasis on public marginalized schools and on the indigenous population, were well aligned with the Country Partnership Strategy (CPS) FY2008-14 (Report No. 42846-MX) objectives of providing comprehensive assistance to promote social inclusion and reduce poverty through human development programs.

Based on the lessons learned from the first phase of the series (Mexico Education Quality Project, P088728, Ln. 7347-MX), the new design simplified the school improvement plans to increase schools’ and parents’ participation.

The Project design took into account most of the potential risks in achieving the PDO. The main risks considered were: inertia to school management and social participation, lack of information on what happens at the school level, complexity in the supervision of funds as they flow into schools, and the risk that schools with indigenous children or in low income households would not benefit from PEC. The necessary actions to mitigate these risks were also considered. The Project design incorporated: (i) specific dissemination and capacity building activities to reach the most disadvantaged schools; (ii) technical assistance to states so that they could help schools gradually change their culture and increase parental participation; (iii) prioritization of indigenous and marginalized schools in PEC Operating

5

Rules; and (iv) smaller/rural schools benefited from a less complex school planning instrument, among others.

The Project design focused on increasing the active role of the school community by, among others, providing direct support to schools only when the school improvement plans were developed by a school community that received capacity building in this area.

Weaknesses The PDO was poorly articulated. It was hard to understand what the Project strove to

achieve. Increasing PEC’s coverage meant expanding SBM to more schools but was not explicitly mentioned. Likewise, social participation meant community participation in different aspects of the school life (participation in the school council, in the decision process to use the direct support to schools, and in developing and implementing projects).

The PDO in the LA was inconsistent with the PAD, including long term development objectives such as improvements in internal efficiency and learning outcomes indicating insufficient quality control. The PAD documentation indicated these objectives were for the end of the APL series, not for the Project.

Some lessons regarding the challenges of undertaking rigorous impact evaluations of PEC due to the self-selection bias of schools were overseen in the Project’s design. The Project could have embedded an Impact Evaluation Strategy, including technical support to develop the design and the rest of the impact evaluation stages.

The challenges associated with the procurement of technical assistance under the first phase of the APL, mainly related to the client’s preference to follow local procedures were not taken into account. In this regard, better specifications of the procurement processes and guidelines, as well as allowing more time to carry out these processes would have been helpful.

The PAD’s economic analysis did not explore the Internal Economic Rate of Return (IERR) of the Project or any other cost-effectiveness methodology.

2.2 Implementation 14. Component 1 was disbursed in a timely manner and was, by far, the largest in terms of Bank support, with 99.3 percent of the loan allocated to it by the end of the Project. Overall, the Project supported approximately 75 percent of the total direct support to schools, since schools received additional funds from states with a federal to state contribution of 3 to 1. Even with increased federal budget in the first years of the Project, SEP was able to transfer resources to the National Trust Fund (Fideicomiso Nacional de Escuelas de Calidad, FNEC) efficiently and from the FNEC to the State Trust Funds (Fideicomisos Estatales de Escuelas de Calidad, FEEC) as soon as the states deposited their counterpart funding. All 31 states in Mexico and the Federal District participated in PEC, with the exception of Sonora and Veracruz in the 2012-13 school year and Chiapas and Veracruz in 2013-14, due to their financial circumstances. 15. The main challenge faced under this Component was that in some cases, the direct support to schools arrived late in the school year, preventing them from implementing their Annual Work Plan (Programa Anual de Trabajo, PAT) as planned. The delays were due to some states’ inability to make their deposits to the FEEC on time. To try to ensure that schools received the direct support at the beginning of the school year, SEP, the implementing agency: (i) shortened the deadlines for

6

states’ deposits in the Operating Rules (as early as June 30 when the school year starts in mid-August); (ii) built a system of incentives where states that made their deposits on time were able to get additional federal resources; and (iii) sent official letters (“oficios”) and continuous reminders to prompt states to deposit in a timely manner. In 2012, 15 states did not disburse direct support to schools’ funds on time (by September); in 2013, only 10 states exceeded the time on the disbursement of these funds. 16. Component 2 (Monitoring and Oversight). Despite the variety of ways in which the states reported their information to SEP, the Federal authority had enough capacity to aggregate the data to generate a dataset with sufficient information to monitor the performance of the Project and PEC. However, the continuous operation, maintenance and upgrading of the Information System for the Quality Schools Program (Sistema de Información del Programa Escuelas de Calidad, SIPEC) and compatible state systems faced important challenges, mainly due to the disparity among the states’ institutional capacity, technical glitches, and the server’s low capacity. San Luis Potosí, Michoacán, Estado de México, and Guanajuato were able to smoothly run their information systems from the beginning of the Project, ensuring that schools reported information on-line to feed the SIPEC. The rest of the states continued to use outdated methods of data collection and analysis, such as pen and paper instruments or basic excel sheets. In an effort to help states with M&E shortcomings, SEP customized and targeted its support to weaker states, resulting in overall strengthening of these states’ M&E capacity. However, the large degree of heterogeneity in terms of the reliability and frequency at which states report data persists. In 2014, the Bank provided technical assistance to review the PEC national databases and recommended harmonizing all historical information to develop one single data set of PEC schools over time.2 17. After the April 2013 restructuring, SEP began targeting technical assistance to the states to particularly improve the prioritization of indigenous schools and marginalized areas. For instance, the last PEC Annual Meetings focused, among others, on targeting criteria and states’ adherence to the PEC Operating Rules. In addition, SEP started formally reviewing the states’ calls for proposals for PEC and the explicit inclusion of prioritization criteria for schools located in indigenous and marginalized communities. 18. The impact evaluation was delayed due to the difficulty in reconciling certain administrative requirements with the requirements of the Bank’s procurement procedures; nevertheless, this activity was not crucial to the achievement of the PDO. However, SEP was successful in the implementation of an information campaign to disseminate the Project’s objectives, activities and results. The dissemination of the simplified school planning instrument was carried out through the Project and reinforced in the last year and was critical to boosting participation of more disadvantaged schools in PEC and to increasing social participation among school stakeholders. 19. Component 3 (Policy Development and Evaluation). The implementation of the activities under this Component proved challenging because the Government was undertaking most of the

2 The Bank also recommended distinguishing PEC schools receiving only technical assistance from those receiving technical assistance and direct support. Aside from these efforts, the information on the roster of schools participating in PEC continues to suffer from the lack of information from a few schools.

7

activities with their own budgetary funds outside of the Project. For instance, the external evaluation of PEC’s performance and preparation of an annual report was developed by the National Council for the Evaluation of Social Development Policy (Consejo Nacional de la Evaluación de la Política de Desarrollo Social, CONEVAL) through annual reports in different areas including design, objectives, performance and consistency and results. In the April 2013 restructuring, this Component was eliminated because none of the activities (apart from the impact evaluation of the program, which was moved to Component 2) were critical to the achievement of the PDO. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 20. Design: The Project’s Results Framework (PRF) was not fully developed. The key indicators did not measure all aspects of the PDO achievement. In other cases, indicators were repetitive or lacked clarity. The PAD also had a lack of consistency regarding key indicators in different sections of the document (page 7 and page 33). It integrated an indicator to measure learning outcomes, one of the long term objectives after Project closing, but not an indicator on internal efficiency, the other long term goal. The PRF included outcome indicators that were more aligned with the Project’s PDO and activities than others and some that were broad and difficult to measure. Not all aspects of the PDO were covered by the original key indicators; in particular, the reorientation of direct support to schools. The original data collection methods explanations lacked clarity and were in some instances absent from the PAD or other Project documents, and several baselines were estimated with preliminary data. 21. Implementation: Data was collected and reported periodically throughout the Project’s implementation. However, M&E remained a weakness throughout most of the Project until the April 2013 restructuring, where indicators were revised and the necessary arrangements and clarifications were made to start collecting data to gather information for some indicators. With the April 2013 restructuring, many of the shortcomings at design were addressed. In particular, the restructuring added an indicator to fully measure the PDO and clarified methodologies and resources for collecting data and tracking Project indicators. For example, an M&E survey to collect data from community members on the strategic SBM model in PEC schools was explicitly included in Component 2 to ensure that it was implemented. The M&E survey was carried out once and could be compared to a previous survey conducted by SEP; however, the second round of the survey has not been conducted because SEP was unable to contract the firm to undertake this activity due to the long administrative process it entailed, including the reconciliation of the Bank’s procurement guidelines with internal administrative procedures. Additionally, the lack of timely information from a few states to the Federal Government prevented SEP from updating the PRF on some occasions. In any case, the biannual Project Implementation Reports and implementation support missions were always useful and provided enough information to understand the status of the Project. 22. Utilization: The implementing agency and the Bank constantly assessed and used Project data to inform decision-making and resource allocation. For instance, information was used to constantly revise the PEC Operating Rules, which experienced various changes such as: (i) changes in the timing in which states needed to deposit their counterpart funds in the FNEC; (ii) changes to further stress marginalized and indigenous schools prioritization; and (iii) changes in

8

the percentages of direct supports to schools to be spent on education quality inputs, to name a few. In addition, the M&E information was also useful to provide just in time support and information to states and indigenous and marginalized schools in order to ensure that they applied for entry into the program and benefited from it. 2.4 Safeguard and Fiduciary Compliance 23. Environmental and social safeguards: The Project had an environmental category C. In some cases, PEC schools conducted minor rehabilitation works that did not entail a negative environmental impact. The Project also triggered OP/BP 4.10 (Indigenous Peoples). The Indigenous Peoples Plan (IPP) Plan Indígena Programa Escuelas de Calidad 2010 (published on the Bank’s and SEP’s webpages) was based on an evaluation of PEC execution in indigenous schools and consultations with stakeholders. The IPP was implemented adequately, ensuring that indigenous schools participated and benefited from the Project and reaching 7 percent of indigenous schools as a proportion of the total number of schools in the program in the school year 2013-2014, when the target set in the IPP was of only 4 percent. 24. Financial Management (FM): The Project’s FM arrangements were adequate to provide reasonable assurance that Project funds were used for the purposes intended. Minor shortcomings in FM existed, but did not prevent the timely and reliable provision of information required to manage and monitor Project implementation. Project audit reports were delivered on time throughout the life of the Project and were considered acceptable to the Bank. 25. Procurement: Ex-post reviews of PEC schools’ procurement were carried out in Guanajuato, Michoacán, Puebla, State of Mexico, Tlaxcala and Aguascalientes. Minor procurement weaknesses were identified, mainly in the integration of records, requests for quotations, and schools investments. These shortcomings did not have material impacts on compliance with procurement policy requirements. 2.5 Post-completion Operation/Next Phase 26. PEC is a top priority for SEP, as it is seen as the program for spearheading the implementation of an ambitious Reform launched in 2013 to improve the quality of education through, among others, strengthening SBM. The SBM policy is currently implemented through different national programs such as PEC, PETC and Support for School Management (Apoyos a la Gestión Escolar, AGE – managed by the National Council for Education Development, Consejo Nacional de Fomento Educativo, CONAFE). In 2007-08, SEP introduced PETC in basic education to increase the number of school-day hours and promote SBM following the PEC scheme. The PETC’s SBM component is exactly the same as PEC’s. These programs share the goal of contributing to students’ learning within a framework of more school autonomy and social participation. In addition, based on the lessons learned from this Project, the use of evaluations to improve pedagogical strategies and managerial capacities at the school level are being introduced. The Bank is supporting the Government with a new School Based Management Project (P147185, Ln. 8446-MX, approved by the Board on October 24, 2014) which will provide support to PEC and PETC. The development objective of this operation is to improve schools’ managerial capacity and parental participation to reduce dropout, repetition and failure rates among schools

9

participating in PEC and PETC. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation Objectives Rating: Pre April 2013 Restructuring – High / Post April 2013 Restructuring High 27. The relevance of the PDO is deemed to be High pre and post the April 2013 restructuring. The Project objectives are relevant to the World Bank Group’s Country Partnership Strategy (CPS) for FY14-FY19 (Report No. 83496) and fit under the CPS theme II “Increasing Social Prosperity,” under the fifth area of engagement “Promoting Labor Markets for Inclusive Growth.” One of the main priorities for Mexico’s growth and development, emphasized both in the reform process and in the National Development Plan (NDP) for 2013-18, is improving the quality of education, which is consistent with the Project’s long term objective of improving learning outcomes. The NDP also focuses on reducing the achievement gaps between rich and poor to increase productivity and long-term growth while reducing social inequalities, which is consistent with the PDO’s focus on indigenous and marginalized populations. In addition, the 2013 constitutional mandate and the Education Sector Program 2013-18 (Plan Sectorial de Educación, PSE) put schools at the center of the education system by fostering national strategies to, among others: introduce mechanisms to allocate direct support to schools and increase schools’ management capacity; strengthen the relationship between the school, parents, and the broader community to collaborate with the school; and increase transparency and accountability, all of which are consistent with the Project’s objectives of expanding the PEC SBM model and increasing social participation to eventually improve public schools’ internal efficiency and learning outcomes. Design Rating: Pre April 2013 Restructuring – Substantial / Post April 2013 Restructuring – Substantial 28. The Project design is considered Substantial before and after the April 2013 restructuring. The Project mainly focused on providing direct support to schools to implement their school improvement plans and providing technical assistance to states, crucial activities to expand the PEC SBM model to more schools. Project activities such as the dissemination of the simplified school planning instrument were highly relevant to increase participation of schools located in indigenous communities and marginalized areas and increase social participation. The provision of technical assistance and guidance to develop school improvement plans in a collaborative way (by the school director, teachers, parents, the community and the students) was crucial to build the schools capacity to spend the direct support in education quality inputs while increasing community participation in different aspects of the school life. Technical assistance for an M&E survey to collect data on social participation and an impact evaluation was included. While not critical to the achievement of the PDO, this technical assistance was instrumental in measuring PDO achievement and in providing stronger evidence of impact that could inform future interventions. The design had minor shortcomings, for example, it could have included technical assistance to design an evaluation strategy and to explicitly train principals at the school level.

10

29. There is emerging literature that SBM can lead to improved learning outcomes and internal efficiency.3 Three impact evaluations show significant reductions in dropout and repetition rates and increments in pass rates in PEC schools (Skoufias and Shapiro 2006, Murnane, Willet and Cardenas 2006, C230 Consultores, 2012). 3. 2 Achievement of Project Development Objectives. 30. Table 1 summarizes the achievement of results by objective, relying on evidence from the PDO Indicators and on supplementary indicators introduced at the April 2013 Restructuring and at the Implementation Completion and Results Report (ICR) stage.

Table 1: Summary of Project Outcomes

Indicator Baseline

2009 End Target June

2014 End of Project

June 2014 Strengthen PEC by increasing overall coverage Percentage of basic and special education schools participating in PEC a

20.30 22.00 pre-restructuring and 23.00 post-

restructuring

23.85 (2012-13)

Number of Schools in PEC b 40,790 50,000 50,624 (2013-14)

Strengthen PEC by increasing overall social participation Percentage of school community members who participate in the design of the PETE a/d

74.20 (2008-9)

80.00 pre-restructuring and 75 post-restructuring

72.70 (2012-13)

Percentage of community members that observe participatory decision-making between parents, teachers and principal about school affairs c/f

90.20 93.00 pre restructuring and 91.00 post restructuring

93.50 (2012-13)

Greater emphasis on marginalized schools Percentage of PEC schools in marginalized and highly marginalized areas b

38.40 50.00 pre restructuring and 55.00 post restructuring

51.35 (2013-14)

Percentage of PEC schools in marginalized, highly marginalized and medium level of marginalization areas f/g

65.30 No target 78 (2012-13)

Greater emphasis on the indigenous population Percentage of indigenous modality schools participating in PECa

15.40 19.30 18.00 (2012-13)

3 Evidence of links between SBM and improved learning outcomes is mixed. Evidence from Kenya shows that parent-teacher associations lead to better learning outcomes (Duflo et al 2014) when parents are empowered through specific capacity building. In The Gambia, SBM was shown to reduce student and teacher absenteeism; SBM only had a positive effect on learning outcomes in communities with high literacy, otherwise the effect could be potentially negative in communities with low literacy (Blimpo, Evans 2011). In Indonesia, SBM was found to have links to improved learning outcomes only when school based management is complemented by linkages between the school committee (Pradhan et al. 2014). Evidence also shows that SBM has improved social participation, governance, transparency, and accountability, leading to lower dropout and repetition rates (Gertler et al 2012). A recent observational study suggests that school autonomy may lead to improved learning outcomes when strong central institutions are in place, but otherwise possibly detrimental (Hanushek et al. 2012). Finally, another study argues that SBM programs need to be in place for at least 5 years to achieve significant results and at least 8 years to “transform” schools (Borman et al. 2003).

11

Indicator Baseline

2009 End Target June

2014 End of Project

June 2014 Percentage of PEC schools with indigenous students b 7 15 13.5

(2011-12) Simplification of the school planning instrument f NO YES YES Reorientation of the direct support to schools Percentage of direct support to schools spent on education quality inputs a

59.70 66.86 74.54 (2012-13)

PEC public schools learning outcomes Average ENLACE test score in Math in PEC primary schools e

503 No target 560 (2012-13)

Average ENLACE test score in Spanish in PEC primary schools e

498 No target 534 (2012-13)

Average ENLACE test score in Math in PEC secondary schools e

494 No target 537 (2012-13)

Average ENLACE test score in Spanish in PEC secondary schools e

485 No target 481 (2012-13)

PEC public schools internal efficiency Average pass rate in PEC primary schools e 96% No target 99%

(2012-13) Average pass rate in PEC secondary schools e 89% No target 92%

(2012-13) Average Dropout rate for PEC primary schools f 0.041

No target 0.038

(2012-13) Average Dropout rate for PEC secondary schools f 0.054

No target 0.052

(2012-13) a revised outcome indicator; b original outcome indicator; c the percentage of schools where community members confirmed that they observed participatory decision-making between parents, teachers and principal about school affairs; d the percentage of schools where community members confirmed that they participated in the design of the PETE; e indicator added at restructuring ; f indicator added at ICR stage; g Page 6 of the PAD states as a trigger from Phase II to Phase III that the number of PEC schools located in areas of medium, high, and very high disadvantaged (marginalization levels) will increase by 30 percent. Strengthen PEC by increasing its overall coverage Rating: Pre April 2013 Restructuring – High / Post April 2013 Restructuring – High 31. The number of schools operating under the PEC SBM model increased from 20.30 percent of the total of basic and special education schools to 23.85 percent in the 2013-14 school year, exceeding both the original target of 22 percent and the revised target of 23 percent. Almost 10,000 additional schools participated in the program, from 40,790 in 2008-9 to 50,624 in 2013-14, also exceeding the 50,000 PEC schools target set at Appraisal. 32. The Project contributed to increasing the coverage of eligible schools by offering direct support to all new schools entering the program. Having a school improvement plan developed by the school community was mandatory to access the direct school support that was then used to support activities related the school improvement plan at each school (which vary considerably depending on specific schools’ needs). Aggregate numbers for 2012-13 show that most of the direct support to schools (39.4 percent) was spent on schools’ technical equipment and furniture and on teaching materials (18.5 percent). Schools also used direct support for rehabilitation (15.3), school materials (8.8 percent), minor constructions (5 percent), capacity building activities (1.5 percent), and others (11.5 percent). Direct support to schools average around US$4,000 per year

12

and, while small compared to the total costs of running schools, are not insignificant when compared to the budget managed directly by the schools. 33. The Project also helped ensure the participation of a higher number of schools through the simplification of the schools’ application process and the school planning instrument (PETE). In addition, the design and implementation of the information campaign to disseminate PEC’s objectives, activities and results contributed by increasing awareness. Finally, the Project supported technical assistance to the states, which strengthened their capacity to reach out to a greater number of schools and encouraged them to apply to and stay in the Program. Strengthen PEC by increasing overall social participation Rating: Pre April 2013 Restructuring – Modest / Post April 2013 Restructuring – Modest 34. The percentage of surveyed school community members who participated in the design of the school planning instrument (PETE) remained at around 73 percent throughout the Project and the pre and post restructuring targets of 80 and 75 percent, respectively, were not achieved. On the other hand, the percentage of surveyed school community members that observe participatory decision-making between parents, teachers and principals increased from an already high 90.2 in 2008-9 to 93.5 percent in 2012-13, exceeding the original target of 93 and the restructured target of 91 percent. 35. The Project, through PEC, helped improve parental participation by providing supplemental direct support to schools that mobilized additional local resources to implement their school plans. The supplemental supports matched the amount of local resources raised by the school. This worked as an incentive for parents and the community to work together to raise funds for the school and become more involved in the development of school plans and their implementation. In addition, the Project engaged parents and community members to actively participate in the school self-evaluation, planning, and implementation processes by enforcing the collaborative development of school improvement plans and building the capacity of school communities on collaborative SBM. The dissemination of the simplified school planning instrument (consisting of a manual with a series of recommendations intended to make the school planning process clearer, easier, and more efficient for schools while easily explaining the role of different stakeholders in collaborative SBM) was also critical to encouraging greater participation of parents and the school community in school affairs since it allowed poorly educated parents to better understand the school planning processes and thus, be better able to contribute to them. Greater emphasis on marginalized schools Rating: Pre April 2013 Restructuring – Substantial / Post April 2013 Restructuring – Substantial 36. The percentage of PEC schools in marginalized and highly marginalized areas increased from 38.4 percent in 2009-10 to 51.35 percent in the 201-14 school year; exceeding the original target and almost meeting the revised target, which was 93 percent achieved. The percentage of PEC schools located in areas with a medium level of marginalization (that still face important socioeconomic challenges) and in highly and very highly marginalized areas reached 78 percent in 2012-13, compared to only 65.3 percent in the school year 2009-10. One of the major factors affecting the full achievement of the target was that Chiapas and Veracruz, two of the states with

13

a higher percentage of schools located in marginalized and indigenous communities, were not able to participate in the last year of the Project due mainly to their financial circumstances, preventing participation of an important percentage of marginalized schools. 37. The Project contributed to increasing the participation of marginalized schools by constantly revising the PEC Operating Rules to stress prioritization of schools located in indigenous and marginalized communities and by providing technical assistance to the states to improve their prioritization of these schools. Specific activities supported under the Project were: (i) regular supervision visits to all states to ensure that prioritization of schools in indigenous and marginalized areas was made adequately; (ii) the organization of capacity building activities at the state level to build the states’ capacity to manage the program and better select schools; and (iii) regional and national meetings by federal and state authorities. In addition, the Project contributed to increasing the number of PEC schools located in marginalized communities by supporting the dissemination of the simplified school planning instrument, which helped school directors, parents, teachers, and students be more aware of and better understand the simplified instrument, put together their school plans even with low capacity and resources, apply to PEC, and increase their chances of being selected. Greater emphasis on the Indigenous Population Rating: Pre April 2013 Restructuring – Substantial / Post April 2013 Restructuring – Substantial 38. The number of PEC schools that reported to have one or more indigenous students increased from 7 percent in the school year 2009-10 to 13.5 percent in the school year 2011-12, meeting 90 percent of the original target. As for the percentage of indigenous modality schools participating in PEC (indicator introduced at restructuring), 18 percent of indigenous modality schools participated in the last year of the Project, meeting 93 percent of the revised target. However, one year earlier, 20.8 percent of indigenous schools participated in PEC, surpassing this target. 39. The Project contributed in the same way to achieving this objective as to the objective of greater emphasis on marginalized schools. Reorientation of the Direct Support to Schools Rating: Pre April 2013 Restructuring – High / Post April 2013 Restructuring – High 40. The percentage of direct support to schools spent on education quality inputs increased from 59.70 percent in the school year 2009-10 to 74.54 percent in the 2012-13 school year, widely exceeding the 66.86 percent target set during the April 2013 restructuring. Education quality inputs included furniture and technical equipment, teaching materials, school materials, and capacity building. The Project helped reorient direct school support to education quality inputs by providing the necessary technical assistance to the states and schools to make adequate and informed choices when preparing and implementing their school plans with PEC resources. For instance, the Project strengthened states’ capacity building on the consolidation of the Modelo de Gestión Escolar con Enfoque Estratégico (MGEE), a SBM model that helps the school community use local information in a systematic way to use PEC support to prioritize needs that are more correlated with student learning. In addition, the Project contributed to reorienting direct support

14

to schools to education quality inputs by incorporating specific percentages of the direct school support to be allocated by schools to certain categories. For instance, through the Project’s length, newly incorporated schools should spend 30 percent of their direct school support on the improvement of students’ learning processes, which includes teachers’, school directors’, and parents’ capacity building activities and the acquisition of school materials; schools with more years in the program should spend 50 percent in the same category. Improved public schools’ internal efficiency Rating: Pre April 2013 Restructuring – Substantial / Post April 2013 Restructuring – Substantial 41. This ICR defines the schools’ internal efficiency as the schools’ ability to retain students and ensure that they advance through the education system as planned. Average dropout rates for PEC primary schools decreased from 0.041 to 0.038 percent from 2008-9 to 2012-13 and from 0.054 to 0.052 percent for secondary schools. Average pass rates in PEC primary schools increased from 96 percent in 2008-9 to 99 percent in 2012-13 and from 89 to 92 percent in PEC secondary schools. Both rates are better for PEC than for non-PEC schools.

42. To contribute to the improvement of the internal efficiency of PEC schools, the Project helped increase school autonomy in managing the direct support to schools, as well as in spending the direct support on education quality inputs. The Project’s efforts to increase community members’ participation in the school improvement plans led to greater engagement in school life, and there is evidence that these efforts have also led to greater teacher accountability. For example, the percentage of community members that observed that parents were informed about student performance increased from 99 to 99.80 percent. In addition, the percentage of parents that observe: (i) teachers encouraging and supporting students’ performance increased from 90.20 to 93.50; and (ii) teacher encouraging the active participation of students increased from 66.90 to 91.50. Improved public schools’ learning outcomes Rating: Pre April 2013 Restructuring – Modest / Post April 2013 Restructuring – Modest 43. PEC schools improved their ENLACE results with greater gains in Mathematics. ENLACE results in math went from 503 and 494 in the 2008-09 school year for primary and secondary, respectively, to 506 and 537 in 2012-13. In the same time period, Spanish ENLACE results for primary improved from 498 to 534 and remained stable at the secondary level. 44. The Project contributed in the same way to achieving this objective as it did to meeting the improved public schools’ internal efficiency. 3.3 Efficiency Rating: Pre-2013 restructuring – Modest / Post-2013 restructuring – Modest 45. According to the economic analysis in the PAD, the increase in the participation of indigenous and marginalized schools in PEC should help to better target the disadvantaged population in Mexico and thus improve the equity of basic education. This was confirmed during the 2010-13 period, when PEC indigenous schools decreased their failure rate by 4.5 percent,

15

producing an increase in indigenous students’ years of schooling of 0.045 for each PEC cohort during the same period. When these individuals enter the labor market, the increased years of education will result in an increase of 0.225 percent in their monthly salary. During the same period, failure rates decreased by 3.4 for disadvantaged students in PEC, producing an increase in these students’ years of schooling of 0.034. When these individuals enter the labor market, the increased years of education will result in an increment of 0.187 percent in their monthly salary (see Annex 3 for further details). 46. In addition, secondary schools increased math achievement by 0.30 standard deviation in 3 years (during the Project), from an average PISA score of 355 to 385. Using the estimates and methodology elaborated by De Hoyos (2010) to measure the economic benefits of increased learning outcomes, the result could be translated into an increase of 16 percent of the annual income of Mexican families in 2010, which translates to a decrease in poverty of 3.88 million inhabitants in Mexico. While the result is for all the secondary schools in Mexico, PEC schools have contributed to this result (PEC schools outperformed other schools in ENLACE in the same period of time). 47. Although no IERR was calculated at Appraisal, the Project’s economic efficiency is also illustrated by the results from recent economic analysis of the program, which yields a 6.5 percent IERR. The costs of the Project were US$366.66 million for the four years of the Project, or an average of US$91.66 million per year. The average federal annual unit cost per school decreased over the life of the Project because each year more schools were accepted into PEC (from 40,790 in 2009 to 50,624 in 2014, exceeding the original target of 50,000) while the budget increased only slightly from MX$1,477,418,501 in 2009 to MX$1,515,911,529 in 2014. The annual cost per school and student at the end of the Project were lower than at Appraisal (US$1,873 now versus US$2,247 at Appraisal and US$14 now versus US$16 at Appraisal, respectively). Therefore, efficiency increased through the Project’s life. In addition, a comparison of PEC’s student unit cost to other education interventions shows that PEC is relatively inexpensive and renders important gains in terms of improved public schools’ internal efficiency. This cost is lower than the unit cost per student of PETC, where the cost per student in 2012 was US$167. 48. Other aspects of efficiency that have to be taken into account are the one year extension of the Project’s closing date and the states’ delays in depositing their contributions for the direct support to schools. The former illustrates modest inefficiencies in completing the Project in a timely manner, while the latter reflects some inefficiency (around 10 out of 32 states delayed their funding in 2013, an improvement from the 15 that had delayed in 2012) in ensuring that the direct support to schools reached them in time to implement their school plans in an effective and efficient way. 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 49. Relevance ratings for objectives are High for both the pre- and post-2013 restructuring periods. The relevance rating for design pre and post restructuring is Substantial. Efficacy ratings are Substantial for both periods. Finally, the Efficiency rating is Modest for both periods. The combination of these ratings yields an overall Moderately Satisfactory rating for both periods

16

(Table 2). In turn, the overall ratings for the two periods are weighted by the disbursement percentages prior to and following the April 2013 restructuring, yielding an overall weighted rating of 4.0–equivalent to Moderately Satisfactory.

Table 2. Weighted Project Rating

Pre-

restructuring Post-

restructuring Overall

Pre- and Post-restructuring outcome rating Relevance Objectives High High

Substantial Design Substantial Substantial

Efficacy Strengthen PEC by increasing its overall coverage

High High

Substantial

Strengthen PEC by increasing overall social participation Modest Modest

Greater emphasis on schools in marginalized areas

Substantial Substantial

Greater emphasis on the indigenous population

Substantial Substantial

Reorientation of the Direct Support to school to education quality inputs

High High

Improved public schools’ internal efficiency Substantial Substantial Improved public schools’ learning outcomes*

Modest Modest

Efficiency Modest Modest Modest Rating Moderately

Satisfactory Moderately Satisfactory


Overall outcome rating Rating value 4 4 Weight 86% 14% Weighted value

3.44 0.56 4

Final Rating


*This is a long term objective, beyond the scope of the Project. As such, the ICR does report on and rate progress, but the overall rating puts less weight on this objective.

3.5 Overarching Themes, Other Outcomes and Impacts

(a) Poverty Impacts, Gender Aspects, and Social Development 50. The Project contributes to the Bank’s twin goals of eliminating extreme poverty and boosting shared prosperity by contributing to the long-term objectives of improving learning outcomes and retention rates among students in public schools, particularly those from marginalized areas, through strengthened SBM. Recent evidence shows that SBM programs can be an effective strategy to increase parental participation in school decisions; reduce dropout, repetition and failure rates and, under certain conditions, increase learning outcomes. 51. At the macro level, a society with more human capital–measured, for instance, by years of schooling or learning outcomes (long term objectives to which the Project contributed to) in a standardized test—enjoys higher rates of long term economic growth. Conversely, from the micro perspective, more human capital translates into higher personal or household incomes, less poverty

17

and, if the increases in human capital take place among disadvantaged households (which is the case in this Project), less income inequality. (b) Institutional Change/Strengthening N/A (c) Other Unintended Outcomes and Impacts (positive or negative) 52. The transformation of the PEC program into a nationwide school autonomy policy was an unintended outcome of this Project. This policy decision has also contributed to the expansion of PETC. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops N/A 4. Assessment of Risk to Development Outcome Rating: Negligible to Low 53. The 2013 education reform has reinforced the policy to strengthen school autonomy, allowing SEP to expand the number of public schools that receive support following PEC’s design in all public programs in basic education that have a SBM and autonomy component. The overall number of basic education schools with support to increase school autonomy is expected to expand considerably with the implementation of the reform. In consequence, social participation, the reorientation of direct support to schools to education quality inputs, schools’ internal efficiency and learning outcomes are expected to continue to improve in all schools benefiting from the PEC SBM model expanded by the new SBM policy. In addition, the new School Based Management Project (P147185, Ln. 8446-MX) will help foster the autonomy policy through the provision of support to PEC and PETC in order to ensure that the development outcomes of PEC (Phase II) are maintained. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Unsatisfactory 54. The Bank provided assistance to the Government to prepare a Project that was relevant and strategic for the Mexican context. The Project design also took into account poverty and social development aspects, putting greater emphasis on public marginalized schools and on the indigenous population. The major shortcomings were the poor articulation of the PDO and the weaknesses of the PRF. In addition, the Bank did not foresee the technical, administrative and procurement challenges related to undertaking a rigorous impact evaluation of PEC and other technical assistance, which had also emerged in the first phase of the APL. Hence, many of the technical assistance activities designed at Appraisal, including the impact evaluation, could not be finalized as part of the Project, though others were carried out as they were part of the activities

18

that the Government undertakes regularly through their own budget resources. These activities and the impact evaluation were not critical to the achievement of the PDO. In terms of FM, the Project was designed in a way that made the flow of funds quite complex for the Government and the Bank. The economic analysis did not present an IERR or any methodology to measure the cost-effectiveness of the Project. (c) Quality of Supervision Rating: Moderately Satisfactory 55. The Bank conducted regular implementation support missions that included meetings with the Ministry of Finance and Public Credit (Secretaría de Hacienda y Crédito Público, SHCP), National Financial Development Bank (Nacional Financiera, S.N.C., I.B.D, NAFIN) and SEP, as well as field visits to PEC schools in different states and in the Federal District. It also sustained a close and continuous dialogue with the Subsecretariat of Basic Education (Subsecretaría de Educación Básica, SEB), which facilitated the close monitoring of Project implementation. Bank implementation support helped identify the challenges being faced at various stages of Project implementation in a timely manner. For instance, it was aware of the delays in the states’ payments and helped the implementing agency find, implement and follow up on alternatives to solve this challenge. The Bank also provided key technical assistance when needed. For instance, it provided feedback to the impact evaluation design and organized a conference on quality of education and SBM to reflect on the achievements of the PEC model and to learn from international experiences to improve Project implementation and the new SBM reform. The Bank processed a useful Level 2 restructuring that helped ensure that the PDO was well measured and targets on indicators achieved. The safeguards and fiduciary teams continuously supervised and developed follow-up reports regarding the implementation of the IPP and followed up on detailed aspects of disbursements of direct support to schools, audits, etc. while building staff capacity in the Directorate of School Management (Dirección General Adjunta de Gestión, DGAG) in both fiduciary and technical aspects. There were moderate shortcomings. To address design and weaknesses in M&E, the restructuring should have been conducted much earlier and the PDO should have been revised to make it clear that learning outcomes and internal efficiency were objectives beyond the scope of the Project. Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 56. Given that Bank performance was rated Moderately Unsatisfactory at Ensuring Quality at Entry and Moderately Satisfactory at Quality of Supervision, the Bank’s overall performance is considered Moderately Satisfactory, taking into account the outcome rating of Moderately Satisfactory. 5.2 Borrower Performance (a) Government Performance Rating: Satisfactory

19

57. The Government showed ownership of the Project and commitment to achieving its development objectives. Throughout the duration of the Project, SHCP provided enough budget to SEP, and PEC in particular, and ensured that the loan was fully disbursed by its revised closing date, with most of the funds reaching the schools directly through the direct supports to schools. SHCP continuously followed up with the Bank, the implementing agency, and NAFIN to ensure that the Project worked as planned, including overseeing the Project implementation arrangements, the implementing agency capacity, and addressing challenges to the successful achievement of the Project’s objectives. Despite a change in the administration (and political parties) in December 2012, the Government ensured the continuity of PEC and requested Bank support for a new project in order to develop activities to foster and expand SBM and autonomy through PEC and PETC. NAFIN also performed well in its role of fiduciary agent of the Government for the Loan, ensuring that FM, governance, provision of counterpart funding, procurement, reimbursements, and compliance with covenants were followed. (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory 58. SEP, through the General Directorate of School Management and Education Innovation (Dirección General de Desarrollo de la Gestión e Innovación Educativa, DGDGIE), and in particular the DGAG, executed PEC’s budget in a timely fashion and ensured that most of the Project activities were implemented as planned. In close collaboration with the states, SEP ensured the expansion of PEC in general and of PEC schools located in indigenous and marginalized areas in particular. The adequate use of direct support to schools and the enhancement of social participation at the school level were also a top priority for the implementing agency. SEP provided technical assistance to the states so that they could run the program as specified in the Operating Rules, including better prioritization of vulnerable schools. There were moderate shortcomings. The main challenge that SEP faced was to ensure that M&E arrangements were in place at the state level. In addition, although SEP made an effort to execute the studies envisioned in the Project, it was not always able to expedite administrative and procurement procedures to ensure that these activities took place on time and with Project funds. In particular, greater attention to the implementation of the impact evaluation would have been desirable. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 59. The overall rating for Borrower Performance is Moderately Satisfactory. 6. Lessons Learned

Early adjustment of PDO and PRF is desirable. A rigorous impact evaluation of PEC is crucial to understand the effect of the program on

learning outcomes. Undertaking rigorous impact evaluations of programs with a voluntary enrollment nature, such as PEC, has been challenging, mainly due to the schools’ self-selection biases introduced to conventional non-randomized evaluation approaches. In the case of PEC, an impact evaluation should only be supported in future Projects if the

20

necessary technical assistance is provided to develop the evaluation’s design and if states commit to operate PEC in a different manner to comply with randomization criteria.

Local Education Authorities (Autoridades Educativas Locales, AELs) play an important role in planning and implementing federal programs on the ground. It is important to find compelling ways to ensure that all states provide their PEC counterpart financing in a timely manner and comply with MIS that are compatible with the federal MIS. It is also crucial to ensure that states comply with the rules and criteria to guide the program while taking into account the disparities regarding the institutional capacity among states to distribute federal resources and technical assistance. It is important to provide each state with a suite of services to implement school autonomy and accountability initiatives, according to their particular needs and shortcomings.

In future operations, Project procurement and FM should be simplified in regards to the direct support to schools to avoid burdening schools with excessive administrative requirements and delays in the reporting of disbursements and proof of expenditures from the Government to the Bank.

Moving into country systems in countries with a solid fiduciary structure (such as Mexico) could help expedite technical assistance procurement processes and ensure that countries benefit from the technical assistance built into the project designs without necessarily compromising healthy procurement procedures.

The simplification and dissemination of school planning instruments help disadvantaged schools put together initial school improvement plans and therefore allow them to benefit more frequently from programs such as PEC. It is crucial, however, to provide schools with the resources and technical assistance to develop their plans to ensure that they design and implement plans that actually improve student learning.

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 60. At the Borrower’s request, all references to “grants” were replaced by the term “direct support.” (b) Cofinanciers N/A (c) Other partners and stakeholders N/A

21

Annex 1. Project Costs and Financing

(a) Project Cost by Component (in USD Million equivalent)

Components Appraisal Estimate

(USD) Actual/Latest

Estimate (USD) Percentage of

Appraisal

Component 1 211,720,000.00 218,543,100.55 103% Component 2 4,300,000.00 906,899.45 21% Component 3 3,430,000.00 0.00 Total Baseline Cost 219,450,000.00 219,450,000.00 100% Physical Contingencies 0.00 0.00 Price Contingencies 0.00 0.00 Total Project Costs 219,450,000.00 219,450,000.00 100% Front-end fee PPF 0.00 0.00 Front-end fee IBRD 550,000.00 550,000.00 100% Total Financing Required 220,000,000.00 220,000,000.00 100% (b) Financing

Source of Funds Type of

Cofinancing

Appraisal Estimate

(USD millions)

Actual/Latest Estimate

(USD millions)

Percentage of Appraisal

Borrower 146.66 0.00 40% International Bank for Reconstruction and Development

220.00 220.00 60%

22

Annex 2. Outputs by Component Component/Subcomponent Project Outputs

Component 1. Direct Support to Schools Provision of support to PEC, through the provision of direct support to Eligible Schools to implement school improvement plans.

The Project supported around 60.3 percent of the Federal Government’s budget for PEC direct support to schools.

Technical assistance provided by procurement and FM specialists to adequately manage the Project and PEC’s budget from the Federal Government.

Constant modification of PEC Operating Rules: (i) changes in the timing in which states needed to deposit their counterpart funds; (ii) changes to further stress marginalized and indigenous schools prioritization; (iii) established clear percentages of direct supports to schools to be spent on education quality inputs.

Component 2. Monitoring and Oversight Provision of Support for the continuous operation, maintenance and upgrading of the PEC National MIS and the accreditation and strengthening of State programs to manage information systems, to ensure that the information obtained through the referred State systems is compatible with the national MIS and provides all the technical and fiduciary information required for the satisfactory implementation of the Project the relevant State.

The electronic module for capturing the strategic plan for school transformation entitled “Plan Estratégico de Transformación Escolar” was developed.

The system of management of PEC was concluded including four modules: Plan Estratégico de Transformación Escolar, Presupuesto, Seguimiento de Indicadores, Mejores Prácticas Docentes and Geo-básica to geographically locate PEC schools.

Since April 2013, SIPEC has been under a review process to work properly. The DGDGIE and DGAG in particular have been working actively with SEB representatives in

charge of the design of new Management and Information System for Education (Sistema de Información y Gestión Educativa) that will work for all aspects of elementary schools including PEC.

Four states haveelectronic information systems to manage program resources: San Luis Potosí, Michoacán, Guanajuato and Mexico State (1st Semester 2011).

Development of the electronic modules of SIPEC for entering the PETE, schools budget, follow up of key program indicators, best teaching practice, and a geo-referencing module to locate all schools participating in PEC.

Provision of Technical assistance to the States to improve their prioritization of Indigenous Schools and

Supervisory visits Advice and support, as well as guidance for the consolidation of the school management model with

Strategic Approach (MGEE) was provided to the 32 States’ Managing Units of PEC.

23

Marginalized Area Schools and monitor their adherence to the PEC operating rules through (i) the carrying out of supervision visits; (ii) the organization and carrying out of capacity building activities at state level; and (iii) the coordination of regional and national meetings attended by PEC’s federal and state authorities.

Visits were carried out in various states in order to strengthen the actions between CNPEC, CGEPEC and to comply with the goals and PEC Operation Rules.

The hours of supervision visits were tracked and can be accessed in the Implementation Status Reports Results Framework.

Capacity Building Activities Participation in various forums such as that of "Participation of unions and teachers' organizations

in the processes of educational reform and improvement." International Seminar on Integral Education, National Congress of Researchers in Education and

the Family, among others. Workshop for the construction of the Standard: "Coordinación de Proceso de Gestión Educativa

Estratégica en Cetros Escolares de Educación Básica” 2011. Participation in capacity building activities with various institutions such as CONAPASE, Crefal,

World Bank and various programs such as the PNL, the PNIEB, PMLE, among others. Workshop on good practices in school autonomy and SBM (Seminario de Buenas Prácticas en

Autonomía y Gestión Escolar) delivered in June 2013 put together all states, the Federal Government, the Bank and experts from Brazil, Chile, Argentina and Colombia to, among others, learn about States’ experiences with PEC and its Operating Rules and to analyze international good practices to gradually provide more autonomy to schools, increase social participation and eventually improve learning outcomes.

National and regional meetings. Working meetings to define the objectives of the pilot innovation "Strengthening School

Governance and Oversight" project. Regional meetings in the West, Central, North and East regions with the participation of the State

Coordinators of PEC. Meetings to track, "Innovation project for the improvement of management and teaching practice in

the classroom." Capacity Building activity to disseminate the applicable guidelines and provide the necessary

guidance and arrangements for the follow-up of the Project for Institutional Strengthening for Management and School Supervision.

Participation in the Second Meeting of School Directors of PEC Schools: "The Model of Strategic Educational Management, from Theory to Practice" Nuevo León.

24

Regional Certification Workshop of Directors that comprise the National Team of Expert Evaluators.

National Evaluation Meeting for PEC 2011. Monitoring and support to the state strategy for the Consejos Técnicos de Zona y Consejos Técnicos.

Escolares, both for the Intensive (August 2013) and the Ordinary Phase (2013-2014 school year). National capacity building program with Supervisors of Basic Education School. Capacity building activities for Consejos Técnicos Escolares. National Meetings of PEC and Escuela Segura.

Design and implementation of an information campaign to disseminate Project objectives, activities and results and the dissemination of the simplified school planning instrument (PETE).

Dissemination of PEC activities: Information campaigns Redesign of the website of the program and updating of its content. Release of news that refer to the activities of the Program on the PEC’s and DGDGIE websites. Production of two TV show called: “A new perspective on the PEC” and “PEC: at my school we

all express.” Production and transmission of television programs to mark the 10th anniversary of PEC. Transmission of short stories at EDUSAT Network, Channel 22 and two open Television

Programs: I. A new perspective of PEC and II. ‘At my school we all express” PEC. Production of the first TV series for “Shared Visions” entitled “Initiatives to Strengthen

Educational Quality in Basic Education.” Carry out of an M&E survey to collect data from community members on the strategic school based management (SBM) model in PEC schools

2012 Survey to measure indicators related to social participation in PEC schools

Carry out an impact evaluation and a qualitative evaluation of the Project.

Impact evaluation design 2014

25

Annex 3. Economic and Financial Analysis 1. De Hoyos (2010) analyzes the consequences of the lack of education reforms that could have increased education quality during the 1970s in Mexico. The author argues that if an educational reform had happened during the 1970s, it would have increased PISA educational achievement by half of a standard deviation in 10 years from an average PISA score of 406 to 456. The result would have been an increase of 28% in the annual income of Mexican families by 2010, which translates to a decrease in poverty of 6.8 million Mexicans. 2. In addition, secondary schools increased math achievement by 0.30 standard deviation in 3 years (during the Project) from an average PISA score of 355 to 385. Using the estimates and methodology elaborated by De Hoyos (2010) to measure the economic benefits of increased learning outcomes, the result could be translated to an increase of 16 percent of the annual income of Mexican families in 2010, which translates to a decrease in poverty of 3.88 million inhabitants in Mexico. While the result is for all the secondary schools in Mexico, PEC schools have contributed to this result (PEC schools outperformed other schools in ENLACE in the same period of time).

3. The incremental participation of indigenous and marginalized schools in PEC should help to better target the disadvantaged population in Mexico and thus improve the equity of basic education. During the 2010 to 2013 period, PEC indigenous schools decreased their failure rate by 4.5 percent, producing an increase in indigenous students’ years of schooling of 0.045 for each PEC cohort during the same period.

4. With the expansion of PEC and its potential effects on indigenous students, the average worker will have an additional 0.045 years of schooling in 2037, reaching 7.5 years of schooling, according to our calculations. Although this seems to be a small change, it is far from trivial. For instance, according to the ENIGH 2012, the average annual household per capita income in urban, semi-urban and semi-rural areas in Mexico is MX$144,525.7 pesos of 2012. Assuming a return to the additional year of schooling in these localities of 5 percent, every generation of project beneficiaries would experience an average increase of MX$325.18 (=MX$144,525*0.045*0.05) per year vis-à-vis the labor remuneration that the same generation would have obtained in the absence of the Project. The monetary effects for the affected fraction of the workforce of the increase years of schooling brought about by the expansion of the Project is defined by the following expression:

B Y ∗ β ∗ ∆S ∗ P 5. Where B is the monetary benefits of the Project at time “t”, P is the workforce, ∆ is a change operator capturing the effects of the Project. Using the expression and assumptions above, plus the workforce of 59.0 million, in 2022 around 2.48 million will be affected by the expansion of PEC in indigenous schools. Additionally, the monetary benefits of the Project after one year of completion are equal to MX$806 thousands pesos (=MX$144,525*0.045*0.002*2.48 million) at 2012 prices. The sum of the flows of economic benefits up to 2037 amount to MX$12.1 million or US$896 thousand dollars at the official exchange rate of MX$13.5 per dollar (May 22, 2014).

26

6. During the same period, failure rate decreased by 3.4 percent for disadvantaged students in PEC, producing an increase in these students’ years of schooling of 0.034. When these individuals enter the labor market, the increased years of education will result in an increment of 0.187 percent in their monthly salary. 7. Finally, the costs of the Project were US$366.66 million (including counterpart funding) for the four years of the Project, or an average of US$91.66 million per year. The average federal annual unity cost per school decreased from Appraisal to Project closing because each year more schools were accepted into PEC (from 40,790 in 2009 to 50,624 in 2014, exceeding the original target of 50,000) while the budget increased only slightly from 1,477,418,501 in 2010 to 1,515,911,529 in 2014. The cost per school and student at the end of the Project were lower than at Appraisal (US$1,873 now versus US$2,247 at Appraisal and US$14 now versus US$16 at Appraisal, respectively). Therefore, efficiency increased through the Project’s life. In addition, comparing PEC’s student unit cost to other similar education interventions shows that PEC is relatively inexpensive and renders important gains in terms of improved public schools’ internal efficiency. This cost is considerably lower than the unit cost per student of PETC (which includes PEC’s SBM component and an extended school day, among others) where the cost per student in 2012 was US$167.

27

Annex 4. Bank Lending and Implementation Support/Supervision Processes

(a) Task Team members

Names Title Unit Responsibility/

Specialty Lending Erik A. Bloom Senior Economist LCSHE Task Team Leader Mariangeles Sabella Senior Counsel LEGLA Senior Counsel Jose C. Janeiro Senior Finance Officer CTRFC Senior Finance Officer

Maria E. Castro-Munoz Sr Social Scientist LCSSO Social Safeguards Specialist

Maria E. Colchao Senior Program Assistant LCSHE Senior Program Assistant

Dmitri Gourfinkel Financial Management Analyst LCSFM Financial Management Specialist

Gabriel Penaloza Procurement Analyst LCSPT Procurement Specialist Ricardo Rocha Silveira Senior Operations Officer HDNGA Task Team Leader Isy Faingold ET Consultant LCSHE Economist Christoph Kurowski Sector Leader LCSHD Sector Leader Chingboon Lee Sector Manager LCSHE Sector Manager Tatiana Proskuryakova Senior Operations Officer LCSHE Senior Operations Officer

Tomas Socias Senior Procurement Specialist LCSPT Senior Procurement Specialist

Juan Carlos Serrano Machorro

Senior Financial Management Specialist

LCSFM Senior Financial Management Specialist

Antonella Novali ET Temporary LCSHE Program Assistant Supervision/ICR Rafael E. de Hoyos Navarro

Senior Economist GEDDR Task Team Leader

Raja Bentaouet Kattan Program Leader AFCF1 Task Team Leader Wendy Cunningham Program Leader LCC1C Program Leader Reema Nayar Practice Manager GEDDR Practice Manager Janet K. Entwistle Senior Operations Officer GEDDR Senior Operations OfficerCiro Avitabile Economist GEDDR Economist Abril Alicia Ibarra Castaneda

ET Consultant GEDDR Education Specialist/ICR Author

Vicente Garcia Moreno Consultant GEDDR ICR Co-Author M. Yaa Pokua Afriyie Oppong

Senior Social Development Specialist

GSURR Senior Social Development Specialist

Juan Carlos Serrano Machorro

Senior Financial Management Specialist

GGODR Senior Financial Management Specialist

Gabriel Penaloza Procurement Specialist GGODR Procurement Specialist Fabiola Altimari Senior Counsel LEGLE Senior Counsel Anna Popova ET Consultant LCSHE ET Consultant

28

Christoph Kurowski Sector Leader LCSHD Sector Leader Erik A. Bloom Senior Economist LCSHE Task Team Leader Isy Faingold ET Consultant LCSHE Economist Maria E. Colchao Senior Program Assistant LCSHE Senior Program Assistant (b) Staff Time and Cost

Stage of Project Cycle Staff Time and Cost (Bank Budget Only)

No. of staff weeks USD Thousands (including travel and consultant costs)

Lending FY09 2.92 14.03 FY10 39.71 214.07 Total: 42.63 228.10 Supervision/ICR FY11 25.91 112.67 FY12 23.16 115.49 FY13 33.54 129.27 FY14 24.89 79.40 FY15 4.50 18.67 Total: 112.00 455.50

29

Annex 5. Beneficiary Survey Results Not Applicable

30

Annex 6. Stakeholder Workshop Report and Results Not Applicable

31

Annex 7. Summary of Implementing Agency’s ICR This is a summary of the Borrower’s “Informe de Cierre de 2008-2013” translated into English and redacted by the Bank. Any mistakes are the sole responsibility of the Bank. A. ASSESSMENT OF THE PROJECT DEVELOPMENT OBJECTIVE This section describes the Project Development Objective and its components.

B. ACHIEVEMENT OF OBJECTIVES The long term development objective specified that the Project should improve quality of education as measured by three indicators: Coverage of PEC, Social Participation, and Educational Outcomes.

Coverage: As shown in the Figure 1, the target of the percentage of schools participating in PEC was exceeded in all years. The Project’s restructuring set a goal for 2013, which follows the trend of steady growth projected in previous years. In 2009, when the baseline was established, the number of schools in PEC was preliminary with a figure of 39,180; however, this baseline was adjusted in the April 2013 restructuring to 40,790 so the percentage of participating schools was 20.3 percent, rather than 18 percent as initially calculated.

Figure 1: Percentage of Basic and Special Education Schools Participating in PEC

Social Participation: Data was collected in 2009 (baseline) and in 2012 (follow up). The data was collected through a survey administered to a representative sample of randomly selected schools nationwide. The main results are presented in Table 1.

Table 1: Main Indicators to Measure Social Participation

Indicator Baseline

2009 2012 ∆

1. Percentage of community members that know the PETE 76.8% 83.5% + 6.7%***

2. Percentage of community members that participate in the design and adjustment of PETE

74.2% 72.7% ---

3. Percentage of community members that observe that decisions related to school aspects are taken in a participatory manner among school directors, teachers and parents.

90.2% 93.5% + 3.3%***

The column ∆ (change) refers to the differences between the indicators in 2008 and 2012 and if these are statistically significant. --- not significant, *significant at 10%, ** significant at 5% and *** significant at 1%.

20.321

222320.1

22.3

26.7

23.8

2009 2010 2011 2012 2013

Schools

Restructuring Target

Schools with technical support and financing ‐ CIA Database

32

Educational Outcomes: Overall educational outcomes of PEC schools improved, in terms of average test scores in math and Spanish in ENLACE. Figures 2 and 3 show ENLACE test scores for PEC schools throughout the length of the Project.

Figure 2: Average ENLACE Percentages for PEC schools, primary

Figure 3: Average ENLACE Percentages for PEC schools, secondary

The following table illustrates some of the achievements of the Project and its PDO.

Achievements Status

1. Number of schools in PEC increases 25% (from 39,180 in 2009 to approximately 50,000 in 2013)

Met. The program had 48,938 in the school year 2012-13 and has as a current figure of 50,624 beneficiary schools for the 2013-14 school year. The Program met its objective of increasing from 39,180 in 2009 to 50,624 in 2013.

2. Percentage of parents that participate in the design of the PETE increases from 74.2% in 2009 to 80% in 2013.

Not Met: As mentioned in the social participation section, the percentage of community members that participate in the design and adjustment of the PETE reported in the first data collection in 2009 was of 74.2% and it was expected that by 2013 the percentage would be of 80%. In the second data collection exercise, the percentage was of 72.7%, with a slight decrease of 1.5%; however, this figure was not statistically significant. The Program aimed for a third round of data collection to better inform these results, but, the administrative process to obtain the required budget for this exercise has not been concluded.

3. The percentage of PEC schools located in highly marginalized and marginalized areas increases from 40% in 2009 to 50% in 2013.

Exceeded: The Program met the specified percentage by obtaining a percentage of schools in highly marginalized and marginalized areas participating in the Program of 51.35% for the school year 2013-14.*

4. The percentage of schools with students with indigenous students

Met. PEC has encouraged the inclusion of schools with indigenous students. For the school year 2010-11, the percentage of schools with indigenous

503.2

513.1531.4

559.8

493.8

499.2

513.2524.2 533.2

2008 2009 2010 2011 2012

Average Percentage in Spanish in Primary DGAG Database

Average Percentage in Math in Primary DGAG Database

488.5485.2

471.0476.0 481.2

505.0

493.7507.4

517.7

537.6

2008 2009 2010 2011 2012

Average Percentage in Spanish in Secondary DGAG Database

Average Percentage in Math in Secondary DGAG Database

33

Achievements Status increases from 7% in 2009 to 15% in 2013.

students that participated in PEC was 17% and for the school year 2013-14 that percentage was of 19.3%.

* These figures were obtained with the M-U-R-CIA (Marginación Urbana Rural establecida por la Coordinación de Información y Análisis) method, which uses the urban level of marginalization determined by AGEBs (Áreas Geoestadísticas Básicas) and the level of marginalization by locality according to information from the national census. The method prioritized the urban level of marginalization in those schools where there is no marginalization level and put in second place the level of marginalization by locality.

(This section also mentions the triggers for a third phase of the APL and their status, not included in this translation.) Achievements by Component

Component 1: Direct Support to Schools PEC Phase X, for the 2010-11 school year. This phase involved 32 states that made their contributions according to PEC’s Operating Rules in an amount of MX$386,642,367.00. The Federal Government contributions were of MX$1,159,927,101.00. Some states also made additional contributions in the amount of MX$130,811,497.30, receiving a federal match of MX$154, 357,566.00 at a rate of 1.18 federal Mexican Pesos per each state’s Mexican Peso. PEC Phase XI, for the 2011-12 school year. This phase involved 32 states that made their contributions according to PEC’s Operating Rules in the amount of MX$416,795,291.00, with a federal match of MX$1,250,385,873.00. In this period, states also made additional contributions of MX$84,247,146.00, receiving a federal match of MX$151,644,862.80, at a rate of MX$1.80 for every state Peso contributed. PEC Phase XII, for the 2012-13 school year. This phase only involved 30 states that made their contributions according to PEC’s Operating Rules in the amount of MX$362,425,019.00, receiving a federal match of MX$1,087,275,056.89. In this period, states contributed additional resources in the amount of MX$95,936,053.00, receiving a federal counterpart of MX$95,936,053.00, at a rate of MX$1.00 for every state Peso contributed. Note that in this phase, Sonora and Veracruz did not participate in PEC. PEC Phase XIII, for the 2013-2014 school year. This phase only involved 30 states that made their contributions according to PEC’s Operating Rules in the amount of MX$459,406,529.00, receiving a federal match of MX$1,378,219,586.80. In this period, states made additional contributions in the amount of MX$31,808,411.04, receiving a federal counterpart amount of MX$31,808,411.04, at a rate of MX$1.00 for every state Peso contributed. Note that in this phase, Chiapas and Veracruz did not participate in PEC. Component 2: Monitoring and Oversight The Project supported the delivery of technical assistance to the 32 states that included: 1. Advisory Services. Provided relevant technical assistance for the effective implementation of

the different operational processes of PEC, such as Operating Rules, calls for proposals, review process, incorporation, follow-up, assessment, transparency and accountability.

2. Accompanying Services. In states that required assistance from PEC’s national coordination unit, CNPEC worked closely with the PEC state coordination to ensure the adequate

34

implementation of the Project. 3. Supervision Visits. In states where areas of improvement were identified, mainly due to

constant changes in state personnel, the Federal Government made several visits to provide support to the states on administrative, technical, and operational shortcomings.

4. Capacity Building Events. Initiatives at both the national and state level to foster stakeholder competencies in educational management and the leadership of supervisors and school directors of participating schools.

5. National and regional meetings. With the goal of following up, providing feedback, and sharing good practices, national and regional meetings were held with state teams and, in some cases, school directors and supervisors, parents, and education authorities.

Other activities under this Component were financed with Government resources. Component 3: Policy Development and Evaluation This component was eliminated in the restructuring approved by the Bank on May 2, 2013. However, many of the activities envisioned were undertaken using non-Project resources. Among others: Annual External Evaluation Reports were conducted by CONEVAL. The Centro de Estudios Educativos, A. C. performed the Feasibility Analysis for PEC’s Impact

Assessment and concluded that the program has the necessary and sufficient characteristics to measure its impact on student learning.

A meeting with staff of the General Directorate of Indigenous Education (Dirección General de Educación Indígena, DGEI) was conducted to present the origins, history, evolution, and main guidelines of PEC and its Strategic Model of School Based Management (MGEE), the simplified PETE, to implement the Indigenous Peoples Plan and promote increased participation of indigenous schools in PEC.

In 2012, the Centro de Estudios Educativos, A. C. conducted a study to integrate a State of the Art of the Strategic and Participatory Management Process of the PEC and Escuela Segura programs.

A Beneficiary Satisfaction Survey was conducted in 2012 by the DGAG to explore the degree of satisfaction of school directors, in terms of the goods and services that PEC provided to schools.

PEC was assessed as a federal program with a “high potential of performance” by the Performance Index of Federal Public Programs (Índice de Desempeño de los Programas Públicos Federales, INDEP 2012), meaning that it has a sound design and that it has adequately achieved its goals, but might lack enough resources to attend all its potential beneficiaries.

C. MAIN FACTORS AFFECTING THE IMPLEMENTATION OF THE PROJECT Factors outside the control of the Government and/or PEC. Although the educational federalism and autonomy exercised by the states has strengthened the program, it has also resulted in a number of drawbacks in its implementation and monitoring:

(i) Budget planning in the states has caused delays in the definition of participating schools, and even non-participation in the program, at certain stages of its implementation.

35

(ii) In some cases, the extensions granted for delivery of the matching resources from states have delayed the access of schools to federal resources and PEC resources in some states.

(iii) The lack of uniformity in the implementation and dispersion of resources subject to state matching complicates the periodic reconciliation of the amounts at the central level and the timely delivery of resources to schools, often causing panic buying and poor strategy implementation and reporting at the school level.

(iv) The resistance of some agencies involved in implementing the program, such as teachers’ unions, which have resisted, in some states, the application of standardized assessments and consolidation of the Program.

(v) The Execution System for the Procurement Plan (Sistema de Ejecución para el Plan de Adquisiciones, SEPA) should improve efficiency in procurement and serve as a tool to promote transparency and strengthen accountability instead of an obstacle.

Factors generally subject to the Government. (i) The rotation of mid-level managers and operational staff caused by the instability of their

working conditions, both at the federal and state level, have caused a lack of continuity and adequate follow-up and monitoring processes and integration of program information.

(ii) The lack of personnel in the state and the national coordination units to support the implementation of PEC in schools in remote areas.

Factors generally subject to PEC. The Information System, SIPEC, was not as effective and efficient as expected in recording, organizing, and exploiting information from the participating schools due to technical failures, server capacity, compatibility, operation and maintenance. Costs and Project Financing (i) Specify the date of the last disbursement of the Bank loan. Letter ICL-413-2014 dated June 18,

2014 issued by NAFIN notified that the Bank completed Disbursement Request No. 8, thereby disbursing the full amount of Loan 7948-MX.

(ii) Comments on the loan disbursement schedule. Although SEB sought to comply with the

disbursement projections established in the PAD, they were not met mainly due to a delay in signing of the Loan Agreement. This did not, however, present any difficulty to SEB in executing the entire loan and only involved a change in the amounts to be disbursed for the fiscal year, as per the table below.

Years PAD Estimates Actual Disbursement

2010 0.00 44,550,000.00

2011 116,000,000.00 70,305,704.90

2012 72,000,000.00 73,657,750.47 2013 3,200,000.00 30,763,060.48

2014 0.00 723,484.15

(iii) Comments regarding the availability of funds. National counterpart funding was considered exclusively for Category 1 (Goods, Works and Consultant’s Services supported under direct support to schools). This represented 10% of the total eligible resources for this Category.

(iv) Summarize the results of the Project audits and the level of compliance with the audit recommendations. The Borrower’s Informe de Cierre provides a detailed explanation of

36

audits: 2/2012 for the execution of 2011 funds; 5/2013 for the execution of 2012 funds; 2/2014 for the execution of 2013 funds; 234 for the Fideicomiso Fondo Nacional para Escuelas de Calidad; and Auditoría Externa for the Fideicomiso Fondo Nacional para Escuelas de Calidad.

(v) Evaluate the system of accounting and financial control of the Project

(Coordination of Financial Management). The Coordination of Financial Management (CGF) at all times complied with the goals and commitments outlined in the Loan Agreement and its Annexes, the Operational Manual, and the Contrato de Mandato, in coordination with each of the participants in the Project implementation areas. This unit (CGF) was responsible for implementing transfers to the states, requesting disbursements, and compiling quarterly and semiannual monitoring reports while ensuring effective and appropriate management of resources and information systems. Some systems implemented during project execution were: Planning system: based on the Expenditure Budget of the Federation (PEF) for each fiscal

year, as well as financial planning as established in the PAD. Cash Flow System: Using defined processes for transfers to states that allowed the

monitoring and recording of financial transactions. Information system: The CGF was responsible for compiling the quarterly, semi-annual,

and monitoring reports describing the financial and operating performance of the Project in a given period in relation to the components.

Control System: physical and electronic records were generated of all documents related to the Loan for internal control and audits carried out for the Project.

In regards to contractual and statutory obligations of the Loan, the SEB complied every time and always under the accompaniment of NAFIN. (vi) Assess the procurement system. The SEPA web-based system was used for the reporting of

the Procurement Plan. The information was uploaded into the system according to DGDGIE’s planning for transfers and procurement. No consulting services were carried out under Bank procurement processes. SEB, therefore, requested a reallocation of Loan proceeds to increase the amount allocated to Category 1 (Goods, Works and Consultant’s Services supported under direct support to schools) in order to increase the number of beneficiary schools. This also reflected SEB’s inability to exercise all of the resources for Category 2 (Incremental Project Costs) and Category 3 (Consultant Services). It is important to note that consulting services were carried out under national processes. The inability to perform these services was due mainly to issues that did not depend directly on the implementing agency.

Category Original Amount

Reallocated Amount Modified Amount

1. Goods, Works and Consultant’s Services supported under direct support to schools

211,720,000.00 6,729,769.00 218,449,769.00

2. Incremental Project Costs 3,300,000.00 -2,299,769.00 1,000,231.00

3. Consultant Services 4,430,000.00 -4,430,000.00 0.00

4. Front-end Fee 550,000.00 0.00 550,000.00

37

D. PROJECT SUSTAINABILITY The Project is financially sustainable, derived from the interest of the SEB to continue receiving the Bank’s support and technical assistance.

PEC is a SEP initiative aimed at strengthening basic education and SBM as part of the provisions of Articles 3. of the Constitution; 77 of the Federal Budget and Fiscal Responsibility; 176 of the Rules; 30, 31 and 40 and Annex 24 of the Expenditure Budget of the Federation for Fiscal Year 2015. The Government has the obligation of improving educational processes and outcomes. It is necessary to provide an inclusive education that respects and values diversity sustained in intercultural relations, which successfully combines fairness with quality in the pursuit of greater equality of opportunity for all Mexicans. In the Decree of Articles 3 and 73 of the Constitution posted February 26, 2013 in their Fifth Transitory, Section III, paragraph a) is amended, establishing the need to "strengthen the autonomy of school management in order to improve its infrastructure, purchase educational materials, solve problems and promote basic operating conditions of participation for students, teachers and parents, under the leadership of the director, be involved in resolving the challenges that each school faces.” In this context, Constitutional Article 3 adds, among other things: "The State shall guarantee the quality of compulsory education so that the educational materials and methods, school organization, educational infrastructure and the adequacy of teachers and managers ensure crowning achievement of student learning." The General Education Law, Rules 12, Section V and XII Bis; 25 and 28 Bis establishes guidelines for the Program to reorient their strategies and actions to contribute to the fulfillment of the constitutional mandate for autonomy in school management. The National Development Plan 2013-18 establishes as a national goal “Mexico with Quality Education.” (A comprehensive account of the Plan and its relation to school autonomy and SBM is also provided in this section – not included in this translation). In this context, the Program compiles the experience and lessons learned from recent years that has driven the transformation of the management of public elementary schools. The goal is to reorient their strategies and actions to help achieve the constitutional mandate for autonomy in school management, as well as to strengthen actions leading to the formation of the Basic Education System and the provision of technical and financial support to public elementary schools in the creation and consolidation of conditions that enable them to strengthen their organizational culture and performance through the development of school management and autonomy capacities. E. PERFORMANCE EVALUATION OF THE PROJECT This section summarizes the available evidence and information from external evaluations and studies of PEC since the beginning of the Second Phase of the School Based Management Project. Program Strengths Coverage and Targeting. There is great progress in terms of coverage (schools in 90% of

municipalities nationwide participate in the program) (PEC, 2012). In recent years, targeting efforts increased participation of indigenous schools and schools in areas of high and very high marginalization. (World Bank, 2012)

Training and Continuous Professional Development. School directors and other project beneficiaries received training for the development of the PETE and the PAT. (Mayaink, 2008)

38

Achievement and School Performance. The Program reduced dropout rates and decreased student failure and repetition rates (C230 AC, 2012).

Communication and Dissemination. The Program offers vast public information on the regulations, beneficiaries, and main results, which can be found on PEC’s website. (Bracho González, 2008)

Operation of the Program. The Program has documented procedures and mechanisms for its operation, such as receipt, recording and processing of applications; the selection of beneficiaries; and the granting of school support at every stage. (White 2011-2012). SEP had steady and constant support from the Bank throughout Project implementation. Monitoring and support was provided by each of the three Task Team Leaders and their teams. The Bank also provided technical assistance to solve problems arising from Project implementation. The Bank played an active role in detecting, reporting and working in those areas that needed improvement. This observation applies equally to the financial agent, NAFIN, which was key to Project performance through the provision of strategic support when needed.

Program Weaknesses Coverage and Targeting. Targeting criteria was limited by the selection criteria, which included

all public schools located in areas of "Very High, High, Medium, Low, and Very Low marginalization” (Briefing Note, Target Population. CNPEC 2011 and White, 2011-2012). There is no strategic planning related to the Program’s coverage, at least in the short and medium term (White 2011-2012).

School Management and Strategic Planning. The Program can create an administrative and academic overload to the schools (Miranda Lopez, 2008 and Centre for Educational Studies, 2012).

Training and Continuous Professional Development. Insufficient training and technical, academic, and institutional support for developing the necessary theoretical and practical skills directly related to school management. (Velázquez, 2010). Lack of knowledge regarding the adequacy and usefulness of the training received by school directors in the development of PETE and PAT. (White 2011-2012)

Achievement and School Performance. The impact of the Program on educational achievement and teacher performance does not grow in relation to the number of participating schools and school directors trained in strategic management. (CONEVAL, 2008)

Perception of the Value and Utility of PEC. Information related to the perceptions about PEC of General Directors in each state are collected, but not about the perception of direct beneficiaries. (White 2011-2012)

Program Operation. There is a lack of effective engagement of Supervisors and Technical Educational Consultants (ATP) to provide advice and support in the Program’s operation at the school level (Centre for Educational Studies, 2012). The positive experiences of the participating schools are not exploited. The schools’ technical and pedagogical reports are not used adequately to better understand the problems they face (in particular for the urban and marginalized schools) (Bracho González, 2008). Lack of stability of PEC staff and high uncertainty in job permanency and job benefits. Particularly in the case of ATPs. (Bracho Gonzalez, 2008)

Accountability. Failure in measuring activity level indicators for the components in the MIR (White 2011-2012). Need of end of Project indicators to measure the improvement in learning outcomes comparing PEC schools vs non-PEC schools (White 2011-2012). There are problems

39

of consistency in the Operating Rules indicators. Formulas are not sufficiently clear or relevant, or have no relevance for measuring institutional program objectives (CONEVAL, 2008).

Information systems and evaluation. The Program needs a comprehensive set of indicators that cover various aspects of school planning and can be measured through surveys administered to representative samples of schools (Bracho González, 2003) (White, 2011-2012). There are limitations to reach valid conclusions about the education system since most data comes from the first years of operation of PEC. (White 2011-2012) There are not sufficient monitoring actions to update the Program’s indicator matrix and insufficient monitoring of the institutional objectives of PEC at the federal and state levels. (Bracho González, 2008). SIPEC fails to provide timely and relevant information. The SIPEC has limitations in monitoring the operation and results of the program. ((Bracho González, 2008 and Santibanez, 2008). The user of SIPEC or the general public cannot verify the source of information with which the application is built and no information on updating databases is provided (White 2011-2012). The databases are disjointed and decentralized, causing confusion and inaccuracies in the data at the national level, entity and program phase (White 2011-2012). Few qualitative studies (different impact assessment) on some of the crucial processes of the program: training of school personnel, processes and formulation of PETE and PAT and their implementation in the schools, social participation and pedagogical changes (White 2011-2012). Lack of and impact evaluation to identify the causal effects of the program with data covering all years of operation (Santibanez, 2008). The Project could not use the funds under the Category 3 of the Bank Loan (Consultant Services) for activities under Component 3 of the Project (Development and Policy Evaluation) and needed to restructure the Project to reallocate from Categories 3 and 2 to 1. This was due to internal administrative obstacles to proceed with the Bank’s procurement guidelines.

Program Opportunities In Relation to the Education System. The contribution of the Program to improve quality of

basic education is complemented with other federal programs that address various issues and efforts to improve educational conditions for students such as Escuela siempre abierta and Oportunidades (White 2011-2012). PEC has the potential to articulate and coordinate all educational actions at the school level from the local to the federal levels. (CONEVAL, 2010)

Risks to the Program School Management and Strategic Planning. Schools have limited autonomy in terms of

personnel management (World Bank, 2012) Education System Facts that Affect PEC’s functioning. The education system does not have a

comprehensive database of schools, the population served, and the communities where they are located (Centre for Educational Studies, 2012). At the federal level, there is a lack of articulation between the programs that aim to reach schools and the state teams have limitations for training and providing support to PEC schools (Centre for Educational Studies, 2012). ENLACE shows limitations to measure the specific impact of PEC in students’ educational attainment over time (Bracho González, 2008 and Center for Educational Studies, 2012).

School-level Facts that Affect PEC’s functioning. There are important challenges to build a common vision that gives identity and helps school stakeholders commit to school-based management (Miranda Lopez, 2008). Schools have weak capacities to build a shared vision and act collectively, identify and prioritize school problems, articulate effective and collective

40

interventions to solve those problems, and lack feedback and evaluation mechanisms. (Miranda Lopez, 2008).

State-level Facts that Affect PEC’s functioning. In some cases, noncompliance of the national selection and targeting criteria by the state education authorities result in a large variation in the understanding of the program and its institutional objectives (Abreu, 2010).

F. LESSONS LEARNED The 4 years of operation of Phase II of PEC have produced important lessons learned that have made the Program and education policies around it evolve, including: PEC X contributed to the improvement of educational attainment through fostering strategic

educational management; PEC XI established strategic educational management to strengthen the organizational culture

and functioning of schools, focusing on students’ needs in order to contribute to improved learning outcomes;

PEC XII helped improve the academic achievement of students in public elementary schools, sharpening targeting criteria (equity) and establishing a regional vision; and,

PEC XIII strengthened autonomy and SBM and encouraged collaboration of the school community in solving the challenges faced by each school for the improvement of learning outcomes.

PEC established a model of SBM based on democratic principles that identify the school as the center of all educational initiatives and drives on the school community to strengthen autonomy, as well as the principles of equity, relevance, effectiveness, and efficiency to promote quality of education through various elements. Among these are shared leadership, collaborative work, strategic planning, monitoring and evaluation for continuous improvement, responsible social participation, shared decision making, flexible practices and relationships, project financing, and transparency and accountability. The development of these elements in PEC participating schools have created a more effective use of resources and better identification of school needs. Ultimately, PEC has helped improve the quality of education as a result of a more efficient and transparent use of resources to generate more relevant and contextualized actions at the school level. PEC has contributed to a more open and welcoming school environment because the community is involved in its administration. Increased participation of all local stakeholders in decision-making processes also encourages more participatory, collaborative decision-making; greater satisfaction for all students; better student performance as a result of lower repetition and dropout rates, and eventually better learning outcomes. Finally, it should be noted that these lessons are players in the current educational reform to strengthen autonomy and school management. In addition, basic education schools have a school organization that ensures maximum learning achievement for all students by ensuring the effectiveness and efficiency of educational services provided in schools.

41

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Not applicable

42

Annex 9. List of Supporting Documents Abreu, R. C.-M. (2010). Evaluación de impacto del proyecto piloto PEC-FIDE. Fundación IDEA A.C. Recuperado de SEP-PDF. Blanco, E. (2011-2012). Evaluación de Consistencia y Resultados 2011-2012. Blimpo, M. and D. Evans. 2011. School-Based Management and Educational Outcomes: Lessons from a Randomized Field Experiment. World Bank's Africa Program for Education Impact Evaluation. Available at: http://siteresources.worldbank.org/EDUCATION/Resources/Blimpo-Evans_WSD-2012-01-12.pdf Borman. G. G.M. Hewes, L.T. Overman and S. Brown. 2003. Comprehensive School Reform and Achievement: A Meta-Analysis. Review of Educational Research. American Education Research Association. Summer. Vol. 73. No. 2. pp. 125–230. Bracho González, T. (2003). Programa Escuelas de Calidad PEC 2002. Segunda evaluación externa. CIDE. Bracho González, T. (2008). Informe Final de la evaluación de consistencia y resultados del Programa Escuelas de Calidad PEC VII. Available at: http://basica.sep.gob.mx/pec/pdf/evaluaciones/informe2007.pdf. C230 A.C. (2012). Evaluación no experimental del PEC. México: C230. Centro de Estudios Educativos. (2012). Análisis de los materiales encontrados e identificación de fortalezas y debilidades de los programas. CONEVAL. (2008). Informe de la Evaluación Específica de Desempeño . CONEVAL. (2010). Informe de la evaluación específica de desempeño 2009-2010. De Hoyos. (2010) Dinero y Crisis Educativa. Revista Nexos. Available at: http://www.nexos.com.mx/?p=14061 Duflo, E. P. Dupas and M. Kremer. Revised September 2014. School Governance, Teacher Incentives, and Pupil-Teacher Ratios: Experimental Evidence from Kenyan Primary Schools. Working Paper available at: http://web.stanford.edu/~pdupas/ETP_paper.pdf Gertler, P.J., H.A. Patrinos and M. Rubio-Codina. September 2012. Empowering parents to improve education: Evidence from rural Mexico. Journal of Development Economics. Volume 99. Issue 1. pp. 68–79. Government of Mexico. 2013. Plan Nacional de Desarrollo, 2013-2018. Poder Ejecutivo Federal (Presidencia de la República). Mexico.

43

Hanushek E.A, E. Linkd, and S.Woessmann. L. September 2013. Does school autonomy make sense everywhere? Panel estimates from PISA. Journal of Development Economics. Volume 104. pp. 212–232. Mayaink, G. (2008). Levantamiento de encuestas de impacto entre los actores escolares (padres de familia, alumnos, directores de planteles, y supervisores escolares) de escuelas incorporadas al Programa Escuelas de Calidad (PEC). Miranda López, F. S. (2008). Programa Escuelas de Calidad.Evaluación Externa 2008. Recuperado de FLACSO-PDF. Murnane R.J., S.J. Willet and S. Cardenas 2006 Did the Participation of Schools in Programa Escuelas de Calidad (PEC) influence students’ outcomes. Working Paper, Harvard University, Graduate School of Education, Cambridge, MA. PEC. (2012). Lorgros yResultados del PEC 2001-2012. México: PEC Pradhan, M.and J. de Ree. June 2014. District Governance and Student Learning in Indonesia. Asian Development Bank. Economics Working Paper Series. 397. Pradhan, M. et al. September 2011. Improving Educational Quality through Enhancing Community Participation Results from a Randomized Field Experiment in Indonesia. Policy Research Working Paper 5795. The World Bank. East Asia and Pacific Region. Human Development Sector Department Santibáñez, L. A. (2008). Evaluación específica de desempeño. c230. Secretaría de Educación Pública (Diario Oficial de la Federación). December 28, 2013. ACUERDO número 703 por el que se emiten las Reglas de Operación del Programa Escuelas de Calidad. Secretaría de Educación Pública (Diario Oficial de la Federación). December 28, 2013. ACUERDO número 704 por el que se emiten las Reglas de Operación del Programa Escuelas de Tiempo Completo. Secretaría de Educación Pública. 2013. Programa Sectorial de Educación 2013-2018. Mexico. Secretaría de Hacienda y Crédito Público, Secretaría de Educación Pública y Nacional Financiera, S.N.C., I.B.D. Informes Semestrales de Avance. Proyecto Escuelas de Calidad, Fase II. Préstamo 7948MX. Skoufias, E. and J. Shapiro. 2006. Evaluating the Impact of Mexico's Quality Schools Program: The Pitfalls of Using Nonexperimental Data. World Bank, Washington, DC. Available at: <https://openknowledge.worldbank.org/handle/10986/9010> Velázquez, C. (2010). Evaluación Específica de desempeño 2010-201

44

World Bank. (2012). Good to Great through School Autonomy and Accountability in Mexico. World Bank. May 20, 2010. Project Appraisal Document on a Proposed Loan in the amount of US$220 Million to the United Mexican States for a School-Based Management Project in support of the Second Phase of the School-based Management Program. Report No. 52955-MX. Washington, DC. World Bank. November 9, 2010. Loan Agreement. LOAN NUMBER 7948-MX. Washington, DC. World Bank. April 19, 2013. Restructuring Paper on a Proposed Restructuring of the School based Management Project – Phase II. Loan 7948-mx Approved by the Board on June 17, 2010 to the United Mexican States. Report No. 73325-MX. Washington, DC. World Bank. May 2, 2013. Amendment to the Loan Agreement (73325-MX). Washington, DC. World Bank. Aide Memoires and Back-to-Office reports for Preparation and Supervision missions. World Bank. Implementation Status Reports (ISRs) for the School-Based Management Project in support of the Second Phase of the School-based Management Program.

45

MAP