world bank documentdocuments.worldbank.org/curated/en/...target years indicator 1 : lower secondary...

Document of

The World Bank

Report No: ICR00003667

IMPLEMENTATION COMPLETION AND RESULTS REPORT

(IBRD-74300)

ON A

LOAN

IN THE AMOUNT OF US$ 80.0 MILLION

TO THE

REPUBLIC OF GUATEMALA

FOR AN

EDUCATION QUALITY AND SECONDARY EDUCATION PROJECT

June 30, 2016

Education Global Practice Management Unit

Central America Country Management Unit

Latin America and the Caribbean Regional Office

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

ii

CURRENCY EQUIVALENTS

(Exchange Rate, Effective November 30, 2015)

Currency Unit = Quetzal

US$ 1.00 = Q7.63

Q1.00 = US$0.13

FISCAL YEAR

January 1 – December 31

ABBREVIATIONS AND ACRONYMS

BIE Bilingual Intercultural Education

CAS Country Assistance Strategy

CNB Currículum Nacional Básico (Standard National Curriculum)

DIGEBI Dirección General de Educación Bilingüe Intercultural (General Directorate

for Intercultural Bilingual Education)

DIGEDUCA Dirección General de Evaluación e Investigación Educativa (General

Directorate for Educational Assessment and Research)

DIGEFOCE Dirección General de Fortalecimiento de la Comunidad Educativa (General

Directorate for Strengthening Community-Based Education)

DIGEMOCA Dirección General de Monitoreo y Verificación de la Calidad (General

Directorate for the Monitoring and Assessment of [Educational] Quality)

DIGEPSA Dirección General de Participación Comunitaria y Servicios de Apoyo

(General Directorate for Community Participation and Support Services)

DINFO Dirección de Informática (Directorate for Information Technology)

DIPLAN Dirección de Planificación Educativa (Directorate for Education Planning)

EMIS Education Management Information System

GER Gross Enrollment Rate

GCR Gross Completion Rate

GoG Government of Guatemala

IBRD International Bank for Reconstruction and Development

IBEC Integrated Basic Education Center

ICR Implementation Completion and Results

INEB Instituto Nacional de Educación Básica (Regular Public Lower Secondary

Education School)

IRI Intermediate Results Indicator

IPP Indigenous Peoples Plan

ISR Implementation Status Report

LSE Lower Secondary Education

M&E Monitoring and Evaluation

MINEDUC Ministerio de Educación (Ministry of Education)

NGO Non-Governmental Organization

NPV Net Present Value

NUFED Núcleo Familiar Educativo para el Desarrollo (Educational Family Nucleus

for Development)

P1/P2/P3 Period 1 / Period 2 / Period 3

PAD Project Appraisal Document

iii

PADEP/D Programa Académico de Desarrollo Profesional para Docentes (Academic

Professional Development Program for [In-Service] Teachers)

PDO Project Development Objective

PEPS Programa de Educación Primaria para Estudiantes con Sobre-edad

([Accelerated] Primary Education Program for Overage Students)

PIU Project Implementation Unit

RFM Results Framework and Monitoring

TI Telesecundaria (Tele-secondary) Institution

TTL Task Team Leader

USAC University of San Carlos de Guatemala

Senior Global Practice Director: Claudia Costin

Practice Manager: Reema Nayar

Project Team Leader: Juan Diego Alonso

ICR Team Leader: Juan Diego Alonso

ICR Authors: Juan Diego Alonso, Wendy de Leon and

Andrew Trembley

iv

GUATEMALA

EDUCATION QUALITY AND SECONDARY EDUCATION

CONTENTS

DATASHEET ................................................................................................................................. v 1. Project Context, Development Objectives and Design ............................................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................................. 7

3. Assessment of Outcomes .......................................................................................................... 13 4. Assessment of Risk to Development Outcome ......................................................................... 24 5. Assessment of Bank and Borrower Performance ..................................................................... 24

6. Lessons Learned........................................................................................................................ 26 7. Comments on Issues Raised by Borrower and Implementing Agencies .................................. 28

Annex 1. Project Costs and Financing .......................................................................................... 29 Annex 2. Outputs by Component.................................................................................................. 30 Annex 3. Economic and Financial Analysis ................................................................................. 34

Annex 4. Bank Lending and Implementation Support/Supervision Processes ............................. 42 Annex 5. Beneficiary Survey Results ........................................................................................... 45

Annex 6. Stakeholder Workshop Report ...................................................................................... 46 Annex 7. Summary of Borrower's ICR and Comments from the Borrower to Draft ICR ........... 47

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................................... 58 Annex 9. List of Supporting Documents ...................................................................................... 59

Annex 10. Indicators Used to Measure Efficacy .......................................................................... 65 Annex 11. Beneficiaries ................................................................................................................ 67 Annex 12. An Assessment of the Indigenous Peoples Plan’s Achievements ............................... 69

MAP .............................................................................................................................................. 71

v

DATASHEET

A. Basic Information

Country: Guatemala Project Name:

EDUCATION

QUALITY AND

SECONDARY

EDUCATION

Project ID: P089898 L/C/TF Number(s): IBRD-74300

ICR Date: 06/30/2016 ICR Type: Core ICR

Lending Instrument: SIL Borrower: REPUBLIC OF

GUATEMALA

Original Total

Commitment: USD 80.00M Disbursed Amount: USD 80.00M

Revised Amount: USD 80.00M

Environmental Category: C

Implementing Agencies: Ministry of Education

Cofinanciers and Other External Partners: None

B. Key Dates

Process Date Process Original Date Revised / Actual

Date(s)

Concept Review: 07/14/2005 Effectiveness: 07/02/2007 06/23/2008

Appraisal: 07/25/2006 Restructuring(s):

06/15/2009

01/29/2010

06/06/2011

04/12/2012

11/21/2013

Approval: 03/06/2007 Mid-term Review: 12/31/2010 12/08/2011

Closing: 06/30/2013 11/30/2015

C. Ratings Summary

C.1 Performance Rating by ICR

Outcomes: Moderately Satisfactory

Risk to Development Outcome: Substantial

Bank Performance: Moderately Unsatisfactory

Borrower Performance: Moderately Satisfactory

vi

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR)

Bank Ratings Borrower Ratings

Quality at Entry: Moderately Unsatisfactory Government: Moderately Unsatisfactory

Quality of

Supervision: Moderately Unsatisfactory

Implementing

Agency: Moderately Satisfactory

Overall Bank

Performance: Moderately Unsatisfactory

Overall Borrower

Performance: Moderately Satisfactory

C.3 Quality at Entry and Implementation Performance Indicators

Implementation

Performance Indicators

QAG Assessments (if

any) Rating

Potential Problem Project

at any time (Yes/No): No Quality at Entry (QEA): None

Problem Project at any

time (Yes/No): Yes

Quality of Supervision

(QSA): None

DO rating before

Closing/Inactive status: Moderately Unsatisfactory

D. Sector and Theme Codes

Original Actual

Sector Code (as % of total Bank financing)

General public administration sector 12 5

Other social services 3 13

Primary education 11 11

Secondary education 74 71

Original Actual

Theme Code (as % of total Bank financing)

Decentralization 13 3

Education for all 25 82

Indigenous peoples 24 1

Rural services and infrastructure 25 1

Social Safety Nets/Social Assistance & Social Care

Services 13 13

vii

E. Bank Staff

Positions At ICR At Approval

Vice President: Jorge Familiar Pamela Cox

Country Director: J. Humberto Lopez Jane Armitage

Global Practice Senior Director: Claudia Costin

Practice Manager: Reema Nayar Eduardo Velez Bustillo

Project Team Leader: Juan Diego Alonso Joel E. Reyes

ICR Team Leader: Juan Diego Alonso

ICR Primary Co-Authors:

Juan Diego Alonso

Wendy de Leon

Andrew Trembley

F. Results Framework Analysis

Project Development Objective (PDO) The objective of the Project is to improve access to a quality lower secondary education for low

income students, especially indigenous communities, through improved primary education

completion rates for overage students, strengthened flexible lower secondary education

modalities and school management.

Revised Project Development Objectives (as approved by original approving authority) The PDO did not change throughout the lifetime of the Project, however the Outcome (or PDO-

level) and Intermediate Results Indicators (IRIs) did change significantly in two instances: at

Restructuring No. 2 (January 2010) and at Restructuring No. 5 (November 2013). These changes

to the Results Framework and Monitoring (RFM) table significantly impacted both the emphasis

of the PDO and the usefulness of the original RFM arrangements for evaluating the Project’s

PDO achievement.

Restructuring No. 2 represented a sizeable change to the original Project’s scope in three key

directions:

i) through a bigger focus on access and away from quality in the PDO’s emphasis, with the

incorporation of the payment of teacher salaries as an eligible category under the

Project, with a significant reallocation of Project proceeds which would eventually take

up about a third of the Project’s money;

ii) through a different approach to measuring impact and results, with the elimination of

impact evaluations to the core interventions envisaged in the original design, after the

newly elected Congress that took over in 2008 started discussions of Project

effectiveness1;

1 A thorough description of the original strategies envisaged for undertaking impact evaluations of the specific interventions to be

funded under the Project – integrated basic education centers, flexible modalities for lower secondary education, and school-

viii

iii) through a change in scope for some of the interventions, e.g. from supporting the

upgrading of pre-service teacher training to supporting a new in-service teacher training

program, or from supporting the accelerated primary education program for overage

students in integrated basic education centers to supporting the said program only in

primary education schools.

With the elimination of impact evaluations for most of the Project interventions/activities, the

number and composition of IRIs changed substantially and the RFM table was significantly

revised.

Restructuring No.5 extended the Project by 23 months and made a substantial revision and

update to the RFM table in line with the extended period of implementation2.

Against this background, the section below lists all indicators that were part of the Project at any

given time3 and the evaluation of achievement of these indicators is undertaken using split

analysis across 3 project phases/periods, as per the Implementation Completion and Results

(ICR) Report Guidelines. The three periods are determined based on the two Restructurings that

significantly affected the RFM table of the Project, as follows: Period 1 (P1), from effectiveness

(June 2008) through Restructuring No. 2 (January 2010); Period 2 (P2), from Restructuring No.

2 (January 2010) through Restructuring No. 5 (November 2013); and Period 3 (P3), from

Restructuring No. 5 (November 2013) through closure (November 2015).

(a) PDO-level Indicators

Indicator Baseline Value

Original Target

Values

(from approval

documents)

Formally

Revised Target

Values

Actual Value

Achieved at

Completion or

Target Years

Indicator 1 : Lower secondary education (9th

Grade) completion rate

Value

quantitative or

qualitative)

50% Nationwide

31% Targeted

Municipalities

53% Nationwide

40% Targeted

P2:

53% Nationwide

40% Targeted

P3:

47% Targeted

55% Nationwide

41% Targeted

Date achieved 01/01/2007 12/31/2012 P2: 12/31/2012

P3: 11/30/2015 11/30/2015

based management for quality – can be found in Annex 3, Part 3 (Strategy for evaluation of impact) of the Project Appraisal

Document (World Bank, 2007), pp.61-65.

2 Further details on the changes effected by these two Restructuring processes and why they are considered the two most critical

in terms of measurement of results can be found in Section H below and Sections 1.2 and 1.6 in the main text.

3 Each of the indicators of the original RFM table and the rationale for including them is explained carefully in Annex 3, Part A

(Results Framework) of the PAD, pp.45-49.

ix

Comments

(incl. %

achievement)

Achievement per period:

P1: 129% Achieved; P2: 129% Achieved; P3: 63% Achieved

Achievement rates are calculated as the proportion (percentage) of the original

target. Since there are 3 periods, two of which have two sub-indicators,

achievement rates for these periods (P1 and P2) are calculated as the simple

average of achievement rate for sub-indicator 1 and achievement rate for sub-

indicator 2.

Indicator 2 : Gross enrollment rate in lower secondary education (Grades 7-9)

Value

quantitative or

qualitative)

61% Nationwide

42% Targeted

65% Nationwide

58% Targeted

P2: 71% Nationwide

52% Targeted

P3: 47% Targeted

69% Nationwide

51% Targeted

Date achieved 01/01/2007 12/31/2012 P2: 12/31/2013

P3: 11/30/2015 11/30/2015

Comments

(incl. %

achievement)


P1: 128 % Achieved; P2: 91% Achieved ; P3: 109% Achieved

Indicator 3 : Intake (Access) in grade 7 in modalities supported by the Project

Value

quantitative or

qualitative)

0 103,500

Nationwide

P2: 35,000 in

Telesecundaria

Institutions (TIs);

16,838 in Educational

Family Nucleus for

Development

(NUFEDs)

P3: Dropped

42,968 in TIs

12,016 in NUFEDs

Date achieved 01/01/2007 12/31/2012 12/31/2013 11/30/2015

Comments

(incl. %

achievement)


P1: 51% Achieved; P2: 103% Achieved; P3: Not Assessed

This indicator was originally included to measure the increased access (net intake)

in grade 7 in the modalities supported by the Project, with an original increase of

103,500 students nationwide. The indicator was revised in Restructuring No. 2 to

only make reference to the net intake in grade 7 for two lower secondary education

modalities: TIs and NUFEDs, and targets were set individually for each of these

modalities (35,000 for TIs and 19,364 for NUFEDs) by end-2012. Target for TIs

x

was revised downwards at Restructuring No. 3 (to 31,308) and the end date

revised to 12/31/2013. The target for TIs was revised upwards again and back to

the original number of Restructuring No. 2 (35,000) at Restructuring No. 4. The

indicator was finally dropped at Restructuring No. 5.

Indicator 4 : Proportion of overage pupils (13-15 years old) who complete primary education

Value

quantitative or

qualitative)

52% Higher than year

1 47% 45%

Date achieved 01/01/2007 12/31/2012 12/31/2013 11/30/2015

Comments

(incl. %

achievement)


P1: Not assessed; P2: 105% Achieved; P3: Not Assessed

This indicator was originally included to measure accelerated primary completion

of a cohort of overage students included in an original impact evaluation of this

intervention. At that point, of course, getting a higher proportion of overage

students to complete primary education was the goal. The indicator, however,

changed meaning after the elimination of impact evaluation studies at

Restructuring No. 2, and instead focused on reducing the proportion of overage

students in the pool of students completing primary education at the nationwide

level. Under this new definition, the lower the overage rate at completion, the

better, since a less overage cohort of completers would increase the probability of

access to lower secondary education. At Restructuring No. 5, the indicator was

revised and downgraded to an IRI.

Indicator 5 : Primary education (6th

grade) completion rate

Value

quantitative or

qualitative)

76% Nationwide

78% Targeted

Higher than 72%

Nationwide

P2: 84% Nationwide

82% Targeted

P3: 87% Targeted

84% Nationwide

87% Targeted

Date achieved 01/01/2007 12/31/2012 P2: 12/31/2013

P3: 11/30/2015 11/30/2015

Comments

(incl. %

achievement)


P1: 117% Achieved; P2: 130% Achieved; P3: Not Assessed

The “Targeted” Municipalities sub-indicator was added at Restructuring No. 2. At

Restructuring No. 5, this indicator was downgraded to an IRI. The original target

was based on the original baseline of 72%, but this baseline was updated in final

data from the Ministry of Education (MINEDUC) collected for this report.

Indicator 6:

Percent of 9th

grade graduates achieving competency on a standardized

examination (average of competency rates of Math and Reading tests) in

Telesecundarias and NUFED schools.

xi

Value

quantitative or

qualitative)

11% TIs

7% NUFED

4.4% TIs

6.7% NUFED

7% TIs

5% NUFED

Date achieved 12/31/2009 11/30/2015 11/30/2015

Comments

(incl. %

Achievement)

158% Achieved for TIs; 29% Achieved for NUFEDs

Simple average TIs/NUFEDs: 93% achieved

Weighted average TIs/NUFED: 128% achieved

(see details in Section 3.2 of ICR’s main text)

This indicator was added for the purpose of the ICR as a proxy to measure

education quality. This examination was designed to measure competency to

complete tasks required by 9th

grade student standards and was only carried out in

years 2006, 2009 and 2013. It uses Item Response Theory to ensure comparability

of pass rates over time. Year 2009 is used as a proxy baseline. Targets are

calculated using: (i) the (negative) elasticity of quality (competency rates for 9th

graders) to access (increase in 9th

grade enrollment) from 2006 through 2009,

since these are the two years where national standardized tests were held for all

types of lower secondary institutions (LSE) institutions prior to the Project’s

implementation; and (ii) the 2009-2013 increase in access (percentage increase in

9th

grade enrollment). Year 2013 shows that average quality, as measured by this

indicator, was much better than expected for TIs and much worse than expected

for NUFEDs. A whole elasticity analysis is developed in Section 3.2 of the ICR’s

main text.

(b) Intermediate Results Indicators (IRIs)

Component 1 – Primary Education Completion and Quality

Sub-Component 1.A – Support of Quality Primary Education4


Original Target

Values (from

approval

documents)

Formally

Revised

Target

Values

Actual Value

Achieved at

Completion or

Target Years

Indicator 1 : For the cohort of students who are overage at baseline: 6th

grade completion rate

Value Not available Not defined Not available

4 The original name of this component at appraisal was “Integrated Basic Education Centers” (IBEC). IBECs was the name

given at appraisal to a regular public school that: (i) would combine both a primary school (grades 1-6) with a lower secondary

education modality (grades 7-9); and (ii) that would offer both the accelerated primary education program for overage students

and one of the three lower secondary education flexible modalities originally envisaged (TIs, NUFEDs and NGO-run programs). The emphasis on IBECs was dropped at Restructuring No. 2 and a fully revised name was given to this Sub-Component.

xii

(quantitative

or qualitative)

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

Indicator was supposed to be measured from an impact evaluation of the

accelerated learning program for overage students at integrated basic education

centers (IBECs). With the elimination of both the impact evaluation and the

“IBEC approach” (see footnote 4), this indicator was dropped at Restructuring

No. 2 and cannot be assessed as impact evaluation never started.

Indicator 2 : For the cohort of students who are overage at baseline: Standardized 6

th grade

test scores of students in accelerated learning program

Value

(quantitative

or qualitative)

Not available Not defined Not available

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

Same comments as those for IRI 1 above.

Indicator 3 : Repetition rate for grade 1-6 in the whole student population

Value

(quantitative

or qualitative)


Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.


Indicator 4 : Total (inter and intra-year) dropout rates for grades 1-6 in the whole school

population

Value

(quantitative

or qualitative)


Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.


Indicator 5 : Proportion of non-repeating first graders who have appropriate age

Value

(quantitative Not available 75% Not available

xiii

or qualitative)

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.


Indicator 6 : Primary education completion rate (6th

grade)

Value

(quantitative

or qualitative)

52% Nationwide

78% Targeted

Higher than 52%

(Nationwide)

Higher than 78%

(Targeted)

P2: 84% Nation;

83% Targeted

P3: 87% Targeted

84% Nation

87% Targeted

Date achieved 01/01/2007 12/31/2012 11/30/2015 11/30/2015

Comments

(incl. %

achievement)


P1: Not assessed; P2: Not asssessed; P3: 100% Achieved

Indicator became an IRI at Restructuring No. 5, after having been a PDO-level

indicator until then. For performance in periods P1 and P2, see PDO-level

Indicator 5 above.

Indicator 7 : Number of students in the Accelerated Primary Education Program (PEPS) that

graduate from primary education

Value

(quantitative

or qualitative)

0 33,181 22,100

Date achieved 01/01/2007 11/30/2015 11/30/2015

Comments

(incl. %

achievement)


P1: Not assessed; P2: Not assessed; P3: 60% Achieved

Indicator added at Restructuring No. 5.

Indicator 8 : Number of classes within schools that offer the PEPS in monolingual modality

Value

(quantitative

or qualitative)

0 150 schools 86 classes 59 classes

Date achieved 01/01/2007 12/31/2012 11/30/2015 11/30/2015

Comments

(incl. %

achievement)


P1: Not assessed; P2: 26% achieved; P3: 69% Achieved

Indicator added at Restructuring No. 2 originally as the number of schools

offering the PEPS. It was revised at Restructuring No. 5 to refer indeed to the

number of classes within those schools offering the PEPS.

xiv

Indicator 9 : Number of classes within schools that offer the PEPS in bilingual modality

Value

(quantitative

or qualitative)

0 200 schools 196 classes 206 classes

Date achieved 01/01/2007 12/31/2012 11/30/2015 11/30/2015

Comments

(incl. %

achievement)


P1: Not assessed; P2: 119% achieved; P3: 105% Achieved


Sub-Component 1.B – Support Teacher Training5


Original Target

Values (from

approval

documents)

Formally Revised

Target Values

Actual Value

Achieved at

Completion or

Target Years

Indicator 10 : Test scores in entering exam for pre-service teacher training (Escuela Normal

Modelo)

Value

(quantitative

or qualitative)

Not available Above reference

level Not available

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

This indicator was supposed to annually measure the impact of the pre-service

teacher training program to be upgraded with funding from the Project at 5

regional Pre-Service Teacher Training Centers (known as “Escuelas Normales

Modelo”). This indicator was dropped at Restructuring No. 2 in line with the

change in scope, since no intervention was funded under the Project for this pre-

service teacher training program.

Indicator 11 : Test scores in exit exam for pre-service teacher training (Escuela Normal

Modelo)

Value

(quantitative

or qualitative)


5 The original name of this component at appraisal was “Pre-Service Teacher Training Development”. It was changed at

Restructuring No. 2 because the focus of this Project’s Sub-Component shifted from supporting the development of a new pre-

service teacher training program to be taught at 5 Teacher Training High Schools (known as “Escuelas Normales”) to supporting

the development of a new professional in-service teacher training program to be taught at the USAC.

xv

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.


Indicator 12 : Number of bilingual teachers trained and certified in upgraded pre-service

teacher training (Escuela Normal Modelo)

Value

(quantitative

or qualitative)

0 Not defined Not available

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.


Indicator 13 : Percentage of bilingual teachers among graduates from upgraded pre-service

teacher training program (Escuela Normal Modelo)

Value

(quantitative

or qualitative)

0 Not defined Not available

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.


Indicator 14 : Number of pre-primary and primary education teachers that completed the in-

service teacher training program (PADEP/D) in 196 targeted municipalities

Value

(quantitative

or qualitative)

0 2,126 (primary

teachers) 13,237 13,661

Date achieved 01/01/2007 12/31/2012 11/30/2015 11/30/2015

Comments

(incl. %

achievement)


P1: Not assessed; P2: 407% Achieved; P3: 103% Achieved

Indicator was added at Restructuring No. 2, but only incorporating primary

education teachers. Since the Program then went on to also include pre-primary

education teachers, Restructuring No. 5 revised this indicator for clarity and

added the group of pre-primary teachers to the target, as well.

Component 2 – Access and Quality of Lower Secondary Education

xvi

Sub-Component 2.A – Feasibility studies and strategies for secondary education curricula

and evaluation systems6


Original Target

Values (from

approval

documents)

Formally

Revised

Target

Values

Actual Value

Achieved at

Completion or

Target Years

Indicator 15 : Developed new curriculum school materials and teacher guides

Value

(quantitative

or qualitative)

0% 100%

40% (New curriculum

school materials and

teacher guides

developed for 2 of the

original 5 types of LSE

institutions)

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)


P1: 40% Achieved; P2: Not Assessed; P3: Not Assessed

This indicator was supposed to measure the development of new material under

a substantial curricular reform for secondary education envisaged to be funded

under this Sub-Component. With the elimination of the whole sub-component at

Restructuring No.2, this indicator was dropped. However, the financing of new

curriculum school materials and teacher guides for the two flexible lower

secondary education modalities supported under the Project (TIs and NUFEDs)

was financed off Sub-Component 2.B.

Indicator 16 : Percent of Project-supported schools receiving new curricular orientations and

materials

Value

(quantitative

or qualitative)

0% 100%

40% (New curricular

orientations and

materials received by

100% of 2 of the

original 5 types of LSE

institutions)

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

This indicator was supposed to measure the proportion of schools supported

under the Project that received new curricular orientations and materials.

Project-supported schools, in this case, also included cooperative schools,

regular LSE schools (known as INEBs) and non-governmental organization

6 This sub-component was dropped from the Project at Restructuring No. 2.

xvii

(NGO)-run schools. With the change in scope at Restructuring No.2, INEBs,

Cooperative and NGO-run schools were dropped as eligible institutions, and all

efforts were focused instead on only two flexible LSE modalities (TIs and

NUFEDs), this indicator was dropped at Restructuring No. 2, along with the

elimination of the whole sub-component. However, with funding from Sub-

Component 3.B, 100% of schools offering either TIs or NUFEDs received new

curricular orientations and materials fully aligned with the Standard National

Curriculum (CNB).

Sub-Component 2.B – Strengthening and Expansion of Pertinent and Flexible Modalities

for Lower Secondary Education


Original Target

Values (from

approval

documents)

Formally

Revised

Target

Values

Actual Value

Achieved at

Completion or

Target Years

Indicator 17 : Review, strengthening and accreditation of Lower Education Modalities

Value

(quantitative

or qualitative)

None

Completed for TIs

+ NUFEDs +

100% of

modalities

supported by

NGOs

67% (Review, strengthening

and accreditation was

finalized for TIs and

NUFEDs)

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)



Indicator was dropped at Restructuring No. 2. However, with funding from this

Sub-Component, the review, strengthening and accreditation of TIs and

NUFEDs was completed.

Indicator 18 : Number of new student places in the integrated lower secondary education

modalities

Value

(quantitative

or qualitative)

Grade 7: 0

Grade 8: 0

Grade 9: 0

Grade 7: 15,000

Grade 8: 15,000

Grade 9: 15,000

Not available

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

Indicator was dropped at Restructuring No. 2, with the decision to shift away

from the “integrated basic education center” concept to focus instead on the LSE

level regardless of whether the school also encompassed primary education or

not within the same premises. In reality, the LSE never worked in an integrated

xviii

fashion since. Even if within the same premises of a regular primary school, any

modality worked independently once the IBEC approach was dropped.

Indicator 19 : Number of new student places in the integrated lower secondary education

schools

Value

(quantitative

or qualitative)

Grade 7: 0

Grade 8: 0

Grade 9: 0

Grade 7: 15,000

Grade 8: 15,000

Grade 9: 15,000

Not available

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

Same comments as those for IRI 18 above. Integrated LSE schools did not

materialize in practice.

Indicator 20 : Number of new student places in cooperative lower secondary education schools

with quality accredited modality

Value

(quantitative

or qualitative)

Grade 7: 0

Grade 8: 0

Grade 9: 0

Grade 7: 12,000

Grade 8: 12,000

Grade 9: 12,000

Grade 7: - 89

Grade 8: 5,511

Grade 9: 7,012

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)



Indicator was dropped at Restructuring No. 2 when cooperative lower

secondary education schools were no longer considered eligible schools to be

funded under the Project. Cooperative LSE schools added a total of 12,434 new

students since 2007, although most of the addition was achieved through

improvement in transition rates to 8th

and 9th

grade within these schools.

Cooperative schools had their modality accredited through Presidential Decree

35-2015 (February 4th

, 2015) and their quality standards set in the new By-Law

for the creation of Cooperative LSE institutions.

Indicator 21 : Number of new student places created in lower secondary education through

strategic alliances with NGOs and foundations

Value

(quantitative

or qualitative)

Grade 7: 0

Grade 8: 0

Grade 9: 0

Grade 7: 2,500

Grade 8: 2,500

Grade 9: 2,500

Not available

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

Indicator was dropped at Restructuring No. 2 when lower secondary education

schools run by NGOs and foundations were no longer considered eligible

xix

schools to be funded under the Project. However, private schools, which

encompass NGOs-run and foundations-run institutions, added a total of 28,235

new places since 2007 (2,096 in 7th

grade, 8,178 in 8th

grade, and 17,961 in 9th

grade).

Indicator 22 : Number of new student places created in lower secondary education through

optimization in existing INEB centers

Value

(quantitative

or qualitative)

Grade 7: 0

Grade 8: 0

Grade 9: 0

Grade 7: 5,000

Grade 8: 5,000

Grade 9: 5,000

Not available

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

Indicator was dropped at Restructuring No. 2 when optimization in existing

regular public lower secondary education schools (INEBs) was no longer a

strategy supported under the Project. However, INEBs added a total of 189,226

new students by 2015, from their 2007 baseline (72,850 in 7th

grade, 62,270 in

8th

grade, and 54,106 in 9th

grade).

Indicator 23 : Rate of finalization of 9th

grade

Value

(quantitative

or qualitative)

74% 80% Not available

Date achieved 01/01/2005 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

This indicator referred, in fact, to the survival rate for LSE (from grade 7 in year

t through grade 9 in year [t+2]) and was supposed to be calculated for targeted

municipalities only and by type of school originally supported by the Project.

The indicator was dropped at Restructuring No. 2 when several types of LSE

schools were dropped from the Project. However, survival rates through grade 9

by 2015 in the municipalities targeted by the Project were the following, by type

of existing LSE institutions to be originally supported under the Project: i) TIs:

71%; ii) NUFEDs: 68%; iii) private: 83%; iv) cooperatives: 73%; and v) INEBs:

66%.

Indicator 24 : Standardized 9th

grade test scores of students in Integrated Lower Secondary

Value

(quantitative

or qualitative)

Not available At regional

average Not available

Date achieved 01/01/2007 12/31/2012 11/30/2015

xx

Comments

(incl. %

achievement)

Not assessed.

Same comments as those for IRI 18 above. Integrated LSE institutions never

materialized as such.

Indicator 25 : Percent of students from ethnic backgrounds, among students in grades 7-9

Value

(quantitative

or qualitative)

20.2% Targeted (2004)

18.7% National (2005)

At regional

population

percent

Not available

Date achieved

01/01/2004 for

Targeted

01/01/2005 for

National

12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

This indicator was supposed to be calculated only for an original set of 30

municipalities with the worst educational efficiency indicators (see World Bank

2007, pp.38-42) and for all original eligible schools. Indicator was dropped at

Restructuring No. 2 for the same reasons listed for IRI 23 above.

Indicator 26 : Percent of student places (in lower secondary education) located in municipalities

with low income quintiles (1 & 2)

Value

(quantitative

or qualitative)

Not available At population

levels Not available

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.


Indicator 27 : Number of Project-financed classes in Telesecondary Institutions that are

functioning with adequate equipment and instructional materials

Value

(quantitative

or qualitative)

0 --

P2: 4,583

classes

P3: 5,080

classes

5,080 classes

Date achieved 01/01/2007 12/31/2012 12/31/2012

11/30/2015 11/30/2015

Comments

(incl. %

achievement)

Achievement per period


Indicator was added at Restructuring No. 2. It was revised downwards (2,901) at

xxi

Restructuring No. 3, and revised upwards (5,080) at Restructuring No. 5,

consistent with progress to date back then.

Indicator 28 : Number of (Telesecondary Institutions) facilitators financed by the Project

Value

(quantitative

or Qualitative)

0 3,740

(total by 2012) 5,893

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)



Indicator was added at Restructuring No. 2 when the payment of TIs facilitators

(teachers) was added as an eligible expenditure category due to the fiscal crisis of

2009. The assumption was that the Project would fund an ever decreasing number

of TI facilitators until a full absorption by the Government of Guatemala (GoG) in

the payroll. The indicator was never monitored/reported in any Implementation

Status Report (ISR) since then, and was finally removed at Restructuring No. 3

without any explanation of the reasons. However, ever since the 2009 financial

crisis, fiscal constraints at the Government level made funding reallocations to

paying TI teachers (facilitators) a priority activity to support increased expansion

of LSE education. As a result, the Project ended up financing a total of 5,893 TIs

facilitators (710 in 2009; 1,720 in 2010; 1424 in 2011; and 2,039 in 2012) due to

the persistence of the need to fill these fiscal gaps. From 2013 onwards, all these

facilitators were absorbed into MINEDUC’s payroll.

Indicator 29 : Lower secondary schools operating under the modalities financed by the Project

that receive intercultural and instructional materials (Telesecondary Institutions)

Value

(quantitative

or qualitative)

0

100 percent

(1,500 with

NUFED)

P2: 967

P3: 967 967

Date achieved 01/01/2007 12/31/2012 12/31/2012

11/30/2015 11/30/2015

Comments

(incl. %

achievement)



Indicator’s unit of measure was changed from percentage to number at

Restructuring No. 2. Wording was revised for clarity at Restructuring No. 5.

Indicator 30 : Number of lower secondary schools operating under the modalities financed by

the Project that receive intercultural and instructional materials (NUFEDs)

Value 0 100 percent P2: 634 600

xxii

(quantitative

or qualitative)

(1,500 with TIs) P3: 600

Date achieved 01/01/2007 12/31/2012 12/31/2012

11/30/2015 11/30/2015

Comments

(incl. %

achievement)



Indicator was added at Restructuring No. 2. Wording was revised for clarity and

target was revised downwards in line with actuals and planned implementation

upon extension of the closing date at Restructuring No. 5.

Sub-Component 2.C – Education Demand Scholarships for Low Income Students


Original Target

Values (from

approval

documents)

Formally

Revised

Target

Values

Actual Value

Achieved at

Completion or

Target Years

Indicator 31 : Standardization of scholarships targeting and monitoring procedures

Value

(quantitative

or qualitative)


Date achieved 01/01/2007 12/31/2008 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

Indicator was supposed to support the standardization and strengthening of the

MINEDUC’s Scholarship Program, which contained a series of scholarship

programs back then and did not have a unified framework for its administration.

This was supposed to be finalized by the end of the first year of implementation

(2008). With the arrival of the Colom Administration in January 2008, the First

Lady’s Office absorbed most of the scholarships from different public institutions

(MINEDUC, Ministry of Labor, Ministry of Economy, etc.) under a national

program named “Social Cohesion Program”. That program was discontinued

come the Perez Molina Administration (2012) and all the Scholarship Programs

originally run by MINEDUC went back to MINEDUC. These comprise a total of

5 programs, one of which was the one supported by the Project. Monitoring

procedures for the latter were significantly strengthened with the issuance of

Ministerial Decree No.0041-2013 (January 2, 2013). This indicator was dropped

at Restructuring No. 2.

Indicator 32 : Region, income level, gender and ethnicity of scholarship beneficiaries in: (i)

old grant modalities; (ii) new grant modality

Value

(quantitative Not available

To be determined

at execution Not available

xxiii

or qualitative)

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

Indicator was supposed to measure the differences among a series of

characteristics of beneficiaries of scholarships under MINEDUC’s then-current

system and the new standardized system that would be the result of activities

leading to the completion of Indicator 31 above. Since standardization did not

take place as originally planned, this indicator was also dropped at

Restructuring No. 2.

Indicator 33 : Indicators of success of beneficiaries: on-time enrollment, promotion, learning,

and grade and full lower secondary education cycle completion

Value

(quantitative

or qualitative)

Not available To be determined Not available

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

This indicator was included under the same rationale as Indicator 32, and was

also dropped at Restructuring No. 2.

Indicator 34 : Number of scholarships financed by the Project that are granted to students from

targeted municipalities that comply with attendance requirements

Value

(quantitative

or Qualitative)

0 7,280 35,280 37,847

Date achieved 01/01/2007 12/31/2012 11/30/2015 11/30/2015

Comments

(incl. %

achievement)



This indicator was added at Restructuring No. 2. The focus was only those

scholarships funded under the Project. The target was revised upwards at

Restructuring No. 3, with the increase in scope of this activity. Restructuring No.

5 kept the same target of Restructuring No. 3, in line with expected progress and

planned implementation upon extension of the closing date.

xxiv

Component 3 – Education System Access in Support of Education Access and Quality7

Sub-Component 3.A – Consolidation and Strengthening of School-Based Management for

Education Access and Quality8


Original Target

Values (from

approval

documents)

Formally

Revised

Target

Values

Actual Value

Achieved at

Completion or

Target Years

Indicator 35 : Percent of school governments organized and trained (as percent of originally

planned targeted schools)

Value

(quantitative

or qualitative)


Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

This indicator was aimed at organizing and training 1,500 school governments

(school councils) from different modalities of lower secondary institutions (350

IBECs; 400 new Cooperative schools; and 250 INEBs). This indicator was

dropped at Restructuring No. 2 due to: i) the elimination of Cooperative schools

and INEBs as eligible lower secondary education institutions to be supported

under the Project; and ii) the elimination of the specific focus on IBECs.

Indicator 36 : Percent of schools with own planning and management of resources (as percent

of originally planned targeted schools)

Value

(quantitative

or qualitative)


Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

This indicator followed the same rationale for inclusion as Indicator 35 and was

therefore also dropped at Restructuring No. 2.

Indicator 37 : Repetition rates, lower secondary

7 The original name of this component at appraisal was “School Management in Support of Education Quality”. It was

changed at Restructuring No. 2 due to the change in scope of this component and for clarity.

8 The original name of this sub-component did not have the word “access”, which was added at Restructuring No. 2, for

consistency with the objectives of both the Project and this particular component.

xxv

Value

(quantitative

or qualitative)

Not available To be determined


Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

This indicator was included at appraisal and was meant to be one of the two

outcome indicators from an impact evaluation on the effectiveness of school

council training and organization of 1,500 lower secondary institutions in

targeted municipalities. In that impact evaluation, repetition rates would be

contrasted before and after the intervention. In that sense, they were expected to

be used for annual monitoring of outcomes after the intervention took place. At

Restructuring No. 2, the scope of this intervention changed and the impact

evaluation eliminated, resulting in the indicator to be dropped.

Indicator 38 : Total (inter-and intra-year) dropout rates in lower secondary

Value

(quantitative

or qualitative)

Not available To be determined


Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

This indicator followed the same rationale for inclusion as Indicator 37 and was

therefore also dropped at Restructuring No. 2.

Indicator 39 : Number of schools with trained school councils

Value

(quantitative

or qualitative)

0 100%

(~1,500)

P2: 1,500

P3: 1,350 1,489

Date achieved 01/01/2007 12/31/2012 11/30/2015 11/30/2015

Comments

(incl. %

achievement)



This indicator was added at Restructuring No. 2 originally as the proportion of

lower secondary education school councils that would be trained in targeted

municipalities. The original target was set at 100 % and meant every public

lower secondary institution located in targeted municipalities (INEBs, TIs, and

NUFEDs) would benefit. The indicator was revised at Restructuring No. 3 in a

two ways: i) its unit of measure changed to the number of school councils,

instead of the percentage of the total; and ii) a numerical target was set at 2,098

by 12/31/2013, in line with the revised scope of this component. The target was

revised further downwards (to 1,350) at Restructuring No. 4 and maintained at

xxvi

that level at Restructuring No. 5 to ensure attaining that goal under the revised

implementation plan approved with the 23-month extension.

Indicator 40 : School councils that receive grants and report expenditures satisfactorily

Value

(quantitative

or qualitative)

0 100% (~1,500) P2: 1,500

P3: 487 158

Date achieved 01/01/2007 12/31/2011 11/30/2015 11/30/2015

Comments

(incl. %

achievement)



This indicator was changed from a percent to number at Restructuring No. 2

originally as the proportion of lower secondary education school councils that

would receive grants and report expenditures satisfactorily, after being trained

with funding from the Project. The original target was set at 100 % and meant

every public lower secondary institution located in targeted municipalities

(INEBs, TIs, and NUFEDs) would receive grants and report expenditures. The

indicator was revised at Restructuring No. 3 in a two-fold way: i) its unit of

measure changed to the number of school councils, instead of the percentage of

the total; and ii) a numerical target was set at 487 by 12/31/2013, in line with the

revised scope of this component. The target was maintained at that level at

Restructuring No. 5 in line with expected progress and planned implementation

upon extension of the closing date.

Sub-Component 3.B – Strengthening of MINEDUC’s capacities to Plan, Manage, and

Supervise Pedagogical, Administrative, and Fiduciary Processes9


Original Target

Values (from

approval

documents)

Formally

Revised

Target

Values

Actual Value

Achieved at

Completion or

Target Years

Indicator 41 : Teaching Staff Payroll and Administrative Supervision

Value

(quantitative

or qualitative)

0% 100%

Not available

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

Not assessed.

9 The original name of this component at appraisal was “Consolidation and Strengthening of Departmental Supervision and

Technical Assistance to Schools”. It was changed at Restructuring No. 2 due to the change in scope of this component and for

clarity.

xxvii

achievement) This indicator was created to show progress towards the strengthening of

educational supervision by 10 Departmental Branches of the MINEDUC

catering to areas targeted by the Project. It was expected to measure the

difference in the proportion of decentralized management processes between

these 10 Departmental Branches (targeted departments) and all the remaining

Departmental Branches catering to non-targeted municipalities. It was dropped

at Restructuring No. 2 due to the change in scope of this sub-component.

Indicator 42 : Targeting, allocation and monitoring of scholarships

Value

(quantitative

or qualitative)

Not available 100%

Not available

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

This indicator followed the same logic of inclusion as Indicator 41 and was also

dropped at Restructuring No. 2.

Indicator 43 : School management training offered to principals, teachers and school

government

Value

(quantitative

or qualitative)

Not available 100%

Not available

Date achieved 01/01/2007 12/31/2012 11/30/2015

Comments

(incl. %

achievement)

Not assessed.

This indicator followed the same logic of inclusion as Indicator 41 and was also

dropped at Restructuring No. 2.

Indicator 44 : Annual Monitoring Reports updating progress toward PDO and IRI, based on

the educational information system

Value

(quantitative

or qualitative)

0 100%

3 (one per

year beginning

in 2011)

0

Date achieved 01/01/2007 12/31/2012 12/31/2013 11/30/2015

Comments

(incl. %

achievement)


P1: Not assessed; P2: 100% Achieved; P3: Not assessed

This indicator was originally added at Restructuring No. 2 under the name of

“Timely availability of information through the educational information system”

with an original percentage target of 100% without any explanation of what this

meant or was expected to measure. The indicator’s wording was revised and

changed to the current wording at Restructuring No. 3, making reference to an

xxviii

Annual Monitoring Report that would report progress towards the PDO-level

indicators and IRIs. It was finally dropped at Restructuring No. 5 for two

reasons: (i) because it was not an IRI itself; and (ii) because MINEDUC had to

prepare a semi-annual progress report, as mandated by the Legal Agreement,

and was therefore redundant to present it here. In fact, MINEDUC never

prepared a specific Annual Monitoring Report, as specified in Restructuring No.

3, but instead took this target as achieved through the submission of the semi-

annual Progress Reports.

G. Ratings of Project Performance in ISRs

No. Date ISR

Archived DO IP

Actual

Disbursements

(USD millions)

1 05/12/2007 Satisfactory Satisfactory 0.00


3 12/14/2007 Satisfactory Moderately Unsatisfactory 0.00




7 11/24/2008 Moderately Satisfactory Moderately Unsatisfactory 0.00

8 05/11/2009 Moderately Satisfactory Moderately Unsatisfactory 7.00

9 06/25/2009 Moderately Unsatisfactory Moderately Unsatisfactory 7.00




13 02/26/2011 Moderately Satisfactory Moderately Satisfactory 26.51






19 09/07/2014 Moderately Satisfactory Satisfactory 66.32


21 10/06/2015 Moderately Unsatisfactory Moderately Satisfactory 79.80

22 11/27/2015 Moderately Unsatisfactory Moderately Satisfactory 79.80

xxix

H. Restructuring (if any)

Restructuring

Date(s)

Board

Approved

PDO

Change

ISR Ratings at

Restructuring

Amount

Disbursed at

Restructuring

(in USD

millions)

Reason for Restructuring &

Key Changes Made DO IP

06/15/2009

(Restructuring

No. 1)

N MS MU 7.00

Reasons for Restructuring:

(i) To align the Project design

and activities with the new

strategies for the sector by

the then-incoming

Administration;

(ii) To help the GoG fill part of

the financing gap caused by

the 2009 fiscal crisis; and

(iii) To make the design more

flexible to implementation.

Key changes made:

(i) Eliminated Sub-Component

2.A, re-allocated funding to

Sub-Component 2.B;

(ii) Incorporated the financing

of TIs facilitators as an

eligible category of

expenditure;

(iii) Simplified the

disbursement table; and

(iv) Incorporated the definition

of “Targeted Areas” as

those referred to in the

Operational Manual.

01/29/2010

(Restructuring

No. 2)

N MU MU 7.00

Reasons for Restructuring10

:

Same as for Restructuring No.

1.

Key changes made:

(i) Increased scope for access-

related interventions (e.g.

payment of TIs facilitators)

10 The first two restructuring processes (Restructuring No. 1 and No. 2) were undertaken in two stages, but were part of a unified

restructuring process. Therefore, the reasons for restructuring are the same in both cases.

xxx

Restructuring

Date(s)

Board

Approved

PDO

Change

ISR Ratings at

Restructuring

Amount

Disbursed at

Restructuring

(in USD

millions)



and reduced scope for

quality-related

interventions (e.g. school

councils);

(ii) Significant changes to

Results Framework through

a revision of PDO-level

indicators and targets, and a

significant revision and

reduction of IRIs (from 30

to 11);

(iii)Further re-allocation of

funds across expenditure

categories;

(iv) Significant revision of

institutional arrangements;

(v) Revision of the Indigenous

Peoples Plan; and

(vi) Significant reassessment of

political and

implementation risks.

06/06/2011

(Restructuring

No. 3)

N MS MS 26.51

Reason for Restructuring:

To revise the scope and funds

allocation of the Project due to

fiscal shortfall exacerbated by

natural disasters in 2010.

Key changes made:

(i) Increased scope for access-


scholarships for low

income students) and

reduced scope for quality-


school councils);

(ii) Reallocation of the

Project’s proceeds in line

with changes in Project’s

scope; and

(iii) Changes to the Results

Framework, mostly through

xxxi

Restructuring

Date(s)

Board

Approved

PDO

Change

ISR Ratings at

Restructuring

Amount

Disbursed at

Restructuring

(in USD

millions)



revised wording and

targets.

04/12/2012

(Restructuring

No. 4)

N MS MS 36.30


To revise the scope, timetable

and funding allocation of the

Project due to:

(i) Changing priorities due to

the then-incoming

Administration (Pérez

Molina Administration);

(ii) Continued fiscal gaps and

Project delays; and

(iii) Further exogenous factors

affecting the Project (2011

2-month long teacher

strike).

Key changes made:

i) Increased scope for access-


payment of TI facilitators)

and reduced scope for

quality-related

interventions (e.g. school

councils);

ii) Elimination of

“Nationwide” sub-

indicators at the PDO-level;

iii) Revision to wording

and targets of some other

indicators (both PDO-level

and IRIs) due to revised

timetable and/or scope for

specific activities (e.g.

scholarships and training

of school councils)

xxxii

Restructuring

Date(s)

Board

Approved

PDO

Change

ISR Ratings at

Restructuring

Amount

Disbursed at

Restructuring

(in USD

millions)



11/21/2013

(Restructuring

No. 5)

MS MS 58.10


To revise the scope, timetable

and funding allocation of the

Project due to:

i) the need to extend the

closing date to ensure full

implementation of the

Project; and

ii) the GoG’s revised support

strategy for the PADEP/D

Program.

Key changes made:

i) Increased scope and

funding for Sub-

Component 1.B (teacher

training program)

ii) extension of project by 23

months; and

iii) significant changes to the

Results Framework (2

PDO-level Indicators

removed, 1 PDO-level

indicator.

xxxiii

I. Disbursement Profile

1

1. Project Context, Development Objectives and Design

1.1 Context at Appraisal

1. Country Context. At appraisal, Guatemala was just celebrating the 10-year anniversary

of its 1996 Peace Accords, which ended 36 years of civil war in this Central American country.

Since the signature of that agreement, ten years of uninterrupted democratic elections had taken

place. Óscar Berger was the President of Guatemala, and had been elected in a run-off election

on December 28, 2003 with 54.1 percent of the votes. However, the political panorama posed a

challenge for President Berger, with only about 30 percent of the total Congress belonging to the

party that had brought him to power and impending national elections to be held in early

September of 2007.

2. Guatemala’s population at the time was about 13 million people, and stood out for being

significantly younger, more rural and ethnically more diverse than any of its Central American

peers. As of 2006, over half of the Guatemalan population was under 18, over half of the

population lived in rural areas, and over a third of the population was indigenous11

. The profile

of poverty and inequality in the country was alarming: more than half of the population was

poor, and more than 15 percent was considered extremely poor; inequality levels, as measured by

the Gini Coefficient, located Guatemala in the top 25 percent for the most unequal countries in

the world12

. Poverty and extreme poverty were concentrated disproportionately in rural areas,

and among indigenous households. More than 7 in 10 poor and more than 8 in 10 extremely poor

lived in rural Guatemala. Poverty rates among indigenous populations were in the order of 75

percent, two times more prevalent than for non-indigenous populations; extreme poverty rates

among indigenous people were in the order of 28 percent, 3.5 times bigger than in non-

indigenous settings13

.

3. Macroeconomic Context. At appraisal, Guatemala was the largest economy of Central

America and was coming off a period of relative macroeconomic stability with an upward trend

in the Gross Domestic Product (GDP) growth, but with relatively high annual levels of

inflation14

. Per capita GDP, which stood at about US$2,241 by 2006, had only increased 2

percent in real terms since 2000. Although the rate of absolute poverty had decreased by 9

percentage points between 2000 and 2006, the majority of people being lifted out of poverty was

in rural areas and was mostly non-indigenous15

. Extreme poverty levels, on the other hand, had

11 Guatemala comprises 25 linguistic groups classified under 4 ethnic groups: Ladino, Maya (21 groups), Garífuna, and Xinca,

each with its cultural and linguistic background and identity. See World Bank (2007), pp.137-8.

12 Guatemala’s Gini Coefficient for 2006 stood at 44.8. More than 50 percent of the income of the country was concentrated only

on the richest 20 percent of the population. See World Bank (2009a), pp.16-27.

13 Ibid.

14 While the average annual GDP growth for the period 2001-2005 had been 3 percent, GDP growth from 2005 through 2006 was

5.4 percent and GDP growth from 2006 through 2007 was 6.3 percent. Inflation averaged 7.3 percent per year during the 2001-

2007 period.

15 The sole exception was the Mam indigenous group, which saw a 15 percent reduction in poverty levels since 2000. See World

Bank (2009a), p.13, Table 2.3.

2

seen no change. Inequality, which was more prevalent in urban areas than in rural areas, had

decreased only in urban areas16

.

4. Education Sector Context. At appraisal, Guatemala boasted substantial progress towards

universal primary education, with a Gross Enrollment Rate (GER) of 113 percent and a Net

Enrollment Rate (NER) of 93.5 percent. The number of out-of-school children of primary

education school age had nearly halved in four years. However, the efficiency of the primary

education system was still low, due to the high levels of overage, repetition, and school dropout.

In other words, while the country had succeeded in providing universal access to primary

education, it still had to focus on guaranteeing primary education completion (grade 6). This

disproportionately affected two sub-groups of students: those living in rural areas and those

coming from indigenous settings. Rural students were 30 percent less likely to complete primary

school than urban students17

. Every year, one out of eight primary students failed their grade, and

these numbers were higher for indigenous students and rural inhabitants. According to one

estimate, 14.1 percent of primary education spending was dedicated to allowing students to

retake grades18

.

5. The low efficiency levels of primary education impacted directly the transition into the lower

secondary education (LSE) level (grades 7 through 9). Nationally, only about 40 percent of

students began LSE, and less than three-quarters of those reached the final year. Differences

between indigenous and non-indigenous student populations in LSE access were stark: whereas

non-indigenous student populations showed a 74 percent GER for LSE, indigenous student

populations presented a 26 percent rate. Public secondary education provision was limited, and

private institutions served over 70 percent of secondary students. Although girls attended LSE at

similar rates as boys, geographic access was a clear issue: urban students were 40 percent more

likely to enter secondary school than rural students. Approximately 30 percent of children were

expected to complete lower secondary, and only 23 percent would enroll in upper secondary.

6. Overall, the primary bottlenecks in the education system at appraisal were efficiency and

quality issues at the primary education level and access issues at the LSE level. These challenges

disproportionately affected indigenous and rural populations, which tended to be poorer.

Although access to upper secondary and tertiary was highly restricted, this was not the most

pressing issues until more students graduated from the LSE level.

7. Government strategy. Since the signing of the Peace Accords in 1996, there had been

significant public demand for pro-poor human development. Rebuilding of government services,

particularly in rural areas, became a national priority. As of appraisal, the Government of

Guatemala (GoG)’s stated long-term goals for the 2004-2015 period were: (i) universal pre-

primary, primary, and LSE; (ii) strengthening student standards to improve education quality;

16 Whereas overall poverty rates fell from 56.2 percent in 2000 to 51.0 percent in 2006, extreme poverty rates fell from 15.7

percent to 15.2 percent in the same period. However, the 0.5 percentage-point difference in the case of the latter was not

statistically significantly different from zero. See World Bank (2009a), p.ix, Table 2.

17 See USAID (2007).

18 Ibid.

3

(iii) institutionalization of a permanent teacher professional development program; and (iv)

improved citizen participation in education decision-making19

.

8. Bank Rationale for Supporting the Project. The 2005-2008 World Bank’s Country

Assistance Strategy (CAS) for Guatemala (World Bank, 2005) gave a high priority to human

development and equitable economic growth, justifying greater investment in education. The

Government’s strategic shift from a focus on primary education access to primary education

efficiency and quality and secondary access was reflected in the closing of the Universalization

of Basic Education Project in 2008 and the start of this Project (World Bank, 2009b).

1.2 Original Project Development Objectives and Key Indicators (as approved)

9. As written in the Project Appraisal Document (PAD), the Project Development Objective

(PDO) was “to improve access to a quality lower secondary education for low income students,

especially in indigenous communities, through improved primary education completion rates for

overage students, strengthened flexible lower secondary education modalities and school

management”. The wording in the Loan Agreement only added, to that in the PAD, the

preposition “in” for clarity.

10. The Project originally included five PDO-level indicators:

Indicator 1: Ninth grade gross completion rate

Indicator 2: Gross enrollment rate in lower secondary (grades 7-9) nationwide and in

priority municipalities

Indicator 3: Intake in grade 7 in modalities supported by the Project

Indicator 4: Proportion of overage pupils (13-15 years old) who complete primary

education

Indicator 5: Sixth grade gross completion rate

11. Through the five restructurings, the PDO never changed. Restructurings, however, did make

adjustments to the PDO-level indicators, as follows:

Restructuring No.2 (World Bank, 2009c) made four changes:

i) added nationwide and “in targeted municipalities” sub-indicators for indicators #1

and #5;

ii) clarified the specific modalities to be supported under the Project, creating one sub-

indicator for Telesecondary Institutions (TIs) and Educational Family Nucleus for

Development (NUFEDs) for indicator #3;

iii) clarified that indicator #4 also included the accelerated primary education modality;

and

iv) thoroughly revised baseline and target (annual and end-of-Project) values for all 5

indicators.

Restructuring No. 3 (World Bank, 2011) made two changes:

i) added a final target for the 2013 year, which had been inadvertently omitted before;

and

19 See MINEDUC (2004).

4

ii) updated some end-of-Project targets.

Restructuring No. 4 (World Bank, 2012b) made two changes:

i) eliminated all nationwide sub-indicators to reinforce the fact that the PDO was

geared towards a targeted population, and that this targeted population had been

identified, throughout the Project, through the “targeted municipalities”; and

ii) updated some end-of-Project targets.

Restructuring No. 5 (World Bank, 2013) made three changes:

i) eliminated indicator #3 due to being more an output than an outcome and for not

being a targeted indicator;

ii) downgraded indicators #4 and #5 to Intermediate Results Indicators (IRIs); and

iii) updated end-of-Project targets to the new closing date (November 30, 2015).

1.4 Main Beneficiaries

12. As specified in the PDO, the primary beneficiaries were low-income students, especially

those in indigenous communities. Interventions targeted low-income and particularly rural areas,

which is where the majority of the indigenous population of Guatemala lives. While flexible

lower secondary education modalities and teacher training were selected where they operated

nationally; scholarships, school councils, and accelerated primary education interventions took

place in priority municipalities systematically selected at the beginning of the project. These

municipalities were selected based on a composite index representing their lag in education

enrollment, their higher-than-average rates of poverty, and the proportion of both indigenous

inhabitants and rural population20

.

1.5 Original Components

13. The Project had three components, each with multiple sub-components:

Component 1: Primary Education Completion and Quality (USD 7.7 million). This

component sought to support an integrated strategy for Basic Education (unifying the

primary and lower secondary cycles in a same school known as “Integrated Basic

Education Center” or IBEC) to increase primary education retention and completion of

overage students, and promote their continued education at the lower secondary

education level. To contribute to long-term quality improvements in primary education,

the component also supported proposed reforms to the Pre- Service Teacher Training

system, by strengthening regional pre-service teacher training centers. This component

includes subcomponents: 1A -Integrated basic education centers, and 1B - Teacher

training development.

Component 2: Lower Secondary Expansion and Quality (USD 51.3 million). This

component focused on integrated strategies to increase the quality and access to lower

secondary education (grades 7-9). It included support for (i) the curricular reform of

20 Esquivel (2006) fully describes the method used to calculate the education efficiency index. Annex 11 presents a more in-depth

analysis of the beneficiaries of the Project.

5

secondary education; (ii) strengthening and expansion of existing lower secondary

education modalities; and (iii) strengthening education demand programs, such as

scholarships and subsidies for enrollment, retention and graduation of low income

students. This component includes subcomponents: 2A- Feasibility studies and strategies

for secondary education curricula and evaluation systems, to guide medium and longer-

term reforms; 2B - Strengthening and expansion of pertinent and flexible modalities for

lower secondary education, and 2C - Education demand scholarships for low income

students.

Component 3: School Management in Support of Education Quality (USD 21.0

million). This component sought to increase the contributions of school-based

management to the quality of learning and also strengthen the supportive and

supervisory role of Education Departmental Offices, improving their decentralized

functions. This component includes subcomponents: 3A - Consolidation and

strengthening of school-based management for education quality, and 3B - Consolidation

and strengthening of departmental supervision and technical assistance to schools.

1.6 Revised Components

14. The Project’s components were significantly revised in Restructuring No. 2, upon the

request by the then-incoming Colom Administration. Although components stayed mostly

aligned with the original design, changes were made to the name, scope, size and composition of

the components.

Component 1 (Original cost remained the same: USD 7.2 million) underwent two

changes:

i) Sub-Component 1A was renamed to “Support of quality primary education” because

of the change in scope. Previously, it was intended to support the implementation of

the accelerated primary education program only in IBECs. With the change in scope,

this component started supporting the aforementioned program but in regular primary

schools only.

ii) Sub-Component 1B was renamed to “Support teacher training” also because the

change in focus from providing revised curricula and training materials for pre-

service training centers to: (i) supporting the drafting of a proposal for the reform of

the pre-service teacher training; and (ii) improving the quality of training and

performance of in-service teaching staff, including the intercultural and bilingual

modalities.

Component 2 was renamed to “Access and Quality of the Lower Secondary

Education" (revised cost: USD 54.5 million) and underwent two changes:

i) Sub-Component 2A (the feasibility studies and impact evaluations) was eliminated

from the Project21

.

21 In reality, Sub-Component 2A had been eliminated in Restructuring No. 1. However, it is important to remember that

Restructurings No. 1 and No. 2 were two phases of one restructuring process that were completed within a period of 7.5 months.

Restructuring No. 1 undertook the most urgent changes and Restructuring No. 2 took care of a more comprehensive approach.

6

ii) Sub-Component 2B changed focus in two significant ways. It changed its focus to

support two modalities: Telesecundarias to significantly expand access in remote and

rural areas, and NUFED to continue a program targeting high-risk students. It was

hanged to not support any of the other modalities originally mentioned - traditional

LSEs, non-governmental organizations (NGOs)-run, Foundation-supported, or

Cooperative. Increasing its focus on only two types, it supported the payment of

Telesecundaria facilitators (teachers) as a core eligible expenditure supporting the

expansion of the Telesecundaria flexible modality, due to the temporary inability of

the GoG to absorb into the payroll the significant number of new teachers envisaged

in that strategy.

Component 3 was renamed to “Education System Management in Support of

Education Access and Quality” (revised cost: USD 17.8 million22

) and also underwent

changes in both sub-components:

i) Sub-Component 3A was maintained in essence, but a more comprehensive approach

was taken to target all lower secondary education school councils, instead of the

originally targeted 1,500 schools.

ii) Sub-Component 3B changed both its scope and its title. The new name for this Sub-

Component was “Strengthening of the Ministry of Education (MINEDUC)’s

Capacities to Plan, Manage and Supervise Pedagogical, Administrative and Fiduciary

Processes” and the new scope added support for institutional strengthening of the

central level, as well.

1.7 Other significant changes

15. The project underwent five restructurings. In addition to changes in the Results

Framework, as discussed above (see Datasheet, Section F), these altered the distribution of funds

between components, allowed funds to be used for relevant teacher salaries, and made several

iterations of changes to the results and implementation indicators (see section H of the

datasheet).These changes were within the parameter of the original PDO, but significantly shifted

the emphasis of the PDO towards increasing access to LSE for low-income students. Against this

backdrop, the “quality” goal within the “access to a quality LSE” PDO part was meant to be

attained through improving the original quality of the two institutions (TIs and NUFEDs) that

were offering the two flexible modalities of choice (Telesecundaria and Alternancia) to be

supported upon the first significant revision to the Project under the then-incoming Colom

Administration (2008-2011).

16. Restructurings No. 1 (2009) and No. 2 (2010) were a 2-stage adaptive restructuring

process conducted before disbursement of any funds besides the initial payment. A new

Administration, elected between Board approval and effectiveness, did not change the

Government’s long-term education strategy23

, but prioritized different means to achieve them in

light of the significant fiscal shortfall affecting the crisis since 2008. Of utmost importance then,

22 A total of USD 0.3 million was left unallocated to deal with contingencies.

23 For an in-depth analysis of the priorities of the Colom Administration, see MINEDUC (2008).

7

this 2-stage restructuring process: (i) allowed funds to be used to support teacher salaries24

– a

recurrent cost originally supported to be funded by the GoG – and (ii) recognized the

Guatemalan Congress’ elimination of funding for the impact evaluations (in subcomponent 2a).

17. Restructurings No. 3 (2011) and No. 4 (2012) continued providing flexibility for the

Project to adapt in light of the global financial crisis. The government continued having

significant fiscal shortfalls, and exogenous factors further aggravated the situation (Hurricane

Agatha, 2-month long teacher strike). Project funding shifted further to towards the immediate

needs of both teacher salaries and scholarships for low-income students, and away from school-

based management, while raising targets associated with activities that were allocated with

greater funding and lowering those with less funding.

18. Restructuring No. 5 (2013) extended the project by 23 months, significantly revising the

RFM table in line with progress to date and planned activities under the extended closing date.

2. Key Factors Affecting Implementation and Outcomes

2.1 Project Preparation, Design and Quality at Entry

19. Several factors affected Project preparation, design, and ultimately quality at entry.

Three main factors could be cited as positive for the operation’s quality at entry; three other main

factors could be considered as shortcomings.

20. The main factors that impacted positively on the quality at entry were:

It benefited significantly from a thorough alignment of the Project’s design with core

strategic documents both from the client’s and the Bank’s side. The Project started being

prepared at the beginning of the 2004 Administration of then-President Oscar Berger.

The education foundations of the Project were built upon: (i) the Long-Term Education

Goals stemming from the Peace Agreement of 1996; (ii) the education development

objectives of the Extreme Poverty Reduction Strategy for Guatemala (EPRSG) 2004-

2007 known as “Guate Solidaria Rural”25

; and (iii) the Bank’s CAS 2005-2008.

It found traction in the number of lessons learned, both from other countries and from the

then-ongoing Bank project in the education sector in Guatemala. Most of the

interventions foreseen by the Project were adoptions of interventions that had proven

effective in other countries or adaptations from previous experiences in former Bank

projects. 26

24 The newly added category of teacher salaries ended up absorbing USD29.6 million, or 37 percent of the loan’s proceeds.

25 See World Bank (2007), p.9, Table 4.

26 For example, Accelerated education programs were part of integrated basic education schools - those providing grades 1-9 in

the same educational facility - in El Salvador, Colombia and Brazil. Integrated basic education schools had shown to impact

positively primary education completion rates and transition between primary education (grades 1-6) and lower secondary

education (grades 7-9). Guatemala received technical assistance from El Salvador and Colombia and adapted El Salvador’s

8

It was innovative in certain ways. The Project was the first nationwide LSE project to be

carried out by the Government. As such, for instance, it was the first project to create,

train, give legal status and make small financial transfers to school councils at the lower

secondary education level27

. In addition, the Project had, arguably, the most

comprehensive Indigenous Peoples Plan (IPP) ever set up for an Education Project

before28

.

21. The main factors that impacted negatively the quality at entry were:

A rather lengthy preparation time. The Project took a total of 2 years of preparation

between its identification mission in February 2005 and its approval in March 2007. By

the time the Project got approved, the Berger Administration was only 10 months away

from ending and 6 months away from the general elections. This meant that the Project

had to almost start anew when the Administration of Álvaro Colom took over in 2008.

An underestimation of political and implementation risks. The Project’s original overall

risk rating was deemed Moderate. However, a re-assessment of political and

implementation risks at Restructuring No. 2 revised the overall risk for the Project at

Substantial29

.

An unnecessarily complex original Results Framework (both for outputs and outcomes),

built massively around a series of built-in impact evaluations. This design did not weigh

in the implications a potential elimination of impact evaluations, envisaged as the core

methodology for measuring results, would eventually have30

.

accelerated learning education materials. Furthermore, flexible modalities for lower secondary education were taken from

Mexico (Telesecundaria) and Belgium and France (NUFED).

27 School councils had been proven and tested for a decade in Guatemala with the community-based management program known

as PRONADE, and was extended into the lower secondary education level. However, although the PRONADE program

disappeared, the community-based management program did continue over time. See Ganimián (2016).

28 This plan was in line with the 1996 Peace Accords and also took into consideration the findings and recommendations from the

Integrated Social Assessment carried out as part of the project preparation in 2006 and other consultations carried out at the

central, departmental and municipal levels since 2000. Some recommendations of the social assessments for the project design

were to: (i) include variables like “ethnicity” and “mother tongue” in information systems; (ii) disseminate clear guidelines for

the accelerated program (ages, flexible schedules, facilities needed, duration and expected participation), with careful selection

and elaboration of culturally pertinent materials; (iii) have teacher training be paced- initial training plus follow-up in service

training to deal with local problems; and (iv) strengthen and adequately equip existing TIs before opening new ones.

29 For example, the two areas related to the implementation of the school grants to lower secondary education-level school

councils were given a Low risk, on the basis of the successful experience up to then with primary education-level school

councils. However, this intervention would eventually not be successful, as measured by the degree of attainment of its

intermediate outcome indicators. For instance, factors like the duration of the primary education cycle (6 years) versus the

duration of the lower secondary education cycle (3 years) seem not to have been adequately weighed in at design, impacting

directly in the incentives for parents to stay in the school council beyond the lifetime of their children’s school cycle, and thereby

impinging in the legal status of authorities of the school council beyond some time.

30 The Results Framework design was complex--included too many indicators--and lacked adequate baselines. The project

tracked 6 PDO-level key indicators, plus several sub-indicators, and 31 IRIs, which also included sub-indicators. In addition, the

PDO-level indicators either did not have a consistent baseline (some used the 2004 or 2005 school year) or did not present a

9

Ambitious goals to deal with the competing goals of improving access and quality of

lower secondary education. Improving education access often hurts education quality

outcomes, as it puts strain on education provision; improving quality often leads to

concentrating resources on those already enrolled. While approaching both access and

quality simultaneously may be more efficient than fluctuating between two goals by

sacrificing neither, the PAD set ambitious targets to improve access while placing an

equal amount of focus on quality interventions. Either more conservative goals for both

should have been set, or the PDO should have focused more on one or the other.

2.2 Implementation

22. As in the case of Quality at Entry, several factors affected implementation, as well. Three

main factors contributed to successful implementation; five other main factors gave rise to

problems.

23. The main factors that helped implementation positively were:

The Bank teams supported the client’s changing priorities. The Project was restructured

five times over almost 7.5 years of implementation31

. Although the number of

restructurings was high, it did reflect the Bank team’s flexibility to accommodate to both

the country fiscal situation due to a number of economic and natural shocks and new

priorities emerging from upcoming Administrations

The Government supported indigenous peoples’ issues, in general, and the IPP, in

particular, especially in the last 4 years of implementation. This is explained in more

detail in Section 2.4 below and in Annex 12.

Fiduciary and safeguards performance and reliability were high. Safeguards,

procurement, and financial management were rated either Satisfactory or Moderately

Satisfactory throughout implementation and all three closed with Satisfactory ratings at

Project closure. Satisfactory safeguards and fiduciary efforts helped the Bank and

Government to focus on core implementation issues during the lifetime of the Project.

24. The main factors that adversely affected project implementation were:

Project effectiveness was severely delayed. The loan was approved by the Bank’s Board

on March 6, 2007. The loan did not get Congressional approval during that calendar year,

mostly due to the then-impending national elections32

. Anecdotal evidence suggests that

baseline at all. The baseline for the IRIs was not available in most cases. This posed a problem in setting attainable targets for the

Project.

31 The loan agreement was declared effective on June 23, 2008, and the loan closed on November 30, 2015.

32 The first round of national elections took place on September 9, 2007. Since no candidate obtained more than 50 percent of the

votes, a run-off election between the top two candidates was held on November 4, 2007.

10

the Congress discounted a change in the governing political party, which would

eventually materialize. After the Colom Administration took over on January 14, 2008,

the Project was approved rather quickly by the new Congress (March 8, 2008) and the

Loan Agreement was signed thereafter (April 12, 2008). The loan agreement became

effective on June 23, 2008, 15.5 months after Bank approval.

There was frequent turnover of decision-makers throughout the lifetime of the Project.

Since Board Approval, Guatemala changed Presidents three times and Ministers of

Education six times. Between identification and closure, the Bank had six Task Team

Leaders (TTLs) on this Project. These changes in Administration and leadership by the

Borrower and the Bank contributed to implementation delays, coordination and

communication challenges, and the repeated need to build relationships between parties.

The chart below shows the leadership changes.

Exogenous and unforeseen events, like the 2008/2009 financial crisis33

, the 2010 Agatha

Hurricane34

, and the 2011 teacher strike35

, slowed implementation progress.

There was a lag between increasing access and adequately supplying schools. The

Project supported the opening of three types of classes: Accelerated Primary Education

for Overage Students (PEPS), Telesecundaria, and NUFEDs. In all cases, there were

significant implementation delays between the effective opening of the class and the

provision of indispensable material for a proper functioning of the intervention, largely

due to delayed preparation of new education materials36

.

The late timing of the mid-term review, which ended up being a missed opportunity. The

mid-term review took place in December 2011, during the transition period between the

end of the Colom Administration and the start of the Perez Molina Administration. No

significant changes were suggested as a result of that review. However, upon the new

Government staff was in place, several changes started being discussed, eventually

33 A global financial crisis affected Guatemala from 2008 onwards, creating a significant fiscal gap and an immediate impact on

some interventions funded by the Project. The immediate fiscal shortfall threatened secondary expansion, forcing the Project to

be restructured to make TIs’ teacher salaries an eligible expenditure, as explained above. 34 In end-May 2010, tropical cyclone Agatha landed in Guatemala, killing more than 150 people and bringing widespread

destruction and devastation to the whole country. The Tropical Storm Agatha blew a 200-feet hole in Guatemala City and was,

arguably, the worst climatic disaster to ever hit Guatemala. Baez et al (2014) document that the Agatha Storm effects on the

economy were disastrous and cost Guatemala 2.2 percent of the GDP. The significant effects on households’ well-being alone

were huge: per capita income decreased approximately 10 percent in those areas most hardly hit by the Storm. One of the coping

strategies most of these households used was to get their children out of school to help in the “economic reconstruction” of their

crops. Estimates of the decrease in school enrollment associated with this climatic event go as high as to 3.3 percent.

35 This was a two-month long teacher strike that took place in 2011. During the strike period, a significant number of teachers

occupied MINEDUC’s offices and prevented MINEDUC staff from accessing their offices.

36 For a significant period of time during the lifetime of the Project, PEPS classes operated in hallways and without textbooks. A

similar situation was faced with the rapid (and massive) expansion of Telesecundaria classes, with delays in providing complete

materials and needed equipment. Finally, NUFED classes were also hurt by delays, although to a much lesser extent than

Telesecundarias. On average, delays between the opening of classes and the provision of materials were in the order of two to

three years. Although many classrooms had some materials, they were usually outdated, insufficient, or designed for other

purposes.

11

leading to Restructuring No. 437

. This could have probably been avoided if the mid-term

review had been held once the new Administration kicked off.

2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization

M&E design

25. The Project’s design included an impact evaluation to identify successful and

unsuccessful interventions originally intended to be supported by the Project. In many ways, the

Project was designed as a series of many novel interventions, with impact evaluations to

determine the overall effectiveness of original interventions.. As mentioned above, the original

Results Framework was overly complex and overestimated the initial M&E capacity of the

Government as many of these indicators were ones that the Government had never tracked

before. Many of these indicators were assumed to be collected through the impact evaluation

work that was cancelled in the first 2-stage restructuring process (Restructurings No. 1 and No.

2). Impact evaluations, however, could have been added considerable value enhancing the

analysis on attribution.

26. The lack of baseline data at design meant initial targets were unrealistic. While both the

Government and Bank teams established specific targets for most outcome and output indicators,

the PAD did not include baseline measurements. This was partly due to the quality and reliability

of data available from MINEDUC at the time of preparation. As the information technology unit

in MINEDUC was strengthened and the Education Management Information System (EMIS)

upgraded during project implementation, an effort was made to better define a baseline.

M&E implementation and utilization

27. The General Directorate for Education Assessment and Research (DIGEDUCA),

MINEDUC’s research arm, commissioned studies on different interventions that were supported

by the Project, as follows:

PEPS studies:

Overage students and the PEPS program (MINEDUC 2012a, 2015h, and

forthcoming.c)

Flexible modalities:

Baseline study for the Project (MINEDUC 2010b)

Standardized national results of 9th

-grade student achievement for years 2006

(MINEDUC 2007), 2009 (MINEDUC 2014a, 2014c) and 2013 (MINEDUC 2014b,

2015f, 2016c)

Student transition from LSE to upper secondary education (MINEDUC 2015c)

Relationship between learning opportunities and learning outcomes in LSE

(MINEDUC 2013a)

37 For an in-depth description of the strategic priorities of the Perez Molina Administration, see MINEDUC (2012b, 2012d).

12

High-performing LSE schools in disadvantaged settings (MINEDUC 2015b, 2016a)

Qualitative evaluation of LSE modalities (MINEDUC 2015d, 2016b, forthcoming.b)

Others:

Assessment of the Academic Professional Development Program for [In-Service]

Teachers (PADEP/D)’s cohorts of graduates, teachers’ self-reported satisfaction and

changes in teaching methods after PADEP/D training (MINEDUC 2012c, 2014d,

2015i, and forthcoming.d)

LSE Teacher Credentials’ Standardization (MINEDUC 2010a)

Prospective Teachers’ Assessment (MINEDUC 2015e)

Use of the library books that were granted as part of the National Reading Program,

funded through the Project for TIs and NUFEDs (MINEDUC 2016d).

28. These reports, most of which were funded from the Project, have allowed the Ministry of

Education to attempt to inform on the efficacy of its programs and represent a significant

increase in the production (both quantity and quality) of the analytical pieces produced in house.

29. In addition, the Ministry of Education’s Strategic Planning Department (DIPLAN) has

been strengthened with funding from the Project. This Department now has an excellent EMIS

that made the cost-benefit analysis undertaken in Annex 3 possible, and has great potential for

further research.

2.4 Safeguards and Fiduciary Compliance

Safeguards

30. The Indigenous Peoples was the only Safeguards Policy triggered under the Project. As a

result, a comprehensive IPP was prepared. The Environmental Assessment Safeguards Policy

was not triggered since school construction was not going to be financed by the Project.

31. Overall, the IPP’s implementation is rated Satisfactory. Not only did the Project help to

achieve substantial impact for indigenous peoples’ issues through the contextualization of

interventions beyond specific targeting, but thanks to the efforts spearheaded through the

Project’s IPP, the broader agenda of multicultural and bilingual education featured so high in the

last four years of implementation that many of the original goals set at the IPP were surpassed.

Annex 12 undertakes a comprehensive assessment of the IPP’s achievements.

Fiduciary Compliance

32. Overall, the degree of fiduciary compliance was Satisfactory. Although procurement

processes were occasionally delayed, at no time was the quality of processes considered

inadequate in any of the Independent or Post-Procurement Reviews. Financial Management was

also fairly good. One external audit (2010) got a qualified (not clean) opinion. However, the

client took rapid corrective measures and implemented stronger internal controls that prevented

any recurrence. Bank fiduciary specialists noted the professionalism and improvement in

fiduciary matters throughout the lifetime of the Project.

13

2.5 Post-completion Operation/Next Phase

33. At present, no follow-on operation is under preparation. A new Administration (of President

Jimmy Morales) took over in January 2016 and is navigating its way through the challenges left

in the aftermath of the end-2015 political crisis. The new Administration has three main pillars in

its Government Plan: 1) Health; 2) Education; and 3) Social Development. The Bank team has

had several meetings with the President’s team to discuss, among other things, the upcoming

Systematic Country Diagnostic (World Bank, forthcoming.a), where interventions in education

emerge as a priority. MINEDUC’s authorities have expressed great interest in working with the

Bank, in general, and having a new project in this sector, in particular. A new World Bank Group

Country Partnership Framework is currently under preparation.

3. Assessment of Outcomes

34. Although the PDO was never changed, the results framework was changed five times during

the lifetime of the Project. As explained in detail in Sections 1.6 and 1.7 above, and Sections F

and H of the Datasheet, Restructurings No. 2 and No.5 represented the most significant changes

to the Project. In Restructuring No. 2, the results framework was updated to reflect the numerous

changes determined by the Colom Administration. The Restructuring No. 5 framework

simplified the overall results framework and updated targets to reflect implementation progress

and align with the closing date extension. The Assessment of Outcomes is split into three

periods: from effectiveness to Restructuring No. 2 (June 2008 to January 2010), Restructuring

No. 2 to No. 5 (January 2010 to November 2013), and Restructuring No. 5 to project closure

(November 2013 to November 2015).

3.1 Relevance of Objectives, Design and Implementation

35. Overall Relevance is rated Modest for periods 1 and 2 and Substantial for period 3. The original PDO was highly relevant throughout all periods of the Project. However, the

original results framework was dependent on data capacity that did not exist at effectiveness and

had a high risk of not being built by the incoming Administration. The results framework was

improved in period 2 but did not have clear definitions of indicators to measure until the start of

period 3. This is one of the main reasons that the design rating improved in period 3. Regarding

implementation, the priorities of the Administration that helped design the Project differed from

those of the Administrations that managed the Project from effectiveness to closure and the

economic situation changed soon after approval. As a result, immediately after effectiveness, the

interventions of the Project were altered to focus more on access than on providing quality

education. The Project adapted to the Government’s new priorities, but did not revise the PDO of

the Project to be consistent with the new focus. This led to a Modest rating in all periods for

Implementation.

14

Table 1. Relevance Ratings

Period 1 Period 2 Period 3

Rel

evan

ce

Objective Substantial Substantial Substantial

Design Modest Modest Substantial

Implementation Modest Modest Modest

Overall Modest Modest Substantial

36. The relevance of Project objectives is rated Substantial for all three periods. The

PDO, which never changed, is consistent with current development priorities for the country and

also for the Bank’s ongoing Country Partnership Strategy (World Bank 2012a).

37. Access to a quality lower secondary education remains a priority for the country38

.

Although a greater supply of LSE schools in the last 8 years played a significant role in the 39

percent increase in enrollment in lower secondary education (from 582,000 in 2007 to about

810,000 in 2015), the GER for LSE is still below 70 percent. It is also still relevant to target

indigenous people, who accounted for 25.2 percent of enrolled students in LSE in 2014 and over

40 percent of the population39

. Regarding primary education completion rates, although the gross

completion rate has significantly improved nationwide (to around 90 percent) and the proportion

of overage population in primary education has decreased (from 50 percent in 2007 to 38 percent

in 2014), there is still 15 percent of the total primary enrollment with 2+ years of overage, with a

marked urban-rural divide (9 percent urban, 18 percent rural).

38. The Bank’s current assistance strategy is aligned with increasing access to lower

secondary education. The Country Partnership Strategy for the period 2013-2016 (latest at the

time of the Project’s closing date) responds to the Government request to assist with developing

the institutional framework needed to furnish adequate education and skills to meet the

development challenges ahead.

39. The relevance of the Project’s design is rated Modest in periods 1 and 2 and

Substantial in period 3. In all three periods, there is a clear conceptual link between the PDO

and the goal of the three components. However, the links between the PDO, the proposed

interventions, and the results framework were at times tenuous.

40. In period 1, the listed interventions aligned with the PDO, but were disconnected from

the then-incoming Government’s priorities and implementation. Project design took place at the

38

The MINEDUC’s preliminary strategic plan for 2016-2020 includes five axes: increased access to children and

teenagers; quality and inclusion (linguistically and culturally pertinent); different modalities; adequate learning

facilities; effective and decentralized institutional management. The preliminary plan shows that educational

coverage was lower in 2015 for lower secondary education (71.6 percent) and for higher secondary education (37.7

percent), than for primary education (96 percent). It also compared the proportion of students who passed

standardized test scores for reading and math for different levels, showing a passing grade between 40-45 percent

for 6th grade; 15-18 percent for 9th grade and between 9-26 percent for graduating students. These figures reflect

the need to continue working towards greater access to a quality education.

39 In 2007, 120,000 students enrolled in LSE were indigenous students (20.5 percent of the total population). In

2014, the percentage of enrolled indigenous students increased to 25.2 percent (194,000 out of 769,000).

15

end of one Government Administration and included interventions such as per-student financing

and school-based management, whose popularity was declining and unlikely to be adopted by an

incoming Administration40

. The results framework included numerous indicators for which data

had never been collected, and had a high risk of never being collected, and lacked meaningful

indicators of quality.

41. In period 2, the results framework and funding allocations more closely aligned with

Government priorities and interventions. However, these priorities focused on preserving access

(to quality education), which was at greater risk during the 2008/2009 financial crisis. Each

restructuring in this period included an incremental growth of funding dedicated towards teacher

salaries (to increase lower secondary access) and away from quality-improving interventions.

Either the PDO should have been redefined to give a greater weight to access, or the

reallocations and indicators should not have sacrificed quality for access.

42. In period 3, the results framework was simplified to two PDO Indicators, the definitions

for many indicators were clarified to eliminate ambiguity, and the entire list of indicators for the

first time only included data the Government could reasonably provide.

43. The relevance of Implementation is Modest in all periods. Restructurings ensured

Project implementation remained relevant to Government priorities, but not the original PDO

which focused both on access and on quality. Five restructurings took place during the life of the

Project, which allowed it to remain relevant to the country’s priorities and to respond to the fiscal

gap beginning with the economic crisis of 2008/200941

. This was mainly reflected in the changes

in fund allocations.

44. While all the changes to the chosen interventions aligned with the objective of improving

access to quality education, it shifted the focus away from quality and towards access, and had an

aggregate effect of changing the project to be almost entirely about access42

.

3.2 Achievement of Project Development Objectives

45. The achievement of the PDO is rated Substantial for periods 1 and 2, and Modest for

period 3. The PDO was to improve access to a quality lower secondary education for low

income students, especially in indigenous communities43

, through improved primary education

completion rates for overage students, strengthened flexible lower secondary education

modalities and school management. Access and quality to a lower secondary education comprise

the two final objectives of the Project, and improved primary completion, strengthened lower

secondary modalities, and school management comprise the three direct outcomes to achieving

the said goals.

40 On the reversal of school-based management policies for Guatemala, see Ganimián (2016).

41 The changes ensured financing for the facilitators of Telesecundaria, until the Government could take over the costs when the

fiscal situation improved.

42 This is well documented in Section D of the Datasheet, which shows that the “Education for All” theme code – namely

representing access-related interventions – ended up absorbing 82 percent of the total Loan proceeds.

43 See Annex 11 for further detail on beneficiaries and targeting.

16

46. Outcome achievement of each of the five PDO parts is measured using indicators, as

shown as the final and direct outcomes of the causal chain (

47.

48. Table 2). Outcome achievement mapped the five PDO Indicators to the PDO parts and

supplemented with Intermediate Outcome Indicators (See Annex 10)44

. There is no indicator in

the results framework that adequately measures lower secondary education quality. Although the

results framework includes gross completion rates, which usually reflect quality, the data suggest

flexible modalities schools have much lower minimum learning requirements to pass. Thus pass

rates on a standardized learning examination at the end of lower secondary were introduced at

the Implementation Completion and Results (ICR) Report stage and used to measure education

quality, and the target was set to maintain quality at baseline levels.

Table 2: Causal Chain and Indicators of Project Objectives

Final

Outcomes

Access to Lower Secondary Education

Access to 7th grade in flexible modalities (Telesecundaria and NUFED)

Gross enrollment rate in lower secondary education (grades 7-9)

Gross Completion of 9th grade

Quality of Lower Secondary Education

Percentage of 9th grade students who pass standardized reading and math

examination

Direct

Outcomes

Improved Primary Completion

Sixth Grade Gross Completion Rate

Proportion of overage pupils who complete primary education (6th grade)

Strengthened Flexible Modalities

Number of Lower Secondary Schools operating under the modalities financed by the

Project that receive intercultural and instructional materials (Telesecundaria and

NUFED)

Project-supported schools receiving new curricular orientations and materials

School Management

Number of school councils that receive grants and report expenditures satisfactorily

Number and percent of school councils trained

Outputs Number of scholarships financed by the project that are granted to students from

targeted municipalities that comply with attendance requirements

Number of teachers that completed the in-service teacher training program

(PADEP/D)

Number of classes within schools that offer the PEPS Program

Inputs

Spending on instructional materials for flexible modalities

Funded scholarships for low income students

Spending on PADEP/D in service teacher training program

Public spending on teacher salaries for PEPS and Telesecundaria

44 Each of the PDO parts are rated under the 4-point scale (High, Substantial, Modest, and Negligible) according to the following

classification: Negligible: 0-65 percent achievement of target; Modest: 65-85 percent achievement of target; Substantial: 85-105

percent achievement of target; and High: Over 105 percent achievement of target.

17

Public spending on education services

Increase Access to Lower Secondary Education (P1: Substantial, P2: Substantial, P3: Modest)

49. An additional 40,729 students started lower secondary due to scaling up flexible

modalities45

. The number of Telesecundaria schools tripled over the life of the project, which

could not have occurred without project support of teacher salaries and education materials. As a

result, the number of Telesecundaria students and the total number of students enrolled in 7th

grade of this modality reached a total of 40,729 students. This falls short of the period 1 target of

103,500, when the project was intended to provide some support to NGO, private, and municipal

schools, but exceeds the period 2 target, which reflected the change in implementation strategy.

The indicator was changed to an intermediate indicator in period 3 and excluded from the

analysis.

50. Gross enrollment rates improved and reached national but not priority area targets. The

national lower secondary GER increased from 61.0 to 68.2 percent, surpassing the period 1

target of 65 percent but falling short of the period 2 target of 73 percent46

. National targets were

removed from the PDO Indicator list in period 3. In targeted municipalities, GER started at 42

percent and reached 51 percent47

. It achieves the targets used in periods 1 and 3, but not in period

248

. The plateau and decline of GER is caused not by declines in enrollment, but a failure of

enrollment to keep up with population growth (see ACEPT, 2015). The National Institute for

Statistics, based on the 2002 census, estimates that Guatemala’s population grows faster than that

of any other country in Latin America, making the constant expansion of education provision a

challenge. To achieve the national targets, enrollment would have had to increase by 4.5 percent

annually, instead of 4.0 percent annually.

51. Gross Completion Rate (GCR) for 9th

grade largely achieved its goal. Nationally, GCRs

increased from 50 to 55 percent, surpassing its targets. In targeted municipalities, the indicator

stabilized at 43 percent in 2014, surpassing its targets from periods 1 and 2 but only achieving 73

percent of the period 3 target. Although GCRs in many countries serve as a proxy for education

quality (by demonstrating the percentage of students who reach the final grade), this assumes

45 Of the two flexible modalities, Telesecundaria was clearly the most important intervention (achievement of 146%). Even if

weighted the same as NUFED (attainment of 27%), the average attainment is still high (86.5%), which falls into the lower range

of the High category (86-100%).

46 The GER nationwide was supposed to increase from 61% in 2007 to 65% in 2012 (this target was moved upwards twice, first

to 72% by 2012 and then to 73% by 2013). This indicator was dropped in December of 2012 (Restructuring No. 4), since it was

not a targeted indicator. However considering that the main intervention of the project- Telesecundaria (around 40% of total

project costs) - was not targeted, this indicator is relevant. The reason this modality was not supported in targeted municipalities

is because the intervention itself entails targeting towards rural and poor communities.

47 As for the GER for 196 targeted municipalities, the indicator was supposed to increase from 42% in 2007 to 58% in 2012 (this

target was revised four times, once downward to 52% and then progressively up, with a final target of 56% by 2015).

48 The target was revised downward in Restructuring No. 2, without a specific explanation, but upward again in Restructuring

No. 5 when it appeared it would surpass the Restructuring No. 2 targets.

18

students they learn all the requisite knowledge and skills before progressing to the following

grade. However, in the modalities supported, grade progression is higher an learning outcomes

are lower than in traditional public secondary schools, suggesting that completion indicates

attending for three (or more year/s) but not demonstrating actual learning. However, given the

opportunity cost for attending school, students who complete have demonstrated improved

educational access.

Improve Quality of Lower Secondary Education

(P1-P3: High for TIs ; Negligible for NUFEDs)

Average Outcome Rating : Modest

52. Though the PDO and PAD state the goal of access to a quality education, no quality

measures were part of the results framework during the lifetime of the Project. In an effort to

document this part of the PDO though, the ICR team presents a proxy measure of learning

achievement that is comparable over time and sufficiently robust to help determine a related ICR

rating for quality. This measure is the percentage of 9th

grade students who received a proficient

grade in Math and Reading and is provided by MINED through the national standardized

assessment of learning outcomes. The information for the three years (2006, 2009 and 2013) for

which the standardized test were undertaken is presented in Table 3 below, by school type.49

Table 3: Quality of Lower Secondary Education Measures Incorporated in the ICR

analysis, by type of LSE institution

Source: World Bank based on information from MINEDUC (2015f).

53. Since these quality measures were not part of the Results Framework, one important

challenge was to identify adequate targets that could have been reasonable at design

considering the stylized fact that increased access usually has a negative effect on the average

quality of learning outcomes. The negative elasticity of quality to access is due to the fact that

increases in access are associated with a new influx of students in the bottom of the distribution

of test scores and, as such, exert a negative effect on the total average of the whole distribution.

Therefore, the expected result on quality would be that it would decrease and therefore the

challenge would be to minimize, to the best of the system’s ability, this decrease (Table 3 above

illustrates this decline in learning outcomes over time).

54. Taking into account the expected negative effects on quality of access-increasing

interventions, the potential impact of the Project on quality is rated Modest. Table 4 presents this

analysis. As can be seen, taking into account both the negative quality-access elasticities

calculated between 2006 and 2009 and the real increase in access between 2009 and 2013, we set

49 Only regular public schools and the two flexible modalities supported by the Project (TIs and NUFEDs) are shown for the

purposes of this analysis due to data limitations.

19

ex-post targets that would have been expected for the quality measures. These targets were then

compared with the real decrease in quality by type of institution supported under the Project.

Table 4 shows that TIs increased the number of 9th

graders in the country by 136 percent from

2009 through 2013. Utilizing the quality-access elasticity for this type of flexible modality, such

a sizeable increase of access should have had a sizeable downward effect on quality. However,

although the effect was indeed negative, it was far less negative than would have been expected

based on the decline in the period prior to the launch of the Project. This implies a positive effect

of the Project on minimizing the expected reduction in quality due to increased access to

populations with lower attainment. With respect to NUFEDs, a much less negative impact on

quality was expected, given its historical (2006-2009) elasticity, and yet the decrease in quality

was much higher than what could have been expected due to the milder increase in access, as

opposed to TIs. A calculation of this overly negative result ends up in a percentage of target

achievement that is significantly low (28 percent).

Table 4: Analysis of Expected Impact on Quality of Increased Access

Source: World Bank based on information from MINEDUC (2015f).

55. Given the need to provide a combined rating of achievement for both TIs and NUFEDS,

the team carried out both simple and weighted average calculations for the achievement of

target, with the weights being given by the size of the specific intervention (or flexible modality),

as measured by the number of students benefitted in each of the cases. The simple average yields

a 92.8 percent achievement of target (achieved), but the weighted average yields a 127.6 percent

achievement of target (surpassed), due to the larger number of beneficiaries in TIs. Even though

these results would then lead to either a “Substantial” (simple average) or “High” (weighted

average) rating for quality, given the performance of the NUFEDS, the ICR team took a

conservative approach and rated this indicator as Modest. For the purpose of Table 5 (below), the

combined achievement of target was then set at 75 percent, which represents the mid-point of the

“Modest” scale.

Improved Primary Education

(P1: Substantial, P2: High, P3: Substantial)

% AssessmentEnrollment

20134 Weights % Assessment

TIs 40.4 -17.2 -0.43 135.7 -57.9 -33.5 157.9 Surpassed 29,042 0.77 157.9 Surpassed

NUFEDs 124.4 -32.9 -0.26 24.9 -6.6 -30.4 27.6 Not achieved 8,777 0.23 27.6 Not achieved

92.8 Achieved 127.6 Surpassed

1 Represents the % increase in the number of students enrolled in 9th grade.

2 Represents the % decrease in average test scores (combined reading and math) for 9th graders.

Achievement of Target

(non-weighted calculation)

Achievement of Target

(weighted calculation)

3 Represents the expected % decrease in average test scores (combined reading and math) for 9th graders given: i) the 2006-2009 elasticity of quality to access; and ii) the %

increase in access 2009-2013.

Note: "Test scores" is an expression utilized for the sake of referring to standard language for measuring learning outcomes. In this particular case, though, test scores are calculated as the average percentage of 9th-graders

which have the minimum level of grade-appropriate proficiency.

Real

Decrease

in quality

2009-2013

(%)

Type of LSE

Institution

Average TIs/NUFEDs

Increase

in access

2006-2009

(%)1

Decrease

in quality

2006-2009

(%)2

Quality /

Access

Elasticity

Increase

in access

2009-2013

(%)

Expected

decrease in

quality

2009-2013

(%)3

4 Represents the number of students enrolled in 9th grade.

20

56. Primary education completion significantly improved. The primary completion rate

increased from 76 to 84 percent nationally and 78 to 87 percent in priority areas, achieving

targets set in all periods. The proportion of overage pupils completing primary education fell

over the course of the project, this may be linked to the reduction in the total number of overage

pupils and not the quality of the services they received. The Accelerated Primary program

reduced repetition, increased completion, and allowed non-PEPS classrooms to focus on non-

repeaters in select schools. The overall increase of primary completion is a better indicator of

overall achievement. Although primary education outcomes have improved significantly, it is

unclear how much of the improvement can be linked directly to project interventions. Although

qualitative studies by the Ministry of Education suggest that the teachers were satisfied with

training, it is unknown whether the training actually improved student achievement. While the

accelerated primary program did have a measurably positive effect on primary completion and a

positive net benefit, the magnitude of the effect is much too small to be a significant driver of the

national primary completion rate.

Strengthened Flexible Modalities

(P1: Substantial, P2: Substantial, P3: Substantial)

57. The Project strengthened lower secondary modalities by equipping them with computers

and instructional and intercultural materials. With funding from the Project, not only did every

Telesecundaria and NUFED in the country get books for teachers and students, but these books

were fully aligned to the national curriculum for each of these two modalities. Equipment and

furniture also made a difference to the 1,700 TIs and the 600 NUFEDs (see Indicator 30),

catering to an estimated enrollment of approximately 140,000 students from low income

communities across the country that would have not enrolled had the project not existed.

58. Provision of additional materials was also included in the project. The project also

supported the provision of education materials. Although the distribution of most of these

materials took place in the final year of the project, they likely will improve the quality of

learning that takes place in Telesecundaria and NUFED schools.

59. These schools also benefitted from funding from the Project with as many as 400

different titles of reading books as part of the flagship National Reading Program launched in

2013 and by the end of the project, flexible modalities were better equipped to deliver a higher

quality educational service by having more resources at their disposition.

School Management

(P1: Substantial, P2: Negligible, P3: Modest)

60. The goal to improve school management included a two-prong approach: (i) to prepare

resources that would help inform parent councils and (ii) train parent councils (called school

governments) how to monitor, manage, and lead their schools. Although some useful reports and

online school report cards tool exist, the original vision of devolving substantial school decision-

making to the local level was not achieved. The goal was to build the capacity of 1,500 school

councils to monitor and lead their schools, as well as apply for school improvement grants. By

the end of the project 1,498 groups had received some form of training but the quantity and

21

quality of training is difficult to assess since the office managing the work until 2012 did not

relay the list of schools who had received training and only 158 of the 1,498 school boards

trained applied for and received school improvement grants since schools were not given the

opportunity to apply for funds under the new Administration. The other funds for improved

education management supported the central Ministry offices, and their effects are likely

disparate but significant throughout the system.

Table 5. Efficacy Rating Detailed


Eff

ica

cy

Access Substantial 103% Substantial 108% Modest 63%

Quality* Modest *75% Modest *75% Modest *75%

Primary Substantial 93% High 120% Substantial 100%

Secondary Substantial 104% Substantial 104% Substantial 105%

Management Substantial 99% Negligible 55% Modest 71%

OVERALL Substantial 93% Substantial 91% Modest 81% Note: Achievement of indicators under each PDO part averaged for rating. Ratings calculated as: Negligible: 0-65%; Modest: 65-

85%; Substantial: 85-104%; High: Surpassed Over 105%. Percent achievement of each PDO part was calculated as the average

of achievement of all indicators linked to that PDO part. For achievement of each indicator, see Annex 10.

* Percentages of achievement for quality are set at the mid-point for Modest, as explained above.

61. Overall, the project made substantial progress towards its intermediate goals by

improving primary completion rate, providing the materials needed at the secondary level, and to

a lesser degree, improving school management (Table 5. Efficacy Rating Detailed). In terms of

improving access, its success was only mitigated by the high growth rate of the population.

However, quality outcomes remained below expectations, and represent a significant

shortcoming. The lower achievement against the Period 3 targets should not be viewed as lower

achievement in the final two years of the Project, as it measures final achievement against the

period 3 results framework.

3.3 Efficiency

62. The Project’s overall efficiency is Substantial (Substantial in all three periods). The

ICR economic and financial analysis estimated an economic rate of return of 21 percent. During

the appraisal process, an economic and financial analysis estimated an economic rate of return of

31 percent. The overall cost-effectiveness of the project is high and it is largely based on the

estimated income benefit of secondary completion (based average incomes in 2014).In addition,

even if the income benefits are half of the original estimate, the cost-benefit analysis would

remain positive. Overall, the project would have had an even more positive impact had there not

been implementation delays, but these delays are reflected in the cost-benefit analysis’ discount

rates, and net benefits remain high.

63. The economic analysis measured the costs and benefits of four interventions: accelerated

primary, two secondary school modalities (Telesecundaria and NUFED), and the secondary

scholarship program. Accelerated primary classes had a small but positive net benefit. Increasing

lower secondary education access, which comprised the largest proportion of funding for the

project, had the largest relative and absolute impact.

22

64. Accelerated Primary Classrooms. PEPS created special classes for over-age students and

allowed them to cover multiple years of material in a single year. It provided an economic rate of

return of 23 percent by reducing repetition, increasing completion, and allowing non-PEPS

classrooms to focus on non-repeaters. Completion rates of overage students were only slightly

higher in schools with a PEPS program: an estimated 603 additional students graduated primary

compared to if PEPS had not existed. Although the effects of the PEPS program were not

dramatic in size, they resulted in a respectable benefit given the low cost of the intervention.

65. Expanded Secondary Education Access. The Project dedicated the majority of its funding

to increasing secondary access through two types of low-cost secondary schools: TIs and

NUFEDs. TIs provided a rate of return of 48 percent, and NUFED schools provided a rate of

return of 42 percent. In theory, Telesecundaria rely more on educational materials and

equipment and less on teacher subject knowledge, and NUFED schools use alternative schedules

and holistic curriculum to target rural student populations otherwise unlikely to complete

secondary education. As a result, the NUFED approach may indeed add value beyond the

academic. The increased income of the lower secondary graduates from NUFED and

Telesecundaria schools leads, however, to the largest economic gains of the project.

66. NUFED schools have lower economic returns than Telesecundaria schools due to

substantially lower graduation rates and slightly higher costs. However, given that NUFED

schools strategically target student populations otherwise unlikely to complete secondary school,

higher costs may be the result of whom they educate.

67. The financial cost of the flexible modalities totaled US$172.8 million over the course of

the project. The loan supported one-third of the costs, and other Government funds supported the

remainder. Telesecundaria comprised 90 percent of the total financial cost, and NUFED the

remaining 10 percent. Based on the change in enrollment since the beginning of the project, the

expansion lead to an additional 91,060 students graduating from Telesecundaria schools and an

additional 15,053 graduating from NUFED schools. After adjusting for net present values, the

net benefit of the flexible modalities was US$868 million, resulting in a rate of return of 48

percent.

68. Scholarships. The scholarship program offered an internal rate of return of 44 percent.

The program provided over 25,000 one-year cash scholarships to lower secondary students.

Applicants who received the scholarship were academically and demographically similar to non-

recipients. However, scholarship recipients were over 40 percent more likely to graduate than

comparable non-recipients. The cost-savings of lower repetition rates and economic benefits of

secondary graduation offset the cost of the scholarships.

69. Quality-Improveming Activities. Two activities explicitly meant to improve lower

secondary education quality -- school boards and an intensive teacher training program – lacked .

sufficient records and data to conduct a reliable economic and financial analysis. The removal of

funding for impact evaluations of the primary education interventions, the cost-effectiveness of

the supported secondary modalities, and the impact of school board and teacher training

interventions, made quantifiable measurement of any effect on school quality with certainty of

23

the cause unreliable, and they were too targeted to have an effect on national education

indicators.

70. One efficient design of the project was that of the PEPS program. Although its small size

limited its effect, the program essentially responds to the high rates of repetition by tracking

students into classes for repeaters. In theory, this design requires little additional resources

(beyond customized learning materials), but has the potential to reduce the high repetition rates.

71. From 2012 onward, the number of scholarships available were stratified in municipality

and community by the local poverty rates. Locally, teachers and committees selected the best

candidates. This mix of quantitative stratification and human interaction to improve targeting

likely resulted in improved outcomes for the scholarship program.

3.4 Justification of Overall Outcome Rating

72. The overall Outcome rating for the Project is Moderately Satisfactory. During the

first two periods, the complex design of results framework resulted in Modest Project Relevance,

but Efficacy was rated Substantial. In Period 3, while the Relevance improved, the achievement

of project targets declined, resulting in a Moderately Unsatisfactory. As a result, the Outcome

ratings for Periods 1 and 2 are Moderately Satisfactory, and Moderately Unsatisfactory for

Period 3. With Relevance and Efficiency ratings varying between Modest and Substantial, the

Efficacy ratings were the key determinant. Given that the Project was rated Moderately

Satisfactory over 73 percent of the project’s time and 58 percent of disbursement, its the

weighted average of the ratings of the thre period results in a final rating of Moderately

Satisfactory.

Table 3. Outcome Achievement by PDO Part

3.5 Overarching Themes, Other Outcomes and Impacts

73. The Project was successful in expanding access at the lower secondary level, but did not

succeed in ensuring quality. Education quality was low at the beginning of the project, and

remained low since. One key achievement of the Project was the implementation of the

Indigenous People’s Plan: as previously mentioned, it resulted in consideration of the needs of a

multilingual and multicultural population in a greater number of decisions made by MINEDUC.


Relevance Modest (2) Modest (2) Substantial (3)

Efficacy Substantial (3) Substantial (3) Modest (2)

Efficiency Substantial (3) Substantial (3) Substantial (3)

Rating by Period Moderately

Satisfactory

Moderately

Satisfactory

Moderately

Unsatisfactory

Disbursement 7% 51% 42%

Final Outcome MODERATELY SATISFACTORY

Note: A detailed list of indicators used to determine Efficacy subratings can be found in Annex 10.

24

4. Assessment of Risk to Development Outcome

74. The risk to development outcome is rated as Substantial. The main risk is that it is too

early to tell if the new Administration will continue to support interventions. It is unclear if the

government will allocate funding to continue paying the University of San Carlos de Guatemala

(USAC) for the PADEP/D program, the school board trainings, or the scholarship program.

However, flexible lower secondary modalities- Telesecundaria and NUFED- have completely

been absorbed by the MINEDUC budget, and their continuation is secured.

75. The government’s ability to determine which interventions to keep is further hindered by

the elimination of impact evaluations, which were meant to identify and prioritize successful

interventions. Furthermore, of the 1,489 school councils that were trained, only 158 received

funds from the Project. School councils stopped trying to legally form once it was clear that they

would not receive Project funds, and the Government did not seem to have taken steps to

continue the program. Given that the training and legalization process were geared towards

applying for Project funds, there will be little future benefit gained from these efforts. Finally, in

the past years, the Government has had several delays in paying for the PADEP/D teacher

training program (one year delay in paying the 5th

cohort). However, each year the government

has ended up paying the USAC. Whether the struggle for securing funding for PADEP/D may

continue or not is still uncertain as of the date of this report.

5. Assessment of Bank and Borrower Performance

5.1 Bank Performance: Moderately Unsatisfactory

76. Quality at Entry: Moderately Unsatisfactory. At entry, the Project design was in line

with the Government’s priorities and relevant to sector and development priorities. However,

there were several shortcomings. The project was designed near the end of the Berger

Administration, and should have taken one of two strategies: either i) design it quickly and begin

implementation before the Administration’s end, or (ii) design the project with a great deal of

flexibility to allow the incoming Administration to make it fit its priorities. Instead, the project

was designed with specific ideas unlikely to be adopted and was not signed until after

Administrations changed, and immediately began discussion of revision. In addition to the five

PDO Indicators, the original results framework required 43 data points for intermediate outcome

indicators (and lacked baseline data in almost all cases).

77. This made a complete revision during Restructuring No. 2 necessary. In addition, the

original plan set challenging targets to meet to increase access for a Project that originally

focused on quality improvements. The PDO should have made a clear priority of one or the other

or the results framework and implementation arrangements should have been more conservative.

In addition, the Bank did not thoroughly appraise initial implementation arrangements within the

MINEDUC, where line units were not made responsible for results, affecting implementation

later on. Accelerated Primary, school councils and the Indigenous People’s Plan all struggled due

to this weakness. Finally, systematic targeting of the scholarship program was contemplated at

design but not completed; in early years, it is unknown whether selection was based on real

needs.

25

78. Quality of Supervision: Moderately Unsatisfactory. The Bank carried out 20

supervision missions, which were used to monitor the progress of the Project, carry out field

visits and to resolve any issues that would delay implementation. The implementation documents

adequately reflect the agreements that the Bank and Borrower came to during these missions.

79. The five restructurings helped improve project relevance to government priorities and

address implementation issues, but the need for repeated changes to the results framework

unrelated to implementation suggests issues remained after each one. While Restructurings No. 1

and No. 2 were critical to update a project prepared under a different government and improve

weaknesses in design, it is unclear whether Restructurings No. 3 and No. 4 were beneficial.

Though Restructuring No. 5 was necessary to extend the project, the fact that the results

framework had to be substantially reworked so late into implementation suggests a failure of

previous restructurings.

80. Procurement and financial management were adequately supervised. However, missions

to assure the compliance with the IPP (social safeguards) were not carried out throughout the

entire Project (mostly in the beginning and at the end). Unfortunately, the impact evaluation of

the key programs to ensure that the programs were implemented as expected was dropped, which

would have been a useful tool to adjust programs along the way and also to measure the impact

of the project at closing. The Bank may have probably insisted on having a stronger M&E

system to track Project performance. In addition, during the project there were six different Task

Team Leaders (TTLs)50

; while some handovers involved some overlap in team staffing, others

presented a significant learning curve for the new TTL.

5.2 Borrower Performance: Moderately Satisfactory

81. Government Performance: Moderately Unsatisfactory. During the life of the Project,

there were four different Presidents of Guatemala. Project approval by Congress took 15 months.

Although there was a lack of reporting about Government financial support of the project,

funding for teacher salaries alone exceeded the originally planned Government support.

82. The Government has paid for NUFED teachers throughout the life of the Project and paid

all Telesecundaria salaries in 2010. The economic and financial analysis estimates Government

teacher salary support over the course of the Project at approximately US$114 million (nominal).

83. Certain institutional arrangements were inadequate to obtain the expected results, such as:

School Councils. Even though one directorate, the General Directorate for Community

Participation and Support Services (DIGEPSA), had vast experience in managing school

councils at the primary level, another directorate, the General Directorate for

Strengthening Community-Based Education (DIGEFOCE) was in charge of lower

secondary school councils. This is the likely cause of the slow progress in the first three

years of the project. When the Administration changed in 2012, almost no information

50 The complete list of TTLs during the implementation stage can be found in Annex 4.

26

about the management of councils was passed on internally. The new Administration did

not know where the previous one had been working, and started the entire process over51

.

This delayed council formation in order to receive funds partly explains why a low

number of councils received funds.

Scholarships. The responsibility of managing the scholarships fell to the Directorate for

Education Planning (DIPLAN) in 2009 and 2010, but was transferred to DIGEPSA in

2011 without appropriate records. In 2012, the no records of students were given to the

new Administration. That year, no funds were allocated: an audit took place and the

Directorate for Information Technology (DINFO) created a formal targeting system and

method for records keeping. In 2013, the program was again placed under the

responsibility of DIPLAN and well managed for the remainder of the project.

PEPS. Assigning the Accelerated Primary program to the Directorate of Intercultural and

Bilingual Education instead of the directorate experienced with delivering primary

education complicated implementation and supervision.

84. Despite having dropped the impact evaluations, DIGEDUCA carried out several studies

on PEPS, flexible modalities, PADEP/D teacher training program, use of library books, among

others.

85. The preparation documents stated that the Government would spend US$20 million to

support the Project, though what this was expected to go to was not made explicit. Government

spending on the project is unknown, as it was not tracked separately from general Government

spending. Teacher salaries generally comprise the largest ongoing cost in education. While the

project was modified to allow the loan to be used for salaries, the majority of the cost to maintain

the schools included in the project was absorbed by the Project. According to estimates used in

the economic and financial analysis, Government funding of NUFED and Telesecundaria

salaries alone likely cost approximately US$114 million.

86. Implementing Agency Performance: Moderately Satisfactory. The Project was

implemented by a Project Implementation Unit (PIU) embedded in the Ministry of Education.

The implementing unit did a good job in tracking day-to-day implementation of the Project with

the various operational, financial and fiduciary offices. Every six months, the PIU sent the Bank

a financial report on the advances of the Project and also the Annual Operation Plan with its

Procurement Plan. Audit reports were presented on time and did not present problems. Largely,

the delays and management issues were caused by changes in Administration and management

outside of the PIU’s area of influence.

6. Lessons Learned

87. Well-designed Safeguard systems can create lasting institutional change. While the

immediate purpose of the Indigenous Peoples’ Plan was to help the Government improve work

related to the project, it served helped the Government develop its own plan to better serve the

51 See, for instance, the Guidelines that were produced and published in MINEDUC (2013b).

27

subpopulation. It helped raise issues and bring to the table stakeholders who are now integrated

into the Ministry of Education’s decision-making processes.

88. Restructurings should ensure internal consistency. The PDO, results framework, and

interventions all need to be aligned with each other for a project to be successful. The decision to

allow funds to pay for teacher salaries shifted in the focus from quality towards access, but a

formal shift of the PDO to explicitly reduce the emphasis on quality would have improved the

final outcome rating.

89. The best reason to try new interventions is to evaluate whether they work. After scaling

up flexible modalities over the course of eight years, the effectiveness and efficiency of these

schools remains unknown due to the elimination of the impact evaluations. Although enrollment,

learning, and non-systematic qualitative data exist, without ongoing analysis of implementation

and results, future administrations cannot know whether they are a worthy investment.

Incorporating evaluation and systems learning into the PDO would emphasize the importance of

gathering information for making informed decisions in the future.

90. Adequate record keeping and tracking of beneficiaries is necessary to maintain program

transparency and continuity, especially when there are changes in government. While this

project faced unusually high World Bank and Government turnover, changes in staff are

common in all projects. While records of financial, legal, and implementation missions were

adequately kept, key information about implementation (such as lists of beneficiaries and

implementation methods) were not passed between administrations or Team Leaders.

Maintaining more detailed records and evidence could safeguard against cases where one party

or the other does not maintain adequate records.

91. Long and complex results frameworks are harmful to monitoring progress. Despite the

temptation to include many indicators for a more nuanced understanding, overburdening the

Government with data requirements reduces the likelihood of receiving any meaningful

reporting. In addition, indicators based on the multiplication of multiple values (such dividing by

the population) increase the number of assumptions made when creating targets. Simpler, raw

numbers with explicit explanations on how they were calculated are more effective. Creation of

targets should be well founded in unit costs of implementation, to allow future revisions to be

founded in the same logic as project design.

92. Small, highly targeted programs can be very good investments. The scholarship program

was originally designed to comprise 3 percent of Project funds. Since 2012, the Government

paired systematic targeting of areas using local poverty rates to provide scholarship vacancies

with school and regional committees to select individual beneficiaries. Receiving a scholarship

increased student chances of graduating by over 40 percent. While a program targeting the

highest achieving poor students may not be scalable, it serves as a reminder that large projects

can be successful by using well-designed small interventions.

28

7. Comments on Issues Raised by Borrower and Implementing

Agencies

93. The Borrower report usefully points out that turnover in the Government and World Bank

team leaders lead to implementation delays. Even as new administrations consider new strategic

priorities, they can plan transitions to ensure that ongoing activities are not disrupted and records

and critical staff are not lost. The Borrower report usefully points out that technical and project

implementation unit staff turnover should be avoided whenever possible.

94. The Borrower Report also points out that World Bank turnover also hurt implementation

support, which is also recognized in the World Bank Implementation section above. Multiple

joint missions of incoming and outgoing TTLs would likely minimize transition problems. The

Government also suggests having the TTL be based either in Guatemala to improve timely

decision-making, or if not, in Washington DC to have more direct access to World Bank

management, which is worth considering for future projects. The project varied between having

TTLs based in Washington and third countries, but never in Guatemala.

95. The Borrower Report also agrees with this report’s finding that the IPP was a useful tool

to highlight indigenous peoples’ issues. While the discussed activities were one positive

outcome, mainstreaming these issues into implementation are likely the greater long-term benefit

caused by the IPP’s conclusion.

29

Annex 1. Project Costs and Financing

(a) Project Cost by Component (in USD Million equivalent)

Components Appraisal Estimate

(USD millions)

Actual/Latest

Estimate (USD

millions)

Percentage of

Appraisal

1. Primary Education Completion

and Quality 7.7 8.8 115

2. Access and Quality of the

Lower Secondary Education 51.2 67.5 132

3. Education System Management

in Support of Education Access

and Quality

21.0 3.5 17

Total Project Costs

(including contingencies) 79.8 79.8 100

Front-end fee IBRD 0.2 0.2 100

Total Financing Required 80.0 80.0 100

(b) Financing

Source of Funds Type of

Cofinancing

Appraisal

Estimate

(USD

millions)

Actual/Latest

Estimate

(USD

millions)

Percentage of

Appraisal

Borrower 20.00 0.0052

0.00

International Bank for

Reconstruction and Development 80.00 80.00 100.00

52 Although actual spending is unknown, the Government likely spent at least $100 million on teacher salaries in schools related

to the Project. See Section 5.2 (Borrower Performance) of the main text.

30

Annex 2. Outputs by Component53

Subcomponent 1a. Support of Quality Primary Education

Summary: While originally designed to use IBECs, it was changed during Restructuring No. 2 to

focus exclusively on the PEPS. To increase retention and completion, primary school students

who had repeated two or more years attend classes that cover multiple years of material in a

single year (2nd and 3rd grade in year 1, 4th, 5th, and 6th grade in year 2). By the Project’s end,

22,100 students had been enrolled in the programs. According to comparisons in the economic

and financial analysis, promotion rates increased 2.8 percentage points in control primary

schools and 3.7 percentage points in PEPS schools per year. The difference of 0.9 percentage

points was applied to the entire life of the project suggests 2,433 fewer repeating students. This

frees up resources that can be used by other students or to lower the average cost of education.

Outputs:

Learning materials for PEPS students $ 357,693

Socio-linguistic characterization of schools (IPP) $ 279,319

Training workshops for PEPS teachers $ 198,534

Consultancies for adapting PEPS learning material to CNB (IPP) $ 147,575

Furniture and basic equipment for PEPS classes $ 62,402

Didactic materials for PEPS teachers $ 33,114

National Reading Program support material for PEPS classes $ 25,039

Total Outputs Sub-Component 1a $1,103,676

Subcomponent 1b. Support Teacher Training

Summary: Originally designed to improve the quality of teaching in primary schools through

improved pre-service, this subcomponent was revised in Restructuring No. 2 to focus on in-

service training of pre-primary and primary students through PADEP/D. The 13,661 teachers

received ongoing training in the student-centered learning model, above its target (see Indicator

14). The Government conducted evaluations of the training program, which found high

satisfaction among teachers but there is no information on whether or not the program improves

teacher effectiveness. One important issue to note regarding PADEP/D is that it originally

promised teachers the certification would increase their salaries, and it was generally believed

the learning time could be applied as credits towards a tertiary degree in primary education.

However, no salary bonus was ever provided, and there remains no tertiary degree in primary

education available at the only public university. Although studies have been conducted by

USAID (USAID, 2010) and MINEDUC (MINEDUC 2012c, 2014d, forthcoming.d), none have

compared learning outcomes among students of PADEP/D certified teachers and those who did

not receive the training. Largely due to the elimination of funding for subcomponent 2a, its

effectiveness remains unproven. While it may have contributed to the increase in primary

53 This section presents a short summary of the main results of each component, broken down by sub-component. Outputs by

sub-component are presented in US dollar-equivalent amounts at the exchange rate valid at closing date of the Project and

rounded to the first integer. This may result in differences between these numbers and the amounts reported in Annex 1.

31

completion observed in the PDO Indicator mentioned above, PADEP/D training recipients were

not tracked, so it is not possible to ascertain whether the program caused any of the completion

rate change.

Outputs:

Financial support for PADEP/D Program $ 7,637,148

Equipment for Teacher Training High Schools $ 42,622

Total Outputs Sub-Component 1b $ 7,679,770

Subcomponent 2a. Feasibility studies and strategies for secondary education curricula and

evaluation systems

Summary: Component 2a was defunded and dropped by Restructuring No. 2. No feasibility

studies were conducted, and the new curricula, flexible modalities, and various interventions

adopted during the Project were not the subject of rigorous evaluation. While several studies

were conducted (see Section 2.3 of the main text), none were integrated into implementation, and

they lack the rigor necessary to determine the actual effects of the interventions. This is

particularly disappointing, given that the Project was designed to try multiple new education

interventions to determine their overall effectiveness. No progress was made under this

subcomponent, nor any indicator monitored before its elimination.

Subcomponent 2b. Strengthening and Expansion of Pertinent and Flexible Modalities for

Lower Secondary Education

Summary: Flexible modalities were originally intended to be allocated $43.8 million of the loan,

but comprised roughly US$54 million of loan disbursements. In addition, the government

provided an estimated US$114 million in teacher salaries. The original intention was to support a

variety of lower secondary modalities: NUFED, Telesecundaria, cooperatives, NGO schools, and

improved regular LSE schools (INEBs). Instead, funding was limited to Telesecundaria and

NUFED schools. Three percent of total loan funding went to NUFED schools, but enrollment did

not substantially expand during the Project. NUFED schools only achieved 64 percent of the

goal to increase intake. In reality, NUFED enrollment increased over the course of the Project,

resulting in an additional 15,000 students enrolling; however, by 2014, it had returned to

enrollment levels similar to those of earlier years. Telesecundarias represent the most

measurable gain from the Project: access increased by 40,729 over the course of the project,

resulting in a total of over 91,050 additional graduates between effectiveness and project close.

Outputs:

Payment of TI teacher salaries $31,123,011

School equipment for TIs $ 9,819,957

National Reading Program materials for TIs $ 4,605,835

Furniture for TIs $ 4,135,175

Learning Materials for TI students and teachers $ 4,007,261

Training workshops for TI teachers $ 2,020,104

National Reading Program materials for NUFEDs $ 1,388,366

Training workshops for NUFED teachers $ 765,011

32

School equipment for NUFEDs $ 486,816

Consultancies for adapting TI modality to CNB $ 285,125

Learning Materials for NUFED students and teachers $ 185,851

Furniture for NUFEDs $ 60,970

Consultancies for adapting NUFED modality to CNB $ 51,301

Total Outputs Sub-Component 2b $58,934,783

Subcomponent 2c. Education Demand Scholarships for Low Income Students The scholarships were originally designed to only comprise US$2.5 million of the loan, but

ultimately comprised approximately US$10.7 million. Ultimately, 37,847 (one-year)

scholarships were provided. Scholarship applicants who received the scholarship were 9 percent

more likely to pass a grade than those that did not, leading to an estimated additional 2,465

students graduating from lower secondary.

Outputs:

Scholarships for Low-Income LSE students $10,699,312

Consultancies for strengthening scholarship system’s M&E $ 339,394

Total Outputs Sub-Component 2c $11,038,706

Subcomponent 3a. Consolidation and Strengthening of School-Based Management for

Education Access and Quality

School based management was originally planned to use US$16.2 million to (i) improve

(subnational) department offices monitoring of the quality of education (supervision tasks) and

(ii) use resources as well as training for principals, teachers, and secondary school governments.

Much of the US$900,000 originally allocated for item (ii) was expected to go towards training

parent-led school governments and to finance projects chosen by them. Allocation to this sub-

component was continuously revised downwards with every new restructuring process. At

Project closure, the government provided some form of training to 1,489 school governments

and spent US$ 245,000 on 158 school government-led projects. However, meetings with

government staff and school government representatives showed little evidence that the training

had any long-term effect on how school governments actually operate.

Outputs:

Training of LSE school councils and communities (incl. materials) $ 371,183

Consultancies for supporting legalization of school councils $ 298,886

Transfers to LSE school councils $ 245,347

Total Outputs Sub-Component 3a $ 915,416

Subcomponent 3b. Strengthening of MINEDUC’s capacity to plan, manage, and supervise

pedagogical, administrative, and fiduciary processes.

This subcomponent was reduced from US$4.8 to US$2.8 million over multiple restructurings.

The subcomponent provided general support to five different directorates in MINEDUC for

activities that broadly strengthened the education system. It supported:

DIGEMOCA, which conducts school evaluations and provides detailed reporting;

DIGEDUCA, which conducted the student achievement examinations used to measure

student learning outcomes in flexible modalities;

33

DIPLAN, which created database system used to track student-level and school-level

information that was critical for monitoring the project;

DIGEPSA, which designed the training materials for the School Management training;

the Directorate of Procurement and Contracts (DIDECO), which manages contracts and

purchasing; and

the General Directorate of Coordination (DIGECOR), which manages the

decentralization of activities to local and regional offices.

It is unclear whether any of the funding was instrumental in ensuring these divisions within

MINEDUC completed their activities successfully, but many of their activities were critical to

the success – and particularly the results monitoring – of education in Guatemala.

Outputs:

Several consultancies (including PIU staff) $ 1,627,924

Equipment and office materials for MINEDUC line units $ 769,286

Project’s Annual External Audits $ 204,771

Other operating expenses for MINEDUC line units $ 48,824

Training to MINEDUC staff (several line units) $ 30,887

Total Outputs Sub-Component 3b $ 2,681,692

34

Annex 3. Economic and Financial Analysis

The cost benefit analysis estimates that the studied interventions result in a net benefit of

US$ 499 million, and an economic rate of return of 21 percent. It yields a cost benefit coefficient

of 2.0. The large net benefit is primarily caused by increased earnings by students who graduate

secondary school. Starting in the year 2013, the project yields a positive net benefit.54

Table A3.1 Net Present Value (NPV) Analysis

(in current US$)

Year NPV Costs NPV Benefits Net Benefit

2008 $19,316,641 $0 -$19,316,641

2009 $55,315,658 $0 -$55,315,658

2010 $67,688,944 $4,026,603 -$63,662,341

2011 $73,708,549 $30,722,082 -$42,986,468

2012 $77,381,865 $54,344,895 -$23,036,970

2013 $70,504,115 $69,088,541 -$1,415,574

2014 $61,645,501 $80,971,754 $19,326,253

2015 $54,404,693 $78,917,428 $24,512,735

2016 $24,750,747 $80,403,883 $55,653,136

2017 $12,742,489 $88,540,339 $75,797,850

2018 $0 $79,717,212 $79,717,212

2019 $0 $71,759,322 $71,759,322

2020 $0 $64,583,390 $64,583,390

2021 $0 $58,125,051 $58,125,051

2022 $0 $52,312,546 $52,312,546

2023 $0 $47,081,291 $47,081,291

2024 $0 $48,229,912 $48,229,912

2025 $0 $43,406,921 $43,406,921

2026 $0 $34,322,261 $34,322,261

2027 $0 $30,190,754 $30,190,754

Total $517,459,202 $1,016,744,185 $499,284,983

The analysis includes the costs and benefits of three interventions and comprise US$70 million

of the US$80 of the loan’s spending, as well as US$114 million in salary expenditures from the

government.55

The three key interventions are:

54 The analysis uses 2007 as the base year, which was the year used in the original economic and financial analysis, and the year

it was approved by the World Bank Board of Directors. However, only costs and benefits were measured from project

effectiveness in 2008.

55 $10 million worth of interventions could not be quantified in an economic and financial analysis, and are simply described at

the end of the section.

35

PEPS. Increased probability of student primary completion in schools with PEPS

programs was compared with that of students in schools without PEPS programs and its

economic benefit was estimated.

Scholarships. Completion of secondary school will be compared between scholarship

recipients and applicants who were not selected to receive the scholarship. The benefits

of increased earnings were estimated.

Flexible Modalities in Secondary Education. The program supported two types of

schools: Telesecundarias, which rely on technology-supported content and teachers

without a specialization, and Núcleos Familiares Educativos para el Desarrollo

(NUFEDs), which provide a non-traditional schedule and focus. The analysis measured

the economic benefit of increased completion rates.

Table A3.2. Costs and Benefits Included in Analysis

Project Benefits Project Costs

Primary Accelerated Education Program

Savings due to reduced primary grade

repetition

Increased earnings due to primary

graduation

Increased earnings due to lower

secondary graduation

Training of facilitators

Staff salaries

Education materials

Scholarships

Savings due to reduced repetition

Increased earnings due to increased

lower secondary graduation

Scholarships

Administrative costs

Flexible Modalities

Increased earnings due to increased

lower secondary graduation

Education materials design &

publication

Equipment for modalities

Training of facilitators

Facilitator salaries

Foregone student income

The two primary benefits are increased beneficiary income due to increased primary and

secondary graduation, and reduced education costs due to lower repetition rates.

The analysis included investment costs and most recurrent costs. The investment costs included

curricular reform, textbook publication, equipment, and training programs, and were funded

primarily by the loan. The primary recurring cost was teaching staff salaries, which were funded

by a mix of loan funds and the government revenues. Although some ongoing costs (such as

many school utilities, facility maintenance, and supplies) could not be included, by including

foregone income, teacher salaries, and materials required for the expansion, most costs are

included in the analysis.

36

Primary accelerated education program

The primary education program was designed to increase retention and completion of overage

students, and had the secondary effect of freeing up resources in schools for non-PEPS students.

The primary costs of the program include the facilitators of the classes, their training, some

limited materials, and, in limited cases, construction of classrooms. All financial costs were

absorbed by the Project funds. The benefits are measured as reduced costs per student due to

increased promotion rates, and increased income due to increased completion. Cost estimates are

based on the government reporting at the end of the project. The PEPS program’s financial cost

is equivalent of US$1.1 million.

Design

To estimate reduced costs due to higher efficiency, promotion rates of PEPS schools were

compared to non-PEPS schools in the same municipalities. Promotion rates increased 2.8

percentage points in control primary schools and 3.7 percentage points in PEPS schools per year.

The difference of 0.9 percentage points was applied to the entire life of the project suggests

2,433 fewer repeating students over the life of the project.

The difference in change in promotion rates used to estimate the number of times students did

not repeat due to the program. This number was in turn multiplied by the average amount spent

per primary student per year. Increased benefits due to more graduates was estimated by

comparing completion rates in PEPS schools and non-PEPS schools in the targeted

municipalities.

Increased private income due to higher graduation rates were measured by comparing pass rates

in PEPS schools and non-PEPS schools, and extrapolating to calculate graduation rates. Tor the

earliest and latest years with data, and the difference between the growths of completion rates

were compared. The difference was used to estimate the number of students in PEPS schools

who otherwise would not have graduated. Given the intended benefit is to increase secondary

completion, and because of the difficulty of estimating the economic benefit of primary

education, the economic benefit was based on the expected number of students who completed

secondary education after completing primary. This was based on multiplying the secondary

completion rate by 603 of students who would not have completed primary without PEPS, to

yield an estimated 320 of secondary graduates due to the PEPS program, and what their benefits

would be from student graduation until 2027.

Increased earnings of 9th grade graduates were calculated based on the increased

earnings of a worker with a 6th grade education.

Increased earnings of 6th

grade graduates were estimated as one-third the benefit received

from lower secondary school.

Decreased costs due to lower repetition rates were calculated based on estimated

repetition rates reduction and the average government expenditures per primary student.56

56 Government expenditures per primary student were calculated as part of the World Bank’s Social Sector Expenditure and

Institutional Review for Guatemala (see World Bank, forthcoming.b).

37

Actual costs of were based on the Government’s final report on loan expenditures. The

intervention’s costs were entirely paid for out of the loan.

Student income was not included given that students are at or only slightly above primary age.

Results

When adjusting for net present value, the project had a net benefit of US$1.4 million, or a rate of

return of 23 percent. Less than 1 percent of the benefit is due to government savings, and 99

percent is due to increased income due to primary and lower secondary graduation.

Table A3.3. PEPS Cost Benefit Results

Net Benefits (US$) $1,476,603

NPV Costs $681,426

NPV Benefits $2,158,029

Rate of Return: 23%

Scholarships

The scholarship program provided 2,500 quetzales (approximately US$330) per year to lower

secondary students in priority municipalities. Recipients who received the scholarship and passed

were guaranteed the scholarship for the following year. The intervention targeted low-income

students and was meant to defray direct school costs and the opportunity cost of working

students. The intervention provided the equivalent of 25,774 one-year scholarships (or 8,591

three-year scholarships.

Design

Costs were measured as the Project’s actual reported costs for the intervention. Two benefits

were estimated: decreased costs due to lower repetition rates from scholarship recipients, and

private economic returns received by the secondary school graduates.

To establish the effects of the scholarship, applicants who did not receive the scholarships serve

as an excellent counterfactual. Starting in 2013, recipients for the scholarship were stratified

geographically using a formula. The number of scholarships each municipality received was

determined by municipal poverty rates and repetition rates. Municipalities then determined the

number of scholarships provided to each school. A group of teachers and the director of each

school then selected several students to apply for the vacancies: on average, three students were

selected to apply for every two vacancies. Municipal or departmental boards would then select

which applicants would receive the allotted scholarship based on their need and grades.

Applicants who did not receive the scholarship were 1) from the same municipalities as

recipients, and 2) selected based on their grades and need. To estimate the benefits of

scholarships, grade promotion of scholarship recipients and non-recipients were compared by

tracking them using the Ministry of Education’s student database. Although this selection

formula for beneficiaries was only used since 2013, the same benefit estimates were applied for

beneficiaries in 2010-2012, when selection processes were less transparent. If past selection

methods were less efficient, this method may overestimate scholarship benefits.

Results

38

In 2013 and 2014, the average promotion rate for a scholarship recipient was 9.7 percentage

points higher than applicants who did not receive a scholarship (Table A.3.4). Using grade

promotion to extrapolate completion of all three years of lower secondary, the graduation rate for

scholarship recipients was 20.9 percentage points higher than non-recipients.57

Due to lower repetition rates, classrooms had 3,412 additional vacancies over the course of the

project, saving the government US$1.1 million over the course of the project. Also, the estimated

additional 2,465 lower secondary graduates will increase their private incomes a total of US$82

million over the course of three decades. When adjusting for net present values, the scholarship

intervention had a net benefit of US$17.8 million dollars and an internal rate of return of 44

percent (Table A.3.5).

Table A3.5. Scholarship Net Benefit Analysis

Net Benefits (US$) $17,766,894

NPV Costs $5,709,611

NPV Benefits $23,476,505

Rate of Return: 44%

Flexible Modalities

The Project sought to strengthen and expand them to improve access to a quality education,

particularly in rural areas58

. The two flexible modalities comprised the majority of the spending

of the loan: NUFED funding comprised 3.6 percent of total loan spending, and Telesecundaria

funding comprised 68 percent. In addition to loan funds, the Government’s other funding sources

funded the majority of the interventions’ education staff. To properly evaluate the project, both

loan funds and the Government’s funding of Telesecundaria and NUFED teaching staff were

included in the analysis.

Design

Financial costs for strengthening activities were covered by the loan, and actual reported values

were included in the analysis. Costs for expansion included the increase in the number of

teaching staff (facilitators) since 2007. Intervention costs included teacher salaries for all years of

the Project, which were covered by the loan for three years and by other government resources in

the remaining years. Although actual facilitator salary data was unavailable for Telesecundaria

57 This value is discussed in the main text as recipients being over 40 percent more likely to graduate than comparable non-

recipieints. This is based on the calculation 72.4 / 51.5 = 1.41.

58 Although the original documents mention strengthening, expanding, and evaluating the project, funding for evaluations was

eliminated in the first restructuring, before most implementation activities had begun.

Table A3.4. Student Outcomes of Scholarship Applicants

Received Scholarship No Scholarship

Promotion Rate 89.8% 80.1%

Graduation Rate 72.4% 51.5%

39

and NUFED schools, salaries were estimated based on the teacher salary schedule for each year

of the project. The actual number of Telesecundaria facilitators was available for each year.

Although the exact number of NUFED facilitators was unavailable, they were calculated using

the average pupil-facilitator ratio in NUFED schools and the yearly number of NUFED students.

The benefit of the interventions were measured as the increased income of the additional lower

secondary graduates over the course of the project. In addition, the foregone income of enrolled

students was included as a cost.

The benefits analysis assumes that NUFED and Telesecundaria schools would have continued

operating in a smaller capacity if the Project had not existed. The number of beneficiaries was

based on the number of additional graduates each year compared to before the expansion began

(2009). The value of the benefit was measured as the increased income until 2027 of a lower

secondary school graduate over that of a primary school graduate. No other benefits were

measured.

Results

The financial cost of the flexible modalities totaled US$172.8 million over the course of the

project. The loan supported one-third of the costs, and other Government funds supported the

remainder. Telesecundaria comprised 90 percent of the total financial cost, and NUFED the

remaining 10 percent. Based on the change in enrollment since the beginning of the project, the

expansion lead to an additional 91,060 students graduating from Telesecundaria schools and an

additional 15,053 graduating from NUFED schools. After adjusting for net present values, the

net benefit of the flexible modalities was US$868 million, resulting in a rate of return of 48

percent.

Table A3.6. Flexible Modalities Cost Benefit Analysis Results

(US$ millions)

Telesecundaria NUFED Combined

Net Benefits 771 97 868

NPV Costs 90 31 121

NPV Benefits 862 129 991

Benefit-Cost

Ratio 9.6 4.16 8.2

Rate of Return: 48% 42% 48%

Methodology and Parameters

The analysis included the actual costs of the three interventions from the project and estimations

of the largest cost not covered by the loan: teaching staff salaries. Accelerated primary and

scholarships were created at the beginning of the project, while the flexible modalities existed

before the project on a much smaller scale. The costs and benefits of the scholarship and

accelerated primary interventions were entirely attributed to the project. For flexible modalities,

the analysis only included additional costs and benefits: the costs of new materials, trainings, and

additional teachers required, and the increased number of graduates from the Telesecundaria and

NUFED schools.

40

The default parameters were adopted from the economic analysis conducted during appraisal.

The default discount rate used was 10 percent; the exchange rate remains 7.63 quetzales per US

dollar.

To estimate the income of students that do not enroll, the probability of youth dropouts finding

employment59

multiplied by the average income of primary completers under the age of 35

(based on the 2014 Household Survey ENCOVI).

To estimate the marginal income of students that graduate, the difference between estimated

average income of lower secondary graduates and average income of primary completers was

calculated. Although only the income benefit of primary and secondary completers was

available, 60 percent of the difference was used (to account for students only attending three

years of lower secondary, as opposed to five years of secondary).

As with the initial analysis, the costs and benefits were measured between the years 2007 and

2027, and the base year for net present values remained 2007. Although all costs were completed

by the end of the project, the benefits extended over 30 years due to increased income.

Sensitivity Analysis

The above results were tested in three scenarios to determine the stability of the results. In each

of the three scenarios, the net benefit and rate of return diminished but remained positive and

high.

Reduced graduation benefit

In case the estimates of benefits are overly optimistic, the cost-benefit analysis above was

modified to estimate that the income benefit are half of the estimation. In this scenario, the

project still has a positive net benefit of US$3.3 million, a rate of return of 20 percent and a

benefit to cost ratio of 1.84.

This scenario is important because the highly positive results of the original analysis are

primarily due the estimated increased income lower secondary graduates received. The economic

returns were based on actual income differences between lower secondary and primary graduates

published in a 2006 paper. However, if the quality of education is lower in the flexible modality

schools compared to that of graduates at that time, or the market for skills changed substantially,

the benefit could have been much lower.

Other Interventions

Several interventions lacked data necessary to measure their economic benefits or their benefits

were impossible to measure. For this reason, they were excluded from the analysis.

Pre-Service Teacher Training

The Project supported the PADEP/D Program to improve education quality in primary schools. It

made use of US$7.6 million of loan funds (in addition to other resources) to provide two years of

59 See De Hoyos et al (2016).

41

training to 25,490 pre-primary and primary teachers and began the training of another 4,000. The

Ministry of Education conducted surveys of graduates to assess their satisfaction and altered

perceptions of teaching methods from the training program, which suggest high satisfaction and

greater appreciation for suggested teaching methods. The Project supported the intervention in

order to improve primary completion rates (Component 1), to increase the stock of students able

to attend lower secondary. However, with the data available, there is no way to assess the effect

the program has had on what takes place in the classroom or on learning. It should be noted that

in 2006, Guatemalan third and sixth grade primary students scored among the lowest countries

on the Second Regional Comparative and Explanatory Study (SERCE), and scored near the

regional averages in 2013 on the third iteration of the study. This narrative of quality illustrates

the dilemma between improving access and quality: primary education access peaked in 2009,

and quality has apparently improved since then; secondary education access expanded during

that time, limiting their ability to improve quality.

School-level Management

Per project design, improving school management focused on two purposes: improving

(subnational) department offices monitor and use resources as well as training for principals,

teachers, and secondary school governments. Much of the US$900,000 originally allocated was

expected to go towards training parent-lead school governments and to finance projects chosen

by them. At close of project, the government provided some form of training to 1,489 school

governments and spent US$ 245,000 on 158 school government-led projects. However, meetings

with government staff and school government representatives showed little evidence that the

training had any long-term effect on how school governments operate.

Institutional Strengthening

The government used US$ 2.7 million for institutional strengthening and studies. The funding

allowed the government’s research arm to publish numerous reports, primarily qualitative in

nature, including those monitoring teacher satisfaction with PADEP/D training, estimating costs

per PEPS enrollee, comparisons between the secondary education modalities, evaluation of the

school library program, and other national programs. These reports allow the Ministry of

Education to attempt to inform on the efficacy of its programs. In addition, the institutional

strengthening supported the Ministry of Education’s planning department, which now has an

excellent education management information system that made the cost-benefit analysis possible,

and has great potential for further research.

42

Annex 4. Bank Lending and Implementation Support/Supervision

Processes

(a) Task Team members

Names Title Unit Responsibility/Specialty

Lending

Joel E. Reyes Senior Institutional

Development Specialist GEDGE Task Team Leader

Juan Manuel Moreno

Olmedilla

Lead Education

Specialist GED05 Senior Education Specialist

Martha Laverde Education Specialist

Angela Demas Senior Education

Specialist GEDGE Operations Officer

Christel Vermeersch Senior Economist GHN04 Impact Evaluation

Specialist

Henry Forero Ramirez Senior Public Information

Specialist GGO16 Senior Information Officer

Manuel Vargas

Madrigal

Lead Financial

Management Specialist GGO23

Senior Financial

Management Specialist

Xiomara A. Morel Lead Financial


Financial Management

Specialist

Fabienne Mroczka Financial Management

Analyst GGO22


Analyst

Keisgner Alfaro Senior Procurement

Specialist

Luis Prada Villalobos Senior Procurement

Specialist GGO05 Procurement Specialist

Patricia E. Macgowan Procurement Specialist

Monica Lehnoff Procurement Specialist GGO04 Procurement Analyst

Luis Prada Villalobos Senior Procurement


Laura B. Rawlings Lead Social Protection

Specialist GSP01 Sector Leader

Supervision/ICR

Joel E. Reyes Sr. Institutional

Development Specialist GEDGE Task Team Leader

Christel Vermeersch Senior Economist GHN04 Task Team Leader

43

Marcelo Becerra Lead Education

Specialist GED04

Task Team Leader

Michael Drabble Senior Education

Specialist GED04 Task Team Leader

Hongyu Yang Senior Education

Specialist GED04

Task Team Leader

Juan Diego Alonso Senior Economist GED04 Task Team Leader

Barbara Bruns Lead Education Specialist

Andrea C. Guedes Senior Operations

Officer GED03 Senior Operations Officer

Martha Laverde Education Specialist

Carlos Tomas Perez-

Brito

Senior Social

Development Specialist GSU04 Operations Officer

Nancy Banegas

Raudales Operations Officer GSP04

Operations Officer

Keisgner Alfaro Senior Procurement

Specialist

Alvaro Larrea Lead Procurement


Monica Lehnoff Procurement Specialist GGO04 Procurement Specialist

Antonio Leonardo

Blasco Senior Financial


Senior Financial Management

Specialist

Jose Simon Rezk Senior Financial



Specialist

Eduardo França Financial Management

Specialist GGO22


Specialist

Ximena B. Traa-

Valarezo Consultant GSU04

Senior Social Safeguards

Specialist

Sajitha Bashir Practice Manager GEDO1 Sector Leader

Tatiana A.

Proskuryakova Country Manager ECCBM Country Operations Adviser

Janet K. Entwistle Representative AFMLS Senior Operations Officer

Wendy Elizabeth de

Leon Samayoa Consultant

ICR Author

Andrew Paul Trembley Consultant ICR Author

44

(b) Staff Time and Cost

Stage of Project Cycle

Staff Time and Cost (Bank Budget Only)

No. of staff weeks USD Thousands (including

travel and consultant costs)

Lending

FY05 14.45 95.86

FY06 19.27 136.35

FY07 12.59 64.14

Total: 46.31 296.35

Supervision/ICR

FY07 2.03 13.64

FY08 15.33 105.08

FY09 33.88 118.84

FY10 53.61 198.81

FY11 27.88 119.19

FY12 11.51 76.47

FY13 18.69 80.51

FY14 20.00 82.52

FY15 22.20 36.34

FY16 32.72 117.94

Total: 237.85 949.34

45

Annex 5. Beneficiary Survey Results

No beneficiary survey was conducted.

46

Annex 6. Stakeholder Workshop Report

No Stakeholder Workshop was conducted.

47

Annex 7. Summary of Borrower's ICR and Comments from the

Borrower to Draft ICR

7.1 BORROWER’S ICR60

INTRODUCTION

This document is the closing report of the "Education Quality and Secondary Education Project",

which was financed with the World Bank-International Bank for Reconstruction and

Development (IBRD)-7430-GU loan, executed according to the loan agreement signed between

the World Bank and the Republic of Guatemala.

The inputs for this document come from the digital and physical files of the project, stored

within the premises of the Ministry of Education. Relevant information was used to compile,

summarize and analyze the IBRD 7430-GU Project from conception and implementation to

closure and evaluation.

This report contains the closure of components that were developed during the life of the Project

IBRD 7430-GU, executed by various units of the Ministry of Education.

A compilation and updated evaluation of the implementation of the project is also included,

taking into account compliance with the objectives originally proposed, compliance with the

commitments will benefit entities fulfilling the commitments of the World Bank, the lessons

learned is, the recommendations made and the sustainability of project interventions.

I. BACKGROUND

The Education Quality and Secondary Education Project was an initiative of the Government of

Guatemala with the support of the International Bank for Reconstruction and Development -

IBRD- aimed at improving access to quality basic secondary level education for low-income

students, specially students from indigenous communities, through improving primary

completion rates of overage students, strengthening flexible modalities of lower secondary

education and school management. For this purpose, the Loan Agreement Number IBRD-7430-

GU was approved by the Congress of the Republic of Guatemala on February 13, 2008 and

signed by the Government on April 12, 2008.

Guatemala, with support from the World Bank, implemented the first national project aimed at

improving access to quality basic secondary education, which contributed to improving the

completion rate in 9th grade in 196 targeted municipalities, as well as an increase in enrollment

in lower secondary education, although in the last three years of the project these indicators

60 The following is an unofficial translation into English of the Borrower’s Report (MINEDUC 2016e) submitted to the Bank on

December 14, 2015. It was slightly abridged to meet ICR Annex guidelines.

48

suffer a break caused by the lack of updated population projections which have not been updated

in 11 years in the absence of a population census. However, these figures are currently being

reviewed by the National Statistics Institute to correct historical series to a growing and positive

trend.

In 2008 the Government of Guatemala signed a Loan Agreement with the IBRD-WB worth

eighty million dollars ($ 80,000,000) for the implementation of the project which would start on

06/23/2008 and initially close on 12/13/2013, but was extended until 11/30/2015 by an

amendment to the Loan Agreement. Disbursements executed in US dollars for each year were

made as follows:

2008 2009 2010 2011 2012 2013 2014 2015

Annual

Disbursements $7.2 $0.0 $13.8 $14.5 $10.5 $17.0 $11.9 $5.1

Cumulative

disbursements $7.2 $7.2 $21.0 $35.5 $46.0 $63.0 $74.9 $80.0

The project is divided into three components, each with specific objectives to be developed over

the life of the project, as follows: a) Primary Education Completion and Quality; b) Access and

Quality of Lower Secondary Education; and c) School Management in Support of Education

Quality.

II. AMENDMENTS AND RESTRUCTURING OF THE PROJECT

The project had 4 amendments and 1 restructuring. The 1st Amendment was signed on June 15,

2009 with the main purpose of including the financing of Telesecundaria Facilitators as a

separate expenditure category and simplify the disbursement table, among other minor changes.

The 2nd

Amendment was signed on March 17, 2010 with the main purpose of aligning the project

design with the objectives and strategies relevant to the Guatemalan educational sector at that

moment and make the project design more flexible regarding its implementation methods,

maintaining the same development objectives of the project.

On June 6, 2011 a restructuring of the Loan Agreement was signed in order to: a) provide

additional funding for Telesecundaria Facilitators and increase the financing threshold for the

Scholarship Program; b) change the targets of some indicators in consistency with the changes

indicated in bullet a) and revise the wording and targets of the rest of indicators; and c) reallocate

resources to support the changes.

The 3rd

Amendment was signed on May 21, 2012 with the main purpose of reallocating

resources between disbursement categories and modify the percentages of financing in the

disbursement table of the Loan Agreement.

The 4th

Amendment was signed on December 19, 2013 with the main purpose of making

changes in the scope of project activities in accordance with the strategies of the MINEDUC,

expanding funding for PADEP/D, reviewing indicators and targets included in the Results

Framework and extending the closing date to November 30, 2015, among other minor changes.

49

III. RESULTS OF PROJECT INTERVENTIONS

3.1 Accelerated Primary:

Under the original design -even though it was only a temporary intervention- this intervention

met the following objectives: a) supported students who were 3 or more years over-age and at

risk of dropping out; b) the Ministry of Education normed relevant educational reforms and

assigned specific environments for this type of students; c) the Ministry of Education designed,

printed and distributed educational materials for a 2 year accelerated primary program, taught in

the same school of the community; d) maintained an ongoing training program during the life of

the project on the methodology of the program for different stakeholders; e) strengthened reading

skills through the provision of books to all centers in operation; f) a flexible promotion system

was put in place to ensure student retention and transition to secondary education.

This intervention included granting targeted schools a financial incentive for renovation

improvements or repairs of the schools. This objective was not developed given that resources

from other donors funded this activity in primary schools. The project focused resources only in

the lower secondary level. This intervention was established in the Ministry of Education due to

the Project. The dropout and repetition rates in primary school remain high and as a result not all

students are able to complete primary education or finish late.

In addition to other causes such as migration61

, late enrollment, repetition, dropout and the poor

quality of education, child labor is one of the main causes of over-age in schools, which mainly

affect rural population and indigenous peoples. In this context, aware that complying to the right

to education is a measure that contributes to the fundamental rights of equality, development and

peace for the people; and given the repetition and dropout rates in the country, the Ministry of

Education agreed to create a program that would address the situation of overage students of

Guatemalan children at the primary level. In 2009 with resources from the IBRD Loan 7430-GU

a Program for Primary Education Completion for Overage Students -PEPS- was created. This

action was the first of its kind within the Ministry of Education to address the problem of over-

age in primary education in Guatemala, with "accelerated education".

The PEPS functioned in 7 departments (Huehuetenango, Jutiapa, Quetzaltenango, Quiché, San

Marcos, Sololá and Totonicapán). The program has benefited children who: a) have promoted

first grade of primary; and b) have 2 or more years above the theoretical age for their grade. The

program operates in two stages: 1st stage (2

nd and 3

rd grades of primary) stage I lasts two years;

the 2nd

stage (4th

, 5th

and 6th

grades) lasts one year. Children graduate from the program with

complete primary education to eventually gain access to lower secondary education in the

following year. The PEPS served a population of 30,865 students with overage (on average

5,144 students per year) during the years 2009, 2010, 2011, 2012, 2014 and 2015, allowing them

to complete the primary level of education.

Textbooks, notebooks and work modules were designed and printed, in line with the National

Curriculum and teachers were trained in the methodology of attention to this sector. At its peak,

61 Buettner and Muenz (2016) document the status of international trends for migration for a series of countries, including

Guatemala.

50

350 bilingual and 75 monolingual schools offered the PEPS program, also receiving reading

books to strengthen their reading skills. More than 300 directors and 81 supervisors were trained

in the 7 departments that have implemented the program.

Since 2009 with the support of the General Directorate for Educational Assessment and Research

- DIGEDUCA- information about the execution, implementation, evaluation and results of the

PEPS has been recorded. The reports that were generated will provide inputs for the new

government education authorities which will assume office in January of 2016, which will enable

improved decision making to further strengthen the achievement of results in the next stages of

implementation of the PEPS or for future programs which also seek to address the overage

problem at the primary level.

The PEPS program has not solved the overage problem which, according to DIPLAN data, was

of over 700,000 enrolled overage students in 2009. The main objective of this program, which

can be categorized as a "pilot" program, was to make the overage problem in the educational

system visible and it should continue to be addressed in the near future. The project -with an

investment of 8.6 million quetzales (approximately US$ 1.13 million) - leaves lessons learned,

educational materials, research and also students who are now part of the regular lower

secondary education level.

3.2 Pre-Service Teacher Training and In-Service Teacher Training –PADEP/D-

The objective of strengthening five regional pre-service teacher training centers with updated

training curricula, with specific pedagogical approaches to suit flexible modalities, rural areas

and intercultural bilingual education was not met. The second amendment abandoned this

objective and instead states that "the project will support the development of a new strategy for

pre-service teacher training."

When the project started its implementation, the future of the Normal Schools was already being

discussed in a broadly representative Technical Committee, so the Ministry of Education and the

World Bank decided not to intervene according to plan and agreed to designate resources to

provide technical assistance to the Technical Committee. However, this to results unfeasible

given the dichotomy of the discussions that take place in a political-technical level.

In the end, the resources of this intervention are allocated to provide educational supplies,

computers and furniture to normal monolingual and bilingual schools. Today Normal Schools no

longer graduate primary level teachers, but only Bachelors and pre-primary level teachers. The

second amendment incorporates in-service teacher training at the primary level –PADEP/D- and

the fourth amendment expands both the allocated resources and also the goals. The Project

invested over 58.0 million quetzales (approximately US$ 7.60 million) from 2012 to 2015,

providing scholarships to more than 13,700 students of the 1st, 2nd, 3rd and 4th Cohort.

The PADEP/D program is a training program for all public school teachers of primary and pre-

primary levels. The program is delivered in centers established at the municipal level. Mostly in

schools that meet criteria for use and, in general, in the capital of the municipality. The program

is voluntary and encompasses a total of 4 semesters. This program has nationwide presence and

has operated since the year 2009.

51

The PADEP/D has graduated 3 cohorts of 2,543 (cohort 2009-2011); 3,155 (cohort 2010-2012)

and 6,100 (cohort 2012-2014) teachers, respectively. These teachers come from a total of 104

municipalities, 94 of which are targeted municipalities that correspond to the Project

intervention. The 4th

cohort is studying its second year, involving a total of 6,122 teachers and

100 municipalities, 41 of which are targeted. The project has funded teaching cohorts 1, 2, 3 and

4.

DIGEDUCA has completed its evaluation of the second cohort of PADEP/D, and is currently in

the process of carrying out the third performance evaluation of PADEP/D teachers and the

performance of students in the third and fourth cohort. The results of these evaluations will be

released in 2016. The project exceeded its original goal of training a total of 2,000 teachers

trained by a higher education institution.

Through an agreement signed between the USAC and MINEDUC, the Project will meet the

objectives of developing a new strategy for in-service teacher training programs, designing a new

study program and designing materials, education courses and specialized subject areas

requested by regional centers.

3.3 Telesecundaria and NUFEDs

Through these forms of educational delivery the Project originally set out to strengthen and

expand existing modalities that existed for lower secondary education (grades 7-9) for rural and

indigenous communities and also to increase access to education for some 90,000 new students,

using the existing school capacity and infrastructure.

Through Telesecundaria, more than 100,000 new student seats were incorporated into the system

allowing to surpass the original target.

The project initially envisaged that the planned expansion of coverage would be through:

Telesecundaria, NUFED Institutes, Cooperative Institutes, INEBs and strategic partnerships with

nonprofit educational providers such as NGOs and Foundations. With the second amendment the

Ministry of Education and the Bank decided to focus on expanding coverage only through

Telesecundaria.

The project strengthened 967 Telesecundaria institutes and 5,080 sections, with a total

investment of more than 426.0 million quetzales (approximately US$ 55.8 million) used on

furniture and computer equipment, audiovisual, reading books, teacher training, educational

materials and recruitment of facilitators. From 2009-2012, 1,414 Telesecundaria facilitators were

hired with an investment of 237.5 million quetzales (approximately US$ 31.1 million). Starting

in 2013, the Government absorbed all costs of facilitators, ensuring the sustainability of the

coverage.

The review, alignment, design and adaptation of the Telesecundaria modality to the Standard

National Curriculum (CNB) was performed for the areas of biology, physics and chemistry;

social studies; foreign language (English); Communication and language; technologic education;

math; and artistic expression. The Project printed and distributed 5,000 documents of the

52

Telesecundaria National Base Curriculum; 4,500 manuals for facilitators in each area and 26,500

learning guides for students in 4 of the 5 originally proposed curriculum areas (Math, English,

Artistic Expression, Science and Communication and Language).

600 NUFEDs were strengthened with an unprecedented investment of up to 22.0 million

quetzales (approximately US$ 2.9 million), used on furniture and computer equipment, reading

books, textbooks for 7th

, 8th

and 9th

grade math and Communication Language, work productivity

modules, educational materials relevant to the modality, teacher training and the adaptation of

the modality to the National Base Curriculum for all three grades of the basic secondary

education.

3.4 Scholarships to increase the educational demand for low-income students

The scholarship program was designed to cover the direct family costs and the opportunity costs

of basic secondary education, especially for rural low-income families and young workers. In

order to improve the beneficiary selection system, a unique code was used to avoid duplication

of grants or exclusion errors.

The program projected that 7,280 poor students from indigenous communities would get

scholarships to complete three years of secondary education. The scholarships were conditioned

to the promotion and completion of the entire educational cycle. In 2011 by restructuring the

Project, 28,000 additional scholarships were added for a new overall target of 35,280

scholarships.

The program had two moments: 5,347 scholarships for the 7th grade cohort in 2010 and 5,275

for grades 7th and 8th in 2011 were granted. In 2012 the program was suspended in order to

review results, access requirements, criteria and mechanisms of control and supervision.

In May of 2013, the Ministry of Education re-launched a revised scholarship program,

introducing more transparent criteria for targeting and greater decentralization in monitoring

retention, learning achievements of students and distribution of the scholarships. This new

program measures the number of scholarships financed by the project that are awarded to eligible

students of targeted municipalities that comply with assistance requirements, financial need and

academic performance. Scholarships reached 96 percent of execution at closing and an

investment of more than 81.0 million quetzales (approximately US$10.6 million).

3.5 School Based Management for Quality of Education

With this intervention, the project set out to strengthen the school and parent councils in basic

secondary education by providing tools and training to these organizations in order for them to

plan activities for school improvement.

The original design of the project stated that Education Councils could prepare a school

improvement project, and even if it was not fully funded by the Project they would receive

US$680.00 per year, up to five times during the period of implementation of the school project.

1,348 councils were created, formed, trained and legalized during the life of the Project, of which

156 received funding for quick-impact projects, in the amount of Q12, 000 per council and these

councils submitted expense reports satisfactorily.

53

The goal of trained councils reached 99.8 percent. The process of legalization of educational

councils involves eight steps and is performed through several directions of MINEDUC, with the

establishment of the receiving entity with the Ministry of Finance the most complex step. The

target of financial transfer to 487 councils, was not reached and reached 32 percent. This sub-

component supports interventions for community empowerment through school-based

management. The formation of educational councils allows decentralizing decision making.

3.6 Capacity Building of the Ministry of Education to plan, manage and supervise

Pedagogical, Administrative and Financial Processes.

This intervention was initially intended to strengthen the capacity of 10 Departmental Directions

of Education to improve the quality of their supervision of the educational and administrative

process in targeted schools within the Project. Through this intervention the DIDEDUCs would

support the Telesecundarias, NUFEDS and school councils with strategies, tools and improved

pedagogical skills.

The Project also planned improvements to the school supervision system, through training,

materials, technical assistance and systems to improve the processes of human resource

management that have been decentralized to regional level.

This strategy was modified with the second amendment via 2 new objectives: a) improve the

capabilities of the Ministry of Education to plan, implement, monitor and evaluate in educational

terms and in terms of management; b) improve MINEDUC fiduciary capacities to adequately

coordinate and implement the Project.

The MINEDUC complied with the commitments established in the Loan Agreement, Sections

and Conditions to keep an up to date Operations Manual, administrative, financial, accounting,

auditing, procurement and disbursement procedures for Project implementation.

Annual financial audits were carried out throughout the life of the project and results showed

correct and transparent use of resources.

Assigned staff was responsible for Project Management, with specialists in procurement, finance

and contract monitoring in accordance with the Standards approved by the World Bank.

Every six months the MINEDUC presented a "Financial Progress Report" to the Bank and also

annual work plans with their procurement plan.

The Ministry of Education, through the DIGEDUCA, provided up to date information combining

quantitative and qualitative research methodologies, on the various project interventions. This

will enable future authorities and the educational community to make decisions, develop

policies, assess compliance with them and design new strategies, and also contribute to the

improvement of educational practices.

DIGEDUCAs research includes findings made on the implementation, development, evaluation

and results of the educational programs of the Project. It provided inputs to those responsible for

the different programs in the central and departmental offices of the Ministry of Education. This

will enable improved decision making to further strengthen the achievement of the expected

results in the next stages of implementation of current interventions or future educational

programs in the secondary education level.

54

IV. INDIGENOUS PEOPLES PLAN- IPP

This Plan was activated by the strong presence of indigenous communities in the areas of

intervention of the Project. It was designed to be implemented across all project components. It

focused on poor indigenous and non-indigenous populations in the regions, municipalities and

communities mostly inhabited by indigenous peoples with special educational needs.

From 2009 to 2011 the IPP developed within the Ministry of Education a program for the

strategic development of bilingual and intercultural education, generating dialogues, debates and

reflection in the search for a comprehensive approach to the Intercultural Bilingual Education

(IBE).

In 2012 under the Education Strategic Plan of 2012-2016, policy number 5 was established as

Multilingual Multicultural Policy which launched IBE as a Government project, broader then a

project of this Loan. This policy transcends and surpasses the IPP. The Bank appoints a specialist

in indigenous issues that enriches the dialogue and the Ministry of Education hires a specialist in

indigenous issues accompanying the process.

Under the IPP modules and notebooks of Accelerated Primary and educational materials were

printed in Mayan languages and curriculum design for this modality was adapted according to

the National Curriculum. In addition, monolingual and bilingual teachers were trained in the

methodology of the program and use of materials.

4,800 schools were socio-linguistically characterized. The characterization consists in

establishing the profile or the socio-linguistic, linguistic and cultural characteristics of the school

and its community context (Maya, Garifuna, Xinca and Ladino), and also the materials and

human resources that are available; that is, the basic conditions available to develop the

educational process. With the characterization, bilingual education indicators are verified in

qualitative and quantitative terms. Through the characterization, the type of school and

community context where it is located are specified.

The in-service Teacher Training Program trains teachers at the University level for teachers in

Intercultural Primary, Intercultural pre-primary, Bilingual and Intercultural Primary and

Bilingual and Intercultural pre-primary. The training is carried out in regions with a majority of

indigenous peoples and teachers.

The Scholarship Program was developed predominantly for poor indigenous and rural students

from targeted regions by the project. 59 percent of scholarships were given to indigenous

population.

Telesecundaria and NUFED institutes from predominantly indigenous regions received money

transfers by way of education councils.

V. LESSONS LEARNED

Any intervention within the design of a project should have clear, measurable and achievable

budgets while ensuring that the design is consistent in order to obtain best results. The original

55

design of the project collided with difficulties from the beginning that affected the early years of

implementation, such as not taking into account hiring teachers as a necessary condition for the

expansion of access, among other adjustments that were necessary to incorporate through 4

amendments and a restructuring.

The indicators of a project must be carefully analyzed, discussed, agreed upon and reviewed

throughout implementation to avoid ambiguities, inaccuracies or misunderstandings that

eventually hinder the evaluation of a project, which is otherwise successful, but the indicators

end up distorting its impact.

The planning, design and development stages of a project have better prospects of

accomplishment when high and middle level authorities, operational staff and management

boards are involved in the process, which allow consensus considering the different perspectives

and levels of experience in order to come up with the best implementation scenarios.

It was a success to have included an Indigenous Peoples Plan in the project design because it

contributed to regulating public policy regarding bilingual education. The design of future

projects or investments must consider what the laws and regulations for indigenous peoples

dictate and also the content of the National Base Curriculum, which now responds to the

demands and needs of indigenous peoples.

VI. SUSTAINABILITY

Of the interventions supported by the project, the expansion of access that was obtained through

Telesecundarias is fully sustainable at the end of the project. After an investment of more than

50 percent of the loan in this modality, the Government has assumed 100 percent of the

operational and administration costs of this modality.

Family and Community based education -NUFED- overseen by the General Directorate Non-

Formal Education, continue to be sustainable with public resources. The technical, financial and

administrative resources that were given through the project have strengthened and supported

this modality, leaving greater capacity to improve the quality of education in this modality.

Pending matters for analysis, negotiation and monitoring with the next government in order to

allocate resources from the national budget to provide technical, administrative and financial

continuity to important interventions that were supported by the Project include: PEPS,

PADEP/D, LSE scholarship program for low-income students and school management via lower

secondary school councils.

VII. RELATIONSHIP WITH WORLD BANK TEAMS, LOCALLY AND WITH

OFFICES OUTSIDE GUATEMALA

From 2009-2015 different changes of officials in the Government of Guatemala and the World

Bank generated implementation delays that forced the project to have an extended closing date.

During the Alvaro Colom Caballeros administration, from January 2008 to January 2012, they

were 3 Ministers of Education, 3 Bilingual Vice-Ministers, 3 Administrative Vice-Ministers, 4

Technical Vice-Ministers and 3 Vice-Ministers of Education Quality. Likewise, 4 project

56

managers were responsible for the Project within the Ministry of Education. In that same period,

the World Bank had 3 different Project Managers from Washington [DC], USA.

In the Otto Perez Molina Administration, from January 2012 to September 2015 and Alejandro

Maldonado Aguirre Administration, from September to January 2016, there were 3 Ministers of

Education, 1 Bilingual Vice-Minister, 1 Administrative Vice-Minister, 1 Technical Vice-

Minister and 2 Vice-Ministers of Education Quality. In addition, 2 project managers were

responsible within the Ministry of Education. In that same period, the World Bank had 2

managers from Washington, USA and 1 based in Tegucigalpa, Honduras.

The relationship between the Government of Guatemala, represented by the Ministry of

Education and the World Bank under the IBRD Loan 7430-GU, always maintained the protocol

rules of respect, consideration and attention.

At times, discussions and negotiations on the implementation of the Project reached critical

stages which were overcome, especially due to the frequent and permanent changes in officials

(already described above), resulting in partners with different points of views, sometimes not

coinciding with the views of their predecessors.

RECOMMENDATIONS FOR A MORE EFFECTIVE AND TIMELY RELATIONSHIP

Future projects with the World Bank should consider having managers based in the country in

order to have a more fluid, timely and immediate communication, or in Washington, USA so

they can have immediate access to senior levels in order to make quick and timely decisions.

Similarly, the Government in agreement with the World Bank should ensure that the internal

management unit of the loan is not subject changes due to political change, given that the

functions performed are eminently technical and specialized. The above paragraphs can

significantly improve future relationships between the Bank and the Government and achieve

better results.

7.2 COMMENTS FROM THE BORROWER TO DRAFT ICR62

The Government of Guatemala (GoG) would like to clarify that although the term

"restructuring" is the one utilized by the Bank to refer to any significant changes to the Project,

we would rather refer to the number of “amendments” to the Project. According to the GoG’s

records, 4 amendments to the Legal Agreement were processed. In addition, there was an extra

modification to the Project on 06/06/2011, which was not considered as an amendment and that

was approved to cover the payment of 414 TI facilitators in addition to the 1,010 originally

agreed in 2011. This modification also financed an increase of 28,000 scholarships, which was

not considered an amendment as such.

The GoG is pleased to note that the Bank conducted a comprehensive evaluation of the

Project’s implementation over its lifetime, highlighting the amendments that were necessary for

62 This is an unofficial translation into English of the Borrower’s comments to the ICR submitted to the Bank on June 28, 2016.

57

the achievement of the PDO, which, by the way, is worth noting to not having changed at any

time. It is also important to note that the Project benefitted from an implementation model for

Education Projects that the World Bank also used for other countries and which proved to be

very valuable indeed.

There are important lessons learned which are reflected in the report, which seem to be recurrent

in projects that are currently underway, and which should be given due attention for future

Project designs, as follows:

A stylized fact for most of the project portfolio with the World Bank is the need for early

amendments due to delays in the original approval of loans by Congress. These

amendments usually took the form of changes to the original design of the Project due to

the need to adapt to dynamic changes taking place from the identification stage to the

implementation phase.

Changes in government priorities after political transitions.

High turnover rates of technical personnel.

The implementation of education projects requires a simple and stable design that may

allow for the efficient execution of external financing resources, as well as strong

analytical foundations and technical assistance to ensure an adequate implementation. A

core planning tool for good implementation is to have a credible annual disbursement

schedule.

The overall rating assigned to the Project was Moderately Satisfactory, showing significant

achievements for secondary education for Guatemala, through support to TIs and NUFEDs.

Last but not least, we note that the Report rates Bank Performance as Moderately Unsatisfactory.

The GoG agrees with the importance of strengthening monitoring and evaluation systems

throughout implementation, ensuring solid and permanent teams work towards supporting the

Project design and minimizing changes in the results framework.

58

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders

There were no cofinanciers or other partners for this Project.

59

Annex 9. List of Supporting Documents

ACEPT (2015); “Los Desafíos del Cálculo de la Cobertura en el Sistema Educativo: Un Estudio

en Profundidad de la Evolución Reciente de las Tasas Netas de Cobertura en Guatemala

(The Main Challenges in the Calculation of Enrollment Rates in the Educational System:

An In-Depth Study of the Recent Progress in Net Enrollment Rates in Guatemala)”,

Buenos Aires, Argentina: Asociación Civil Educación para Todos (Non-Governmental

Organization “Education for All”).

Aide Memoires from 28 Project’s Missions held from February 2005 (Identification Mission)

through December 2015 (Implementation Completion Report’s Mission)

Baez, J.; Lucchetti, L.; Genoni, M.; and Salazar, M. (2014); “Gone with the Storm: Rainfall

Shocks and Household Well-Being in Guatemala”, IZA Discussion Paper No. 8792.

Bonn, Germany: Institut zur Zukunft der Arbeit (Institute for the Study of Labor)

http://ftp.iza.org/dp8792.pdf.

Buettner, T. and Muenz, R. (2016); “Comparative Analysis of International Migration in

Population Projections”, KNOMAD Working Paper No. 10. Washington, DC: Global

Knowledge Partnership on Migration and Development (KNOMAD);

http://www.knomad.org/docs/demographic_change/KNOMAD%20WP%2010%20Comp

arative%20Analysis%20of%20International%20Migration%20in%20Population%20Proj

ections.pdf.

De Hoyos, R.; Rogers, H. and Székely, M. (2016); “Out of School and Out of Work: Risks and

Opportunities for Latin America’s Ninis”. Washington, DC: The World Bank.

Esquivel, Francisco (2006); “Situación del Sistema Educativo Guatemalteco (An Overview of

the Guatemalan Education System)”; Background Paper prepared for the Preparation of

the Quality Education and Secondary Education Project – BIRF 7430-GU;

http://siteresources.worldbank.org/GUATEMALAINSPANISHEXT/Resources/500796-

1166830633691/Situacion2004Inf.completo.pdf.

Ganimián, A. J. (2016); “Why Do Some School-Based Management Reforms Survive While

Others Are Reversed? The Cases of Honduras and Guatemala”; International Journal of

Educational Development, 47 (1), pp.33-46.

Implementation Status Reports (ISRs) No.1 (May 12th

, 2007) through No.22 (November 27th

,

2015); http://www.worldbank.org/projects/P089898/education-quality-secondary-

education?lang=en&tab=documents&subTab=projectDocuments.

MINEDUC (2004); “El Desarrollo de la Educación en el Siglo XXI (The Development of

Education in the 21st Century). Guatemala City: Ministry of Education.

http://www.oei.es/quipu/guatemala/educacion_sigloxxxi.pdf.

MINEDUC (2007); “Informe Tercero Básico 2006 (Report on the Results of the 2006 National

Assessment of 9th

Graders). Guatemala City: Ministry of Education.

http://www.mineduc.gob.gt/digeduca/documents/informes/Informe%20Tercero%20B%C

3%A1sico%202006%20FINAL.pdf.

MINEDUC (2008); “Inclusión Educativa: El Camino del Futuro – Un Desafío para Compartir.

Informe Nacional – República de Guatemala (Educational Inclusion: The Path to the

60

Future – A Shared-Responsibility Challenge. National Report – Republic of Guatemala).

Guatemala City: Ministry of Education.

http://www.ibe.unesco.org/National_Reports/ICE_2008/guatemala_NR08_sp.pdf.

MINEDUC (2010a); “Informe: Evaluación de Homologación para Carreras del Nivel Medio

2009 (Report: An Assessment of Teacher Credentials’ Standardization for Secondary

Education 2009)”; Guatemala City: Ministry of Education.

http://www.mineduc.edu.gt/recursoseducativos/Descarga/digeduca/informes/ficha_25.pdf

MINEDUC (2010b); “Línea Base “Calidad Educativa y Ampliación de la Educación Secundaria

– Proyecto BIRF 7430-GU (Education Quality and Secondary Education Project – BIRF

7430-GU: Baseline)”; Guatemala City: Ministry of Education.

http://www.mineduc.gob.gt/digeduca/documents/investigaciones/calidadEducativa_1901

2011.pdf.

MINEDUC (2012a); “El rezago escolar en el sistema educativo guatemalteco y el Programa de

Educación para Estudiantes con Sobreedad – PEPS (Overage in the Guatemalan

Education System and the [Accelerated] Primary Education Program for Overage

Students)”; Guatemala City: Ministry of Education.

http://www.mineduc.gob.gt/digeduca/documents/investigaciones/PEPS.pdf.

MINEDUC (2012b); “Estrategia para una Educación de Calidad para la Niñez y Juventud

Guatemalteca (A Strategy for a Quality Education for Guatemalan Children and Youth)”;


http://www.mineduc.gob.gt/portal/contenido/anuncios/estrategiaCalidadEducativa/docum

ents/Documento%20Estrategia%20para%20una%20educaci%C3%B3n%20de%20calida

d.pdf.

MINEDUC (2012c); “La Formación Continua y el Programa Académico de Desarrollo

Profesional Docente: Evaluación de la Primera Cohorte de Estudiantes 2009-2011 (In-

Service Teacher Training and the Academic Program for Professional Teacher

Development: An Asessment of the First Cohort of Students 2009-2011)”; Guatemala

City: Ministry of Education. https://www.scribd.com/doc/142299843/PADEP-D.

MINEDUC (2012d); “Plan de Implementación Estratégica de Educación 2012-2016 (2012-2016

Education Sector Strategic Plan’s Implementation Program)”; Guatemala City: Ministry

of Education. http://uvg.edu.gt/educacion/maestros-

innovadores/documentos/curriculo/Plan_2012_2016.pdf.

MINEDUC (2013a); “Análisis de las Oportunidades de Aprendizaje que se relacionan con el

Desempeño de los Estudiantes del Ciclo Básico del Nivel de Educación Media (An

Analysis of the Relationship between Learning Opportunities and Learning Outcomes in

Lower Secondary Education)”; Guatemala City: Ministry of Education.

http://www.mineduc.gob.gt/digeduca/documents/investigaciones/ODAS.pdf.

MINEDUC (2013b); “Guía Informativa para la Organización y Legalización de los Consejos

Educativos del Nivel Educación Media, Ciclo de Educación Básica (Guidelines for the

Organization and Legalization of Lower Secondary Education School Councils)”;

Guatemala City: Ministry of Education. http://uvg.edu.gt/educacion/maestros-

innovadores/documentos/curriculo/Protocolo_0.pdf.

61

MINEDUC (2013c); “Protocolo No. 0 - La Educación Bilingüe Intercultural en el Sistema

Educativo Nacional: Líneas Generales (Zero Protocol – Intercultural Bilingual Education

in the National Education System: General Overview)”; Guatemala City: Ministry of

Education. http://uvg.edu.gt/educacion/maestros-

innovadores/documentos/curriculo/Protocolo_0.pdf.

MINEDUC (2014a); “Informe de Factores Asociados: Ciclo Básico, Nivel Medio 2009 (Report

on the Determinants of Learning Outcomes for the 2009 National Assessment of 9th

-

Grade Student Achievement). Guatemala City: Ministry of Education.

http://www.mineduc.gob.gt/digeduca/documents/informes/INFORME_FA_IIIB2009.pdf.

MINEDUC (2014b); “Informe Departamental y Municipal de Tercero Básico 2013 (2013

National Assessment of 9th

-Grade Student Achievement: Departmental and Municipal

Report). Guatemala City: Ministry of Education.

http://www.mineduc.gob.gt/digeduca/documents/informes/2013/Informe_Departamental

_IIIB2013.pdf.

MINEDUC (2014c); “Informe de Tercero Básico 2009 (Report on the Results of the 2009

National Assessment of 9th

-Grade Student Achievement). Guatemala City: Ministry of

Education.

http://www.mineduc.gob.gt/digeduca/documents/informes/Informe_IIIB2009.pdf

MINEDUC (2014d); “La Formación Continua y el Programa Académico de Desarrollo

Profesional Docente: Evaluación de la Segunda Cohorte de Estudiantes 2010-2012 (In-

Service Teacher Training and the Academic Program for Professional Teacher

Development: An Asessment of the Second Cohort of Students 2010-2012)”; Guatemala

City: Ministry of Education.

http://www.mineduc.gob.gt/digeduca/documents/2014/investigaciones/PADEP_2.pdf.

MINEDUC (2015a); “Caracterización Sociolingüística y Cultural de Comunidades y Escuelas

de Párvulos, Preprimaria y Primaria de Guatemala 2010 al 2014 (Sociolinguistic and

Cultural Characterization of Communities and Pre-Kindergarten, Kindergarten and

Primary Education Schools in Guatemala from 2010 through 2014); Guatemala City:

Ministry of Education. Unpublished document.

MINEDUC (2015b); “Escuelas con Alto Rendimiento en Situaciones Adversas (High-Performing

Schools in Disadvantaged Contexts); Guatemala City: Ministry of Education.

http://www.mineduc.gob.gt/digeduca/documents/investigaciones/2015/Escuelas_alto_ren

dimiento.pdf.

MINEDUC (2015c); “Estudio sobre la Movilidad de los Estudiantes del Ciclo Básico hacia

Diversificado del Nivel Medio en Guatemala (A Study of the Transition from Lower to

Upper Secondary Education in Guatemala); Guatemala City: Ministry of Education.

http://www.mineduc.gob.gt/digeduca/documents/investigaciones/2015/Informe_Resultad

os_Movilidad_de_B%C3%A1sicos_a_Diversificado.pdf

MINEDUC (2015d); “Evaluación de la Calidad Educativa de las Modalidades de Entrega

Educativa del Ciclo Básico (An Assessment of the Quality of Educational Modalities for

Lower Secondary Education); Guatemala City: Ministry of Education.

http://www.mineduc.gob.gt/digeduca/documents/investigaciones/2016/Evaluacion_calida

d_educativa.pdf.

62

MINEDUC (2015e); “Informe de los Resultados de la Evaluación de Docentes Optantes a Plaza

2009-2014 (Report on the Results of Teachers Applying for Public Schools’ Job Posts

2009-2014); Guatemala City: Ministry of Education.

http://www.mineduc.gob.gt/digeduca/documents/informes/docentes/Informe_Doc_Optan

tes_Plaza_2009_2014.pdf.

MINEDUC (2015f); “Informe de Resultados de la Evaluación Nacional de Tercero Básico 2013

(Report on the Results of the National Assessment of 9th

-grade Student Achievement);


http://www.mineduc.gob.gt/digeduca/documents/informes/2013/Informe_IIIB2013.pdf

MINEDUC (2015g); “Memoria de labores de la Dirección General de Educación Bilingüe

Intercultural 2012-2015 (2012-2015 Summary of Main Achievements of the General

Directorate of Intercultural Bilingual Education); Guatemala City: Ministry of Education.

Unpublished document.

MINEDUC (2015h); “Programa de Educación para Estudiantes con Sobreedad (PEPS): La

Percepción de Sus Actores en San Marcos y Totonicapán ([Accelerated] Primary

Education Program for Overage Students: Stakeholders’ Views from San Marcos and

Totonicapán); Guatemala City: Ministry of Education.

MINEDUC (2015i); “Una Oportunidad de Cambio en Mi Labor Docente, Una Oportunidad

para la Educación de Guatemala – Compendio de Experiencias Exitosas de

Participantes y Egresados del PADEP/D 2013 (An Opportunity for Changing My

Teaching Practices, An Opportunity for Guatemala’s Education – A Compilation of

Successful Experiences from Participants and Graduates from 2013 PADEP/D);


https://issuu.com/digeduca/docs/buenas_practicas_padep.

MINEDUC (2016a); “Establecimientos Efectivos del Nivel Medio con Estudiantes en

Condiciones Socioeconómicas Adversas (Well-performing Secondary Education Schools

with Students from Disadvantaged Socioeconomic Backgrounds); Guatemala City:

Ministry of

Education.http://www.mineduc.gob.gt/digeduca/documents/investigaciones/2016/Estable

cimientos_efectivos_nivel_medio.pdf.

MINEDUC (2016b); “Evaluación de la Calidad Educativa de las Modalidades de Entrega

Educativa del Nivel Medio (An Assessment of the Quality of Educational Modalities for

Secondary Education); Guatemala City: Ministry of Education.

http://www.mineduc.gob.gt/digeduca/documents/investigaciones/2016/Evaluacion_calida

d_educativa.pdf.

MINEDUC (2016c); “Informe: Factores Asociados de Tercero Básico 2013 (Report on the

Determinants of Learning Outcomes for the 2013 National Assessment of 9th

Graders’

Learning Outcomes). Guatemala City: Ministry of Education.

http://www.mineduc.gob.gt/digeduca/documents/informes/basicos/FACTORES_ASOCI

ADOS_IIIB2013.pdf.

MINEDUC (2016d); “Redescubriendo la Lectura: Experiencias Positivas e Innovadoras de

Lectura con Jóvenes del Nivel Medio (Rediscovering Reading: Innovative and Successful

63

Stories on Reading Skills for Secondary Education Students”); Guatemala City: Ministry

of Education.

MINEDUC (2016e); “Informe de Cierre del Proyecto ‘Calidad Educativa y Ampliación de la

Calidad Secundaria – BIRF 7430-GU’ (Completion Report of the ‘Education Quality and

Secondary Education Project – BIRF 7430-GU’); Guatemala City: Ministry of Education.

MINEDUC (forthcoming.a); “Análisis de la Metodología que Orienta el Funcionamiento de

Cuatro Modalidades Educativas del Ciclo Básico del Nivel Medio en el Sector Oficial de

Guatemala y los Resultados que Obtuvieron en Lectura y Matemática los Estudiantes de

Tercero Básico Evaluados en el Año 2013 (An Analysis of the Methodology that Drives

the Operation of Four Lower Secondary Education Modalities of Public Schools in

Guatemala, and the Reading and Math Results for 9th Grade Students Tested in 2013)”;


MINEDUC (forthcoming.b); “Cumplimiento de los Aspectos Metodológicos de las Modalidades

INEB, IEBC, Telesecundarias y NUFEDs del Ciclo Básico del Nivel Medio (Compliance

with Methodological Aspects of INEBs, IBECs, TIs and NUFEDs Modalities for Lower

Secondary Education)”; Guatemala City: Ministry of Education.

MINEDUC (forthcoming.c); “Evaluación de la Implementación del Programa de Educación

para Estudiantes con Sobreedad – PEPS (An Assessment on the Implementation of the

[Accelerated] Primary Education Program for Overage Students)”; Guatemala City:

Ministry of Education.

MINEDUC (forthcoming.d); “La Formación Continua y el Programa Académico de Desarrollo

Profesional Docente: Evaluación de la Tercera, Cuarta y Quinta Cohorte de Estudiantes

(In-Service Teacher Training and the Academic Program for Professional Teacher

Development: An Asessment of the Third, Fourth, and Fifth Cohort of Students)”;


USAID (2007); “Equidad de la Educación en Guatemala (Equity of Education in Guatemala)”;

Serie de Investigaciones Educativas, Volumen 4 (Educational Research Series: Volume

4); Guatemala City: United States Agency for International Development (USAID).

USAID (2010); “Incentivos para la Profesionalización Docente: Bases Teóricas y Técnicas

(Incentives for Teacher Professional Development: Theoretical and Technical

Background)”; Guatemala City: United States Agency for International Development

(USAID).

World Bank (2005); “International Bank for Reconstruction and Development Country

Assistance Strategy for the Republic of Guatemala”; Report No. 31776-GT. Washington,

DC: World Bank.

http://documents.worldbank.org/curated/en/2005/04/6252359/guatemala-country-

assistance-strategy.

World Bank (2007); “Project Appraisal Document on a Proposed Loan in the Amount of

US$80.0 Million to the Republic of Guatemala for an Education Quality and Secondary

Education Project”; Report No. 36712-GT. Washington, DC: World Bank.

http://documents.worldbank.org/curated/en/2007/01/12331199/guatemala-education-

quality-secondary-education-project.

64

World Bank (2009a); “Guatemala: Poverty Assessment – Good Performance at Low Levels”

Report No. 43920-GT. Washington, DC: World Bank.

http://siteresources.worldbank.org/INTLACREGTOPPOVANA/Resources/GuatemalaPo

vertyAssessmentEnglish.pdf.

World Bank (2009b); “Implementation Completion and Results Report (IBRD-70250) on a Loan

in the Amount of US$62.16 Million to the Republic of Guatemala for a Universalization

of Basic Education Project”. Report No. ICR0000478. Washington, DC: World Bank.

World Bank (2009c); “Restructuring Paper on a Proposed Project Restructuring of Education

Quality and Secondary Education Project (IBRD 74300) in the Amount of US$ 80.0

Million to the Republic of Guatemala”. Project’s Restructuring No. 2. Washington, DC:

World Bank.

World Bank (2011); “Restructuring Paper on a Project Restructuring of Education Quality and

Secondary Education Project Loan (IBRD 74300) Approved by the Board on March 6,

2007 to the Republic of Guatemala”. Report No. 60951-GT. Project’s Restructuring No.

3. Washington, DC: World Bank. http://www-

wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2011/06/08/00035616

1_20110608023123/Rendered/PDF/609510PJPR0P080GT00RP0Revised0final.pdf.

World Bank (2012a); “International Bank for Reconstruction and Development and International

Finance Corporation Country Partnership Strategy for the Republic of Guatemala for the

Period FY 2013-2016”; Report No. 69229-GT. Washington, DC: World Bank.

http://documents.worldbank.org/curated/en/2012/08/16707995/guatemala-country-

partnership-strategy-period-fy2013-2016.

World Bank (2012b); “Restructuring Paper on a Project Restructuring of Education Quality and





1_20120727095127/Rendered/PDF/676690PJPR0REP0il0201200final00MD10.pdf.

World Bank (2013); “Restructuring Paper on a Project Restructuring of Education Quality and





4_20131126150203/Rendered/PDF/795750PJPR0P080379879B00PUBLIC00ACS.pdf.

World Bank (forthcoming.a); “Building Bridges in Guatemala – Systematic Country

Diagnostic”. Washington, DC: The World Bank.

World Bank (forthcoming.b); “Guatemala: Social Sector Expenditure and Institutional Review”.

Washington, DC: The World Bank.

65

Annex 10. Indicators Used to Measure Efficacy Indicator Baseline Period 1

(Effectiveness-R2)

Period 2

(R2-R5)

Period 3

(R5-Closure)

Actual

Access 1. Intake (Access) in grade 7

in modalities supported by

the Project

0 103,500 35,000 TIs;

16,838 NUFED

(Downgraded) 40,729 TIs;

12,444 NUFED

2. Gross enrollment rate in

lower secondary education

(Grades 7-9)

61% Nation;

42% Targeted

65% Nation;

58% Targeted

71.46% Nation;

51.56% Targeted

56% Targeted 69% Nation;

51% Targeted

3. Lower secondary education

(9th Grade) completion rate

50% Nation;

31% Targeted

53.40% Nation;

40% Targeted

53.40% Nation;

39.99% Targeted

47% Targeted

55% Nation;

41% Targeted

Quality 4. Average of Math and

Reading Pass rates,

Secondary Completers

11% TIs;

7% NUFED

4.4% TIs

6.7% NUFED

7% TIs;

5% NUFED

Primary

Education

5. Proportion of overage

pupils (13-15 year old) who

complete primary education

52% Higher than 52% 47.86% Nation (Dropped) 44.67% Nation

6. Primary education (6th

grade) completion rate

76% Nation;

68% Targeted

Above 72.3%

Nation

84.14% Nation;

82.59% Targeted

87% Targeted

84% Nation;

87% Targeted

Strengthened

Flexible

Modalities

7. Project-supported schools

receiving new curricular

orientations and materials

0 100% (~1,500) 967 TIs;

634 NUFED

967 TIs;

600 NUFED

967 TIs;

600 NUFED

8. Number of Project-

financed classes in

Telesecondary Institutions

that are functioning with

adequate equipment and

instructional materials

0 (Added in R2) 4,583 4,583 5,080

School

Management

9. Number of school councils

that receive grants

0 1,500 487 158

10. Percent of school

governments organized and

trained (as of originally

planned targeted schools)

0 100% (~1,500) 100% (~1,500) 1,350 1,489

66

For each indicator, achievement was calculated during each period as the achievement at

Project closure minus the baseline value divided by that period’s target minus the

baseline value. This was calculated, of course, for those indicators that were valid during

a given period. If the indicator was discontinued (dropped, downgraded) for a given

period, no calculations of achievement were undertaken in those cases. In cases where

indicators that have both National and Targeted area values or NUFED and

Telesecundaria values, achievement of the two sub-indicators are averaged for a scalar

measure of achievement (See Table in the ICR’s main text).

67

Annex 11. Beneficiaries

All interventions of the project were targeted using one of two targeting mechanisms:

Intervention Beneficiaries

1a. Support of Quality Primary Education Priority Municipalities

1b. Support Teacher Training Independent Strategy

2a. Feasibility studies and strategies for

secondary education curricula and evaluation

systems

CANCELLED

2b. Strengthening and Expansion of Pertinent

and Flexible Modalities for Lower Secondary

Education

Telesecundaria: Rural Areas

NUFED: Rural & high-risk students

2c Education Demand Scholarships for Low

Income Students

Low-income students in Priority

Municipalities

3a. Consolidation and Strengthening of

School-Based Management for Education

Access and Quality

Priority Municipalities

3b. Strengthening of MINEDUC’s capacity to

plan, manage, and supervise pedagogical,

administrative, and fiduciary processes.

Central Ministry Offices

Priority Municipalities During project design, the World Bank preparation team prepared a study to determine the areas

with the highest need for improvement in education outcomes63

. The study agglomerated

municipalities into 237 micro-regions comprised of roughly equally sized populations. It

calculated an index (the Indice de Rezago Educativo, or Education Lag Index) using three

education outcomes: overage sixth grade students, ninth grade completion, and ninth grade net

enrollment. Municipalities in micro-regions scoring on the lowest two-thirds of the index were

included on a list of priority municipalities.

Although the list of priority municipalities was determined by outcomes, they are strongly

correlated with the two beneficiary groups specified in the PDO (low-income and indigenous

communities). In priority municipalities, 75 percent of the population live in poverty (compared

to 47 percent in the rest of the country), and 27 percent live in extreme poverty (compared to 9

percent elsewhere)64

. In addition, 52 percent of the schools in priority areas listed the

predominant language of the community as something other than Spanish, compared to 32

percent nationwide65

.

Telesecundaria and NUFED Schools Telesecundaria and NUFED schools did not follow the Priority Municipality targeting, as both

programs existed before the start of the project. However, both types of schools were designed

63

Esquivel (2006) fully describes the method used to calculate the education efficiency index. 64

Calculated by the authors using Household Survey ENCOVI 2011 data. 65

Calculated by the authors using 2014 MINEDUC school census data.

68

for particular situations: the Telesecundaria model is designed for rural areas, where a higher

percentage of indigenous and there is a higher incidence of poverty. NUFED schools are

designed to target low-income and at-risk communities. While the average Guatemalan school is

in a community with a poverty rate of 63 percent, the NUFED school average is 68 percent, and

Telesecundaria school average is 67 percent. In terms of location, 99 percent of Telesecundaria

schools and 93 percent of NUFED schools are in rural areas, compared with 69 percent of

schools overall. While 32 percent of schools nationwide report the community language as

something other than Spanish, 34 percent of Telesecundarias and 38 percent of NUFED schools

report another language. While these numbers are surprisingly low, it may be the result of the

fact Spanish has overtaken indigenous languages in many predominantly indigenous

communities.

Scholarships Starting in 2013, recipients for the scholarship were stratified geographically using a formula.

The number of scholarships each municipality received was determined by municipal poverty

rates and repetition rates. Municipalities then determined the number of scholarships provided to

each school. A group of teachers and the director of each school then selected several students to

apply for the vacancies: on average, three students were selected to apply for every two

vacancies. Municipal or departmental boards would then select which applicants would receive

the allotted scholarship based on their need and grades.

Teacher Training Although PADEP/D was originally expected to focus in priority municipalities, it was scaled up

to include other areas deemed important to the Government. There is no clear systematic

explanation for which municipalities were chosen. In addition, PADEP/D training expanded

from only primary teachers, as originally designed, to include pre-primary teachers.

Central Ministry Offices The interventions designed to strengthen regional data collection and analysis took place in the

national offices in the Capital and, to a lesser degree, related to the Education offices located in

the capital of all departments.

69

Annex 12. An Assessment of the Indigenous Peoples Plan’s

Achievements

As of 2006, poverty and extreme poverty in Guatemala were concentrated

disproportionately in rural areas, and among indigenous households. More than 7 in 10 poor

and more than 8 in 10 extremely poor lived in rural Guatemala. Poverty rates among indigenous

populations were in the order of 75 percent, two times more prevalent than for non-indigenous

populations; extreme poverty rates among indigenous people were in the order of 28 percent, 3.5

times bigger than in non-indigenous settings66

.

At appraisal, rural and indigenous students were less likely to complete primary school and

also less likely to enroll in lower secondary education level than their peers. Rural students

were 30 percent less likely to complete primary school than urban students67

. Every year, one out

of eight primary students failed their grade, and these numbers were higher for indigenous

students and rural inhabitants. The low efficiency levels of primary education impacted directly

the transition into the LSE level (grades 7 through 9). Differences between indigenous and non-

indigenous student populations in LSE access were stark: whereas non-indigenous student

populations showed a 74 percent GER for LSE, indigenous student populations presented a 26

percent rate. Geographic access was a clear issue: urban students were 40 percent more likely to

enter secondary school than rural students.

The Indigenous Peoples Safeguards Policy was triggered under the Project due to the

presence of indigenous communities in the intervention areas of the Project. As a result, a

comprehensive IPP was formulated with the participation of the Ministry of Education and

technical assistance of the World Bank. The IPP was in line with the 1996 Peace Accords and

also took into consideration the findings and recommendations from the Integrated Social

Assessment carried out as part of the project preparation in 2006. The IPP was applicable to all

components of the Project. Among the interventions of the plan, were the following: inclusion of

ethnic and language variables in information systems; classification of bilingual schools;

application of intercultural, multicultural and bilingual criteria to the curriculum; teacher

training; creation and production of materials for bilingual and multicultural education; student

scholarships; and consolidation of school based management among others. The IPP was

originally calculated at US$42.5 million. In addition, the Project had, arguably, the most

comprehensive IPP ever set up for an Education Project before.

The Indigenous Peoples Plan was slightly modified in 2010 along the lines of the needs

presented by the Vice-Ministry of Bilingual Intercultural Education (BIE) and its General

Directorate for Intercultural Bilingual Education (DIGEBI). The major changes were: (i) a

separate IPP budget was no longer included; (ii) added 4 new indicators and increased the

percentage of scholarships that would go to indigenous students from 50 percent to 70 percent

and (iii) component “0” was included, but only for the purpose of following-up on those items.

66 Ibid. 67 See USAID (2007).

70

The Project as a whole was restructured 4 times in the period between 2010 and 2014. Despite

the multiple restructurings of the project, the last IPP supervision mission that was carried out

concluded that the IPP continued to be implemented as agreed since 2010.

The Project helped achieve substantial impact for indigenous peoples’ issues through the

contextualization of interventions beyond specific targeting68

. The Project succeeded in

monitoring the institutionalization of coordinated efforts of DIGEBI and the other Directorates of

MINEDUC for the supply of culturally-adequate education programs for Intercultural Bilingual

Schools (see MINEDUC 2015g). The IPP pushed the incorporation of bilingual and intercultural

education, generating a comprehensive approach to IBE. During the Project, modules and

educational materials were printed for PEPS and the curriculum design for this modality was

adapted. Teachers were trained in the adequate implementation of PEPS methodology. The in-

service teacher training program trained teachers at the University level for teachers in

Intercultural Primary, Intercultural pre-primary, Bilingual and Intercultural Primary and

Bilingual and Intercultural pre-primary. In addition, schools were socio-linguistically

characterized, which is a necessary tool for implementing targeted IBE strategies. The

characterization will allow bilingual teachers to be adequately placed, according to the

community and school context. Relevant statistics relating to indigenous groups were also

collected.

The greatest long-term achievement of the Indigenous Peoples’ Plan was that it contributed

to a broader agenda of multicultural and bilingual education. This was featured so high in

the last four years of implementation that many of the original goals set at the IPP were

surpassed by the time the Project closed. The IPP resulted in consideration of the needs of a

multilingual and multicultural population in a greater number of decisions made by MINEDUC.

While the immediate purpose of the Indigenous Peoples’ Plan was to help the Government

improve work related to the project, it served helped the Government develop its own plan to

better serve the subpopulation. It helped raise issues and bring to the table stakeholders who are

now integrated into the Ministry of Education’s decision-making processes. In 2013, under the

Education Strategic Plan of 2012-2016, policy number 5 was established as Multilingual

Multicultural Policy which launched IBE as a Government project, broader then a project of this

Loan (see MINEDUC 2013c). This policy transcends and surpasses the IPP. The Bank appoints a

specialist in indigenous issues that enriches the dialogue and the Ministry of Education hires a

specialist in indigenous issues accompanying the process.

The IPP (2010) was a substantial and important part of the Project, and has been

implemented satisfactorily as a joint effort of MINEDUC Directorates. At project closing,

IBE is part of the Government’s Plan, which more adequately responds to Guatemala’s diverse

country context (see, for example, MINEDUC 2015a). It is positive that at closing, the

education policy is broader than the original IPP plan that was included in this Project.

68 The Scholarship Program was developed predominantly for poor indigenous and rural students from targeted regions by the

project. 59% of scholarships were given to indigenous population. Telesecundaria and NUFED institutes from predominantly

indigenous regions received money transfers by way of education councils.

71

MAP