evaluation of qbep s township education … · evaluation of qbep’s township education...

EVALUATION OF QBEP’S TOWNSHIP EDUCATION IMPROVEMENT PLAN ACTIVITIES: Final Report September 2015 to January 2016 by Montrose Commissioned by: UNICEF Myanmar on behalf of QBEP

Submitted: 17 June 2016

In partnership with:

Evaluation of QBEPs Township Education Improvement Plan Activities

ii

Evaluation of QBEP’s Township Education Improvement Plan Activities: Final Report

© United Nations Children’s Fund, Yangon, 2016

United Nations Children’s Fund

P.O. Box 1435

Yangon, Union of Myanmar

[email protected]

June 2016

The purpose of publishing evaluation reports produced by UNICEF Myanmar is to fulfil a corporate

commitment to transparency through the publication of all completed evaluations. The reports are

designed to stimulate a free exchange of ideas among those interested in the topic and to assure those

supporting the work of UNICEF that it rigorously examines its strategies, results and overall

effectiveness.

This End of Project Evaluation of Township Education Improvement Plan was prepared by Dr. Naing

(of EMPOWER) and Jane Sail (of Montrose). The evaluation was managed by an Evaluation Specialist,

Mathias Kjaer, within Education Section, under the guidance of the Education Chief and the Evaluation

Specialist at UNICEF Myanmar. It was supported by a Reference Group, which included seven

representatives from the Ministry of Education’s Department of Basic Education and UNICEF. The

Regional Evaluation Adviser and the Regional Education Adviser, within UNICEF Regional Office for

East Asia and the Pacific, provided guidance and oversight.

The purpose of this report is to facilitate the exchange of knowledge among UNICEF personnel and its

partners. The contents of the report do not necessarily reflect the policies or views of UNICEF. The

text has not been edited to official publication standards and UNICEF accepts no responsibility for

error. The designations in this publication do not imply an opinion on the legal status of any country

or territory, or of its authorities, or the delimitation of frontiers.

The copyright for this report is held by the United Nations Children’s Fund. Permission is required to

reprint/reproduce/photocopy or in any other way to cite or quote from this report in written form.

UNICEF has a formal permission policy that requires a written request to be submitted. For non-

commercial uses, the permission will normally be granted free of charge. Please write to UNICEF

Myanmar to initiate a permission request.

mailto:[email protected]


iii

CONTENTS

ACRONYMS AND ABBREVIATIONS ................................................................................................ V

MAP OF MYANMAR SHOWING QBEP CORE TOWNSHIPS ............................................................. VII

EXECUTIVE SUMMARY .............................................................................................................. VIII

1. CONTEXT .............................................................................................................................. 1

1.1 Object of evaluation: TEIP project description ................................................................... 1

1.2 Context of basic education in Myanmar ............................................................................. 6

2. EVALUATION PURPOSE AND OBJECTIVES .............................................................................. 9

2.1 Evaluation purpose, objectives and intended audience ..................................................... 9

2.2 Evaluation criteria and scope .............................................................................................. 9

3. METHODOLOGY.................................................................................................................. 12

3.1 Overall approach ............................................................................................................... 12

3.2 Data collection .................................................................................................................. 13

3.3 Analysis methods .............................................................................................................. 14

3.4 Data sources ...................................................................................................................... 15

3.5 Sampling strategy .............................................................................................................. 15

3.6 Ethical considerations ....................................................................................................... 16

3.7 Limitations and mitigation strategies ............................................................................... 17

4. FINDINGS ........................................................................................................................... 19

4.1 Relevance .......................................................................................................................... 19

4.2 Effectiveness and efficiency .............................................................................................. 25

4.4 Sustainability ..................................................................................................................... 40

5. CONCLUSIONS .................................................................................................................... 44

6. LESSONS LEARNED .............................................................................................................. 49

7. RECOMMENDATIONS ......................................................................................................... 50

BIBLIOGRAPHY ........................................................................................................................... 54

ANNEXES ................................................................................................................................... 55

ANNEX 1: TERMS OF REFERENCE ................................................................................................. 55

ANNEX 2: REVISED QBEP RESULTS HIERARCHY ............................................................................. 60

ANNEX 3: EVALUATION MATRIX .................................................................................................. 61

ANNEX 4: DETAILED METHODOLOGY ........................................................................................... 66

ANNEX 5: ADDITIONAL FINDINGS ................................................................................................ 77

List of figures

Figure 1: Overview of decentralisation structures in Myanmar ............................................................................. 8

Figure 2: Outline of sequential mixed methods approach ................................................................................... 13

List of tables

Table 1: Townships that had completed development of TEIP at the time of evaluation (December, 2015) ....... 6


iv

Table 2: Basic education indicators in Myanmar (Source: National Census, 2014) ................................................ 8

Table 3: Number of interview sessions and surveys conducted and number of participants involved in data

collection (October to November 2015) ............................................................................................................... 15

Table 4: Opinions on relevance and appropriateness of TEIP and other current systems (October – December

2015) ..................................................................................................................................................................... 24

Table 5: Overview of activities to address TEO capacity needs ............................................................................ 26

Table 6: Reasons for not implementing TEIP (October – December 2015) .......................................................... 29

Table 7: Opinions on decentralised school plans and township education plan (October – December, 2015) .. 41

Table 8: Opinions on foundations needed for sustainability of TEIP (October – December 2015) ...................... 43

Table 9: Lessons learned and suggestions for incorporation into future projects ............................................... 50


v

ACRONYMS AND ABBREVIATIONS

CBO Community Based Organization CE Consejos Escolares (School Council) CEC Consejos Escolares del Centro (Center School Council) CEE Consejo Escolar del Estado (State School Council) CESR Comprehensive Education Sector Review DBE Department of Basic Education DEO District Education Officer DEPT Department of Education Planning and Training DFAT Department of Foreign Affairs and Trade (Australia) DRD Department of Rural Development DSW Department of Social Welfare ECCD early childhood care and development ECD early childhood development EFA Education for All EMIS Education Management Information System EU European Union FCI Inter-Territorial Compensation Fund FGD focus group discussion FESR Framework for Economic and Social Reform GI group interview HDN Hlaing Thar Yar Development Network IDI in-depth interview KII key informant interview LODE Right to Education Law M&E monitoring and evaluation MDEF Multi Donor Education Fund MNEC Mon National Education Committee MOE Ministry of Education NESP National Education Sector Plan NGO non-governmental organization OECD Organization for Economic Co-operation and Development/Development Assistance

Committee OSC out-of-school children PCM Project Cycle Management PRA Participatory Reflective Appraisal QBEP Quality Basic Education Programme SBM School Based Management SEAMEO South East Asia Ministries of Education Offices SEO State Education Officer SIP School Improvement Plan SITE School-based In-service Teacher Education SPSS Statistical Programme for Social Science SSA School Self-Assessment SWOLs Strengths, Weaknesses, Opportunities and Limitations TDAC Township Development Assistant Committee TED Township Education Department TEIP Township Education Improvement Plan TEMIS Township Education Management Information System TEO Township Education Officer TGAD Township General Administrative Department


vi

THD Township Health Department TMC Township Municipality Department and Committee TOR terms of reference US United State of America UNEG United Nations Evaluation Group UNICEF United Nations Children’s Fund WVM World Vision Myanmar


vii

MAP OF MYANMAR SHOWING QBEP CORE TOWNSHIPS


viii

EXECUTIVE SUMMARY

The Township Education Improvement Plan (TEIP) is a project component under the Quality Basic Education Programme (QBEP), a joint partnership between UNICEF and the Multi Donor Education Fund (MDEF) partners comprised of Australia, Denmark, the European Union (EU), Norway and the United Kingdom. QBEP is the second phase of this partnership, running from 2012 to 2015 and built upon the first phase (MDEF 1) that ran from 2007 to 2011. The TEIP project was launched in Myanmar in May 2013 with the intention of covering 34 townships, ten townships under a whole state approach1 in Mon State and 24 townships in other states and regions. At the time of evaluation, TEIP had been operational for two years and five months. QBEP's overall goal was “improved access to and quality of education for all children of Myanmar” while its purpose and main objective was that, “an inclusive and informed National Education Sector Plan (NESP) and supporting structures are developed and implemented”. The TEIP project is situated under Output 3 "increased Ministry of Education (MOE) capacity for planning, monitoring and evaluation (M&E)" and was intended to support NESP through strengthening systems that support quality basic education (Outcome 1). TEIP was one of two main activities under Outcome 1, namely strengthening the capacity of the MOE at all levels to better plan, monitor and evaluate their primary education activities and respond to ongoing education reforms. QBEP was designed against a backdrop of rapidly changing operating environment and amongst national moves toward decentralisation; TEIP was intended to be a key instrument for the decentralisation of education planning and management in Myanmar. The MOE was the main implementing agency of the TEIP project while UNICEF's role was to assist MOE counterparts in planning and organising activities, providing them with technical assistance, monitoring TEIP activities and providing advocacy targeted at the subnational level education offices of the Department for Basic Education (DBE). At the time of this report, approximately US$108,178 had reportedly been spent on TEIP over the course of QBEP and 25 out of the 34 townships had completed the second draft development of a TEIP.

Evaluation purpose, objectives and intended audience

In 2015, UNICEF contracted Montrose to undertake an independent end-of-project evaluation of TEIP. Montrose undertook this contract in partnership with EMPOWER, a Myanmar consultancy group. The call for evaluation was in response to a midterm review, carried out in 2014 and commissioned by the Government of Myanmar, MDEF partners and UNICEF, which found that QBEP would benefit from more independent evaluation of their activities. The purpose of this end of the project performance evaluation, per the terms of reference (TOR), was to assess TEIP as a QBEP funded activity, strengthen its ongoing implementation and provide lessons for future application. The evaluation was intended to examine the relevance, effectiveness, efficiency and likely sustainability of QBEP’s TEIP activities to date, identify lessons learned and formulate recommendations for future programming. The evaluation was therefore both summative and formative in nature. It is anticipated that the evaluation will be able to support the new Government in deciding their education priorities as well contribute toward a decision on whether to continue QBEP support for decentralisation reform.

1 The whole state approach adopted in Mon State aimed at improving school readiness and primary-level education planning,

delivery and quality at the state, township and school level through systematic capacity development that was inclusive of all

key education actors. In addition, the whole state approach was hoped to support the Government of Myanmar's progress in

decentralisation by strengthening partnerships between education officials, non-state actors and civil society.


ix

Specifically, the evaluation had the following specific objectives: 1. To test programming assumptions and the underlying theory of change; 2. To identify areas where implementation can be strengthened under QBEP for possible expansion

under QBEP; 3. To explore a finding from the QBEP midterm review questioning the relevance of its TEIP

activities; and 4. To respond to an overarching midterm review recommendation that QBEP assess its activities

before scaling up. The evaluation sought to provide an independent assessment of QBEP’s TEIP activities based on the Organization for Economic Cooperation and Development's Development Assistance Committee (OECD/DAC) criteria listed in the evaluation questions. Per the TOR, the evaluation was not intended to examine impact or higher level results; this is because two and half years of implementation was not sufficient to produce higher level results. The three major evaluation questions are listed below:

1. Are TEIP activities relevant to the planning needs of Township Education Officers (TEOs) to manage the delivery of basic education in their townships?

2. To what degree are TEIP activities effective and efficient in developing education plans that are inclusive, informed, and realistic?

3. What elements of the TEIP activities should be sustained as part of NESP following the end of the current QBEP funding?

The primary audience of the evaluation are intended as UNICEF and MDEF partners. Relevant MOE officials at the central level and relevant MOE officials at state, district and township levels are also expected to use the evaluation findings.

Evaluation methodology

In the absence of specified indicators from project documents, the criteria for the evaluation were developed by the technical team lead and were based on lines of enquiry per OECD/DAC recommendations. Where available, TEIP and QBEP project documents were used to inform evaluation criteria. The TEIP manual provided benchmarks for determining indicators of effectiveness and relevance. Per United Nations Evaluation Group (UNEG) report guidelines, evaluation criteria and subsequent data collection, analysis and reporting included considerations of equity and gender equality for relevance, effectiveness, efficiency and sustainability of the TEIP project, as appropriate. In keeping with UNICEF’s commitment to human rights principles, the evaluation criteria included considerations of participation and inclusion. The lines of enquiry and criteria developed by evaluators were actively reviewed and revised by a Reference Group comprising officials from the DBE and UNICEF staff. Their input was sought throughout the course of the study for comments and feedback on the inception report, data collection tools, sampling strategy and draft evaluation report. The evaluation employed a quasi-experimental design known as ‘Static Group Comparison Design’, in which data collection was undertaken at intervention sites and comparison sites only at the end of the project (post-test design). Six TEIP sites and one comparison site were non-randomly selected for study. A maximum variation sampling method was applied to identify different townships with diverse operational contexts across five states and regions. A sequential mixed methods data collection approach was applied to the design and implementation of the evaluation. A total of n=461 surveys and interview sessions were carried out, comprising focus group discussions, in-depth interviews, participatory workshops, general interviews, school surveys and township education department surveys. In total, n=939 participants were involved in the data collection process. Data sources ranged from secondary document sources provided by UNICEF, such as the TEIP manual, available QBEP programme documents and reports by the TEIP manual design consultant. In addition, a


x

broader literature review using non-UNICEF documents was conducted in order to learn lessons from other decentralisation initiatives around the world. Major limitations to the evaluation relate to programmatic issues, such as suboptimal document management, lack of standalone project documents, and lack of baseline data—as well as issues with research design, such as the self-administered questionnaire and elements of qualitative methods. In addition, elements of the methodology such as self-administered questionnaires, participant recall, qualitative methods, and limited sample size posed some challenges. However, the study took measures to mitigate these drawbacks to the extent possible, within the available resources and other priori limitations, meaning that the conclusions drawn from this study should still be considered valid. Ethical considerations were taken into account in terms of the design and implementation of the evaluation.

Main findings and conclusions

A number of promising practices and results were evidenced in the course of this evaluation in spite of several programmatic challenges. TEIP was launched at a time when Myanmar was undergoing a transition to democracy and decentralisation, which made the concept of TEIP relevant to the country’s political and administrative context. TEIP activities, as a support to the decentralised planning and management of education services, could have offered an opportunity to capture the specific needs of individual townships and respond to them through designing and delivering tailored solutions. They could have strengthened management capacity to deliver these tailored education services at the subnational level, both conceptually and in practice. In sum, the concept of developing skills normally confined to the central MOE levels, in support of decentralisation, was found relevant. The evaluation found that the MOE, as implementer, and UNICEF, as a supporting agency for technical and financial assistance, accomplished the development of first draft TEIPs in all 34 townships and second-draft TEIPs in 25 out of 34 target townships. TEIP training activities led to promising practices that could be replicated in potential future decentralised programmes. The key achievements of note were:

Producing a pool of capable and committed TEIP workshop facilitators and TEIP committee members who could continue to facilitate future iterations of TEIP;

Strong participation of women in the role of workshop facilitators contributing to improved gender balances in MOE;

Initiating the practice of undertaking needs assessments and planning by a group of first line and second line leaders at the township level; and

Attitudinal change of MOE staff in recognising the importance of collecting valid data and utilising proper data analysis for planning at school and township levels.

That capacity was developed through TEIP is clearly evident, and an important positive finding of effectiveness was that unintended stakeholders were able to benefit from participation in the TEIP process, such as Deputy TEOs, Assistant TEOs, and male and female head teachers. During the course of the evaluation, the team found several promising practices that could be replicated in the future. Notable accomplishments of effectiveness were the participation of lower level MOE staff members and the consideration taken of basic township level needs and context in the course of developing the first draft of a manual, as well as the creation of a pool of TEIP facilitators. The latter became operational and able to assist TEIP committee members in acquiring knowledge and skills and in completing second versions of TEIP, which was an unintended, but notable achievement. Both TEIP committee members and facilitators (comprising approximately equal numbers of men and women) strengthened their capacity in needs assessment and planning.


xi

The whole state approach was found to have produced several benefits: (a) strengthening needs assessment and planning capacity of township, district, and state level staff; (b) strengthened relationships and collaboration between UNICEF and MOE at different levels; (c) trust building between the MOE and the Mon National Education Committee (MNEC); (d) extended provision of capacity building opportunities such as TEIP and the School-based In-service Teacher Education (SITE) pilot project, as well as financial and material support to impoverished MNEC schools; (e) access to school grant scheme by MNEC schools; and (f) gradual convergence of two parallel education systems of the MOE and MNEC. In terms of sustainability, three elements were found that could support future implementation of TEIP and other similar decentralisation initiatives. Firstly, considerable levels of political support, concern, and interest were shown and/or provided to the education sector by government departments and state/region governments. Secondly, an overwhelming proportion of study participants expressed their support for decentralised planning and implementation of education services, as well as for establishing and operating school funds and township level education funds. Thirdly, half of the TEIP townships sampled, as well as the one non-TEIP township, had established and were running township-level education funds or foundations, and many schools had their own school fund mechanism. Having a fund mechanism was found to be one of the key enabling factors for sustainability. These findings are highly commendable and the evidence of changed attitudes is one potential sign of lasting project effectiveness. However, the evaluation found several problematic elements that called into question the relevance of the TEIP project as a whole. In response to the four underlying objectives of the evaluation, the first finding is that two key programming assumptions were misplaced. The first being that funding would be available for the implementation of the TEIPs developed by TEIP committees. This was not the case, however, and lack of funding from QBEP and the MOE was a key reason for why the intended implementation of TEIPs did not take place. The lack of plan implementation meant that only half of the skills relevant to TEO needs were addressed. The second assumption was that the operating environment in terms of the readiness of decentralised systems in Myanmar was sufficient for TEIP activities to be relevant or effective. Again, this was not found to be the case. While the concept of TEIP as a tool to improve subnational education capacity made sense in isolation, within the specific Myanmar context the extent to which TEIP activities were relevant remains dubious. This is partly because centralised decision making is still the norm and it appears that clear instructions regarding how the decentralisation process would work were not provided to subnational education departments by the central level MOE. In addition, insufficient consultation between UNICEF and MOE decision makers meant that instructions on the TEIP process were not clearly explained to, or understood at, the subnational level. In order for TEIP to support the NESP, and related to the second evaluation objective, it was found that all areas of the project need to be strengthened. This means that any future expansion of the project would require a rethinking of the design with clarification of project objectives and should be based on the findings of comprehensive needs assessments, political economy analysis and risk analysis. The evaluation considers the question raised by the midterm review of the relevance of TEIP activities an important one. As mentioned, while the idea of TEIP may have been relevant in theory, in practice this is less conclusive as the needs of TEOs were not well captured, the plans were not implemented in any township, centralised systems and attitudes combined with lack of instructions from the Union level meant that decisions could not be made, and the overall objectives of TEIP were not clear to stakeholders. In response to the final evaluation objective regarding the potential scale up of TEIP, this report found that before a decision is made, all the lessons learned and recommendations provided need to be addressed and actioned. If this cannot be done, then TEIP should not be scaled up as it will remain subject


xii

to the flaws demonstrated in the current modality and will divert finite resources away from other priority education needs. The unknown specifics of decentralisation at the time of TEIP conception and the inability to predict how fast, or otherwise, the progress of meaningful decentralisation in Myanmar would take may account for the misalignment of TEIP activities to needs. However, it is arguable that such a risk could have been highlighted and mitigated if a more thorough TEIP-specific political economy or risk analysis had been conducted at the start of the project. Similarly, with more pre-project preparation, the uncertain sources of funding could have been identified at an earlier stage, before the project was allowed to run on untested assumptions and be required to halt half of its intended activities. It is surprising that donor oversight was not more rigorous in this respect.

Lessons learned

Several lessons can be taken from this evaluation, primarily in terms of the importance of rigorous project design and management. These can be applied to the improvement of all future projects as well as for any potential iteration of TEIP. The following are the main lessons identified by the evaluation team:

The importance of rigorous planning and project-specific design activities (as opposed to overall programme design activities): Planning and implementation of the project was hindered by a political, economic, social, and cultural environment that was not entirely conducive to accomplishing the results that were originally intended. These contextual challenges appear to have encouraged alteration of both expected results and processes of TEIP throughout the course of the project, which is a possible explanation for the failure of the project to meet all of its intended objectives. Inadequacies in project design contributed to unclear objectives, exacerbated by continuous changes to the project. Greater risk assessment could also have led to a reassessment of the contextual feasibility of the project as a whole.

The necessity of systematic project documentation and communication: Challenges during project implementation and associated changes to the project design were neither adequately documented nor communicated with key stakeholders. Project documents that exist should be efficiently organised to support internal institutional organization as well as aid external evaluation.

The need to facilitate better conditions for monitoring and evaluation: During the monitoring and evaluation stage, challenges and progresses of the project should have been recorded through regular progress reports, which could then have been taken into account when undertaking the end of the project evaluation. This would have helped the evaluation set more exact evaluation questions and criteria, lines of inquiry and evaluation methods.

The importance of inclusive stakeholder engagement: Weak involvement of decision makers from the MOE had a detrimental effect on the project’s overall relevance. This demonstrates the importance of adequate involvement of key stakeholders in the design stage. Beneficiary stakeholder input and feedback on project activities and relevance is also key.

Key recommendations

The following main recommendations are based on the findings, conclusions and lessons learned and are organised in order of priority.

1. Clearer decision-making for decentralised planning and implementation: For policy makers and decision makers in MOE and UNICEF and donors It is recommended that the objective of any future replication of TEIP is made clearer. Specifically, it is recommended that the objective should be decentralised planning and implementation of tailored intervention measures that address specific needs of the education sector. This should be complementary to the development of other sectors at the national and subnational level. The timeframe is dependent on whether a future iteration or expansion of TEIP is agreed upon; if so, this should be an immediate priority and be addressed in the first stages of design.


xiii

2. Need to address root causes of education bottlenecks for greater effectiveness and efficient use of resources: For policy makers and decision makers of the MOE and UNICEF and donors It is paramount that the future decentralised planning and implementation of the education sector be conceptualised and implemented to address fundamental root causes of education bottlenecks and their consequences, such as the examination system, quality of teaching and competency of students, utility of education in real life and quantity versus quality. Addressing these fundamental issues will enhance the effectiveness and efficiency of activities and maximise the value of finite, precious resources invested. As above, the timeframe for this recommendation is dependent on whether a future iteration or expansion of TEIP is agreed upon; if so, this should be addressed in the immediate design period.

3. Integration of different sectors and education service providers to promote mutually

complementary development and inclusiveness and equity of education: For schools and Township Education Departments, District Education Departments and State Education Departments Future decentralised projects or programmes should ensure that at each important stage of the project or programme, relevant stakeholders from other sectors such as Township Government Administrative Departments and the Department of Rural Development, private schools, monastic schools and NGOs that provide education services or funds are involved. This approach will help TEIP integrate with economic, health, and other sectors so that joined up actions will occur to support broader township level development. The timeframe for this recommendation is ongoing throughout the project lifespan.

4. Strengthening the entire process of results based management for greater effectiveness,

efficiency, and sustainability: For facilitators and programme managers of TEIP from MOE and UNICEF Any future TEIP design should encompass the entire results based management cycle with the involvement of key stakeholders. Each step of results based management cycle should be designed and implemented fully and rigorously. This recommendation should commence during the design stage and the timeframe should be considered the lifespan of the programme.

5. Embedding logical links and three essential attributes of programme impact: For policy makers and decision makers of MOE and UNICEF, and facilitators, programme managers and implementers of MOE Impact and outcome are a function of coverage, quality and duration of implementation of any activity, strategy, project, or programme. Only when these three attributes are sufficient within each type of activity, will outcome and impact level changes be attained.

6. Step-wise gradual expansion and whole district approach: For policy makers and decision makers of MOE and UNICEF and donors Before scaling up TEIP, it is essential that all the above recommendations are addressed and the current modality is modified. Once these areas have been covered, a step-wise gradual expansion approach should be applied via two phases. Each phase should be implemented over a three to five year project life-span to ensure that the duration of implementation is sufficient to trigger outcome level results and follow a thorough results based management process.


1

1. CONTEXT

1.1 Object of evaluation: TEIP project description

Context of TEIP within QBEP The Townships Education Improvement Plan (TEIP) is a project component under the Quality Basic Education Programme (QBEP), a joint partnership between UNICEF and the Multi Donor Education Fund (MDEF) partners comprised of Australia, Denmark, the European Union (EU), Norway and the United Kingdom. QBEP is the second phase of this partnership, running from 2012 to 2015 and built upon the first phase (MDEF 1) that ran from 2007 to 2011. The TEIP project was launched in May 2013 with the intention of covering 34 townships, ten townships under a whole state approach in Mon State and 24 townships in other states and regions. During the four years of QBEP's life-span, UNICEF Myanmar, with MDEF funding, planned to reach an estimated 5,500 primary schools in 34 core townships through a range of project components. Strategic planning took place for the whole state approach in Mon State, with the aims of improving school readiness and primary-level education planning, delivery and quality at the state, township and school level through systematic capacity development that was inclusive of all key education actors. In addition, the whole state approach was hoped to support the Government of Myanmar's progress in decentralisation by strengthening partnerships between education officials, non-state actors and civil society.2

QBEP was designed against a backdrop of political change and amongst national moves toward decentralisation; its original intent was to "capitalise on the ongoing political reforms, including steps toward decentralisation", whereby the original rationale held that "a combination of capacity building and supply provision in select disadvantaged townships combined with national level capacity development… would result in improved education access, equity, quality and management, and thereby increase children's completion of primary-level education."3 In recognition of the highly centralised decision making and management structures within the MOE, which limited participation from outside central government and provided little opportunity for horizontal or inter-sectoral interaction, the shift toward decentralisation was a strong contextual influence behind the design rationale of QBEP. Decentralisation was viewed as an opportunity for fresh capacity development and system building for the MOE.4 Expected results chain The original QBEP design document 2012, states its purpose as, "increased number and proportion of children accessing and completing quality basic education in targeted townships". The four key Outputs (Results) were: 1) expand coverage of quality early childhood development (ECD) services, 2) improve quality of teaching and learning, 3) enhance planning, management and monitoring and 4) enhance coverage, quality and relevance of second chance education.5 Under Output 3, "number of townships with Township Education Plans" was an intended indicator and township education planning was intended to coordinate capacity-building for improved education management at the township level. However, a midterm review of QBEP was carried out in August 2014, commissioned by the Government of Myanmar, MDEF partners and UNICEF. The findings highlighted the need for key changes to be made for the then remaining phase of QBEP. As a result of the review's recommendations, the programme's theory of change and underlying objectives were revised. Under the revised QBEP results hierarchy,

2 UNICEF, Joint Performance Improvement Plan (JPIP), 2014, p. 6. 3 Ibid, p. 11. 4 UNICEF, QBEP Design Document, 2012, p. 7. 5 Ibid, p. 7.


2

QBEP's intended goal was “improved access to and quality of education for all children of Myanmar” while its purpose and main objective was that, “an inclusive and informed NESP and supporting structures are developed and implemented”. The TEIP project is situated under Output 3 (increased MOE capacity for planning, monitoring and evaluation), and was intended to support the NESP through strengthening systems that support quality basic education (Outcome 1). TEIP was one of two main activities under Outcome 1, namely strengthening the capacity of MOE at all levels to better plan, monitor, and evaluate their primary education activities and respond to ongoing education reforms. Please refer to Annex 2 for the full QBEP results hierarchy. TEIP rationale A TEIP-specific rationale was not evident in the project documents available. However, within the broader QBEP context, an overall rationale for addressing township level capacity stemmed from a situation analysis that found the education governance structures in place in need of capacity development and found that institutions at different levels needed strengthening.6 In particular, a stakeholder analysis found that while TEOs were ‘nodal’ points in basic education delivery, they had limited capacity or training to carry out their roles, which were also recognised as being too broad for the limited number of available staff to cover.7 The midterm review found the concept of TEIP to have been introduced in a ‘vacuum’, whereby, "planning was not a function of township education offices".8 In the transition from MDEF 1 to QBEP, an annex document noted that the Department of Education Planning and Training (DEPT) had drafted a format for 'Township Education Development Plans'.9 However, no further reference to this activity was found and this was not included in any known rationale for TEIP. TEIP objectives TEIP was intended to be a key instrument for the decentralisation of education planning and management in Myanmar with the following stated objectives10:

To enable achievement of the long-term national education plan through its operationalisation at the township level;

To enable townships to plan and implement initiatives for the improvement of township education services according to the local conditions and needs;

To address disparities in education service provision at the township and sub-township levels through micro-planning processes; and

To provide guidance and directions to schools for their school planning processes. The anticipated benefits of TEIP were stated as being: greater ownership and leadership of townships in education planning and development process; greater equity in education provision within and between townships; improved effectiveness and efficiency in addressing the diverse needs of townships in their education service delivery; greater budget predictability through multi-year planning and budgeting processes; and enhanced accountability for education sector performance at all levels of the system. According to the TEIP manual, the planning, review/approval, and implementation process was meant to be categorised into seven phases:

1. Plan preparation 5. Plan implementation and progress reporting;

6 UNICEF, QBEP Design Document, 2012, p. 7. 7 UNICEF, MDEF 2 Linked Annexes, 2011, p. 122. 8 MDEF, MOE, UNICEF, QBEP Midterm Review, 2014, p. 24. 9 UNICEF, MDEF 2 Linked Annexes, 2011, p. 34. 10 UNICEF, TEIP Manual Version 5, 2014, p. 7.


3

2. Draft plan review and revision process; 6. Annual report and review; and 3. Revised plan review and approval; 7. Annual work planning and budgeting. 4. Plan dissemination and communication;

For more detail on TEIP manual development and content, please refer to the section on 'TEIP manual' below. TEIP theory of change The 2012 QBEP design document did not present a specific theory of change for TEIP (nor for any of the other QBEP components) and the midterm review recognised that the, "paucity of basic information about the [QBEP] programme and its constituent components"11 was an overall weakness of QBEP, which was found lacking in direction. As noted above, UNICEF responded to these findings by revising the overall QBEP theory of change, as well as by clarifying the intended linkages between revised Outcome 1 activities. This clarification can be conceptualised as follows: In spite of this effort, there was still no specific logical framework with clearly defined inputs, activities, outputs, outcomes, or impacts developed for TEIP and neither does there appear to have been an assessment of assumptions and risks specific to TEIP. Nonetheless, a more unique theory of change was retroactively developed for the purpose of the TEIP evaluation and, as outlined in the TOR, is conceptualised as follows: This logic appears to be limited to equipping TEOs with the skills necessary to develop TEIPs; it does not address the skills or funding necessary to actually implement the plans. This therefore does not align with one of the stated objectives of TEIP which is to "enable townships to plan and implement initiatives for the improvement of township education services according to the local conditions and needs." This misalignment has proved disruptive to project success as different stakeholders have demonstrated

11 MDEF, MOE, UNICEF, QBEP Midterm Review, 2014, p. 21.

IF QBEP provides evidence-based advocacy to inform and support government and civil society efforts to undertake joint education sector reforms and help strengthen MOE capacity at all levels to better plan, monitoring, and evaluate education activities and implemented those reforms; THEN QBEP will strengthen the systems supporting quality education in Myanmar.1

IF TEOs are provided with training and technical guidance on how to: (1) solicit input from a variety of local and national stakeholders on the primary education needs of their townships; (2) utilise that information to develop a situational analysis of the education needs of their townships; and in turn (3) develop a three-year TEIP for responding to those needs and priorities; THEN TEOs will be better equipped to engage in a more inclusive and transparent planning process responsive to local needs and priorities and to strengthen the management of primary education at the township level; BECAUSE the current limited capacity, coordination, and collaboration within TEOs and vertically to higher level education governance structures is restricting the level of effective planning, management, and delivery of quality basic education at the sub national level.


4

different interpretations and understandings of the project's objectives. It must be noted that in spite of this stated objective, no TEIP has yet been implemented. The stated objective of implementing TEIPs appears to have been based on multiple assumptions that were untested and unsupported by a political economy analysis. One assumption was that the training provided through TEIP activities was matched to the needs and existing skills level of TEOs. However, although a workshop was held to inform the development of the TEIP manual, no known documentation was produced that specifically recorded what TEO needs were, nor how these would feed into the design of TEIP training activities. Other assumptions were that the decentralisation process would be sufficiently robust for township level offices to act upon their plans and, most importantly, that funding would be available for the plans to be actioned. Project activity description According to the linked annexes, under MDEF 1 some steps were taken to address township management and monitoring issues, and township management committees (of TEOs/Assistant TEOs and some cluster heads) were formed in core townships and were supported in undertaking orientation workshops, monitoring and data collection. However, capacity development provision was considered fragmented and focused only on programme needs and procedures.12 Under QBEP, the movement toward decentralisation meant that activities were intended to support more sustainable improvements in education development at the township level, particularly through more comprehensive township-planning. The QBEP design document states that TEIP activities were intended to focus on TEOs and Assistant TEOs by providing training and support in how to undertake a situational analysis of education in their townships, as well as teaching skills for how to consult with a range of agencies and stakeholders and facilitating community participation. The intention was to focus on developing Township Education Plans based on their analyses, where possible incorporating not only formal education but also early childhood development, non-formal education and monastic education activities.13 The TEIP manual states that the scope of TEIP was to cover early childhood care and education as well as primary, middle, and high schools, and non-formal education. It was to include government schools as well as non-state education providers including ethnic, monastic, private and NGO schools. The TEIP process described in the manual starts with township education offices establishing a TEIP Development Committee, each with three working groups including a Planning Working Group, Financing Working Group and Coordination Working Group and including TEOs, Assistant TEOs, head teachers and cluster heads. Based on the outline provided in the TEIP manual, TEIP activities were designed in a manner that required significant communication and cooperation between township, district, and state education offices. Documents, reviews, comments and approvals were intended to flow both upwards (from township to district to state) as well as downwards. In preparation for the development of a TEIP, TEOs were meant to conduct a stakeholder analysis and situation analysis. They were meant to attend training workshops and consult with school heads. The indicative time frame outlined to develop a first draft TEIP was five to six weeks. After drafting the TEIP, TEOs were meant to submit their plan to the District Education Office (DEO) for review and comment. TEOs were then responsible for incorporating comments; the whole process of review and revision was

12 UNICEF, MDEF 2 Linked Annexes, 2011 p. 34. 13 UNICEF, QBEP Design Document, 2012, p. 25.


5

intended as one month. The second draft TEIP was then meant to be sent by the DEO to the State Education Office (SEO) for review and comment. The DEO was responsible for finalising the TEIP while the SEO was responsible for ultimate approval. The timeframe for this stage was intended as two to four weeks. The total timeframe anticipated to reach a completed and approved TEIP was outlined as 14 weeks. After State approval of TEIPs, TEOs were meant to communicate the plan to major stakeholders, including the community. After which, plans were intended to be implemented and its activities monitored. Key outputs were intended as an annual operational plan and quarterly TEIP monitoring reports. The annual work plan was meant to be revised based on TEIP implementation and the plan and annual budget were to be approved the State Education Department. The role of key stakeholders The MOE was the main implementing agency of the TEIP project. Their roles included the following responsibilities, but did not include provision of funding for TEIP implementation:

1. Establishing a team of TEIP workshop facilitators to facilitate three workshops for each of the 34 townships;

2. Organising three workshops for each of the 34 townships, during which training was given on the development of TEIPs and draft TEIP plans were reviewed;

3. Forming a TEIP committee at each of 34 townships; 4. Drawing up TEIPs; 5. Reviewing draft TEIPs by TEIP workshop facilitators; and 6. Revising and resubmitting TEIPs.

UNICEF's intended role in TEIP was to assist MOE counterparts in planning and organising activities; providing them with technical assistance; monitoring TEIP activities; and providing advocacy targeted at the DBE subnational level education offices. In practice, UNICEF was also responsible for employing a consultant to develop a manual for TEIP, co-facilitating three workshops for each of the 34 townships and providing funding of approximately US$2,500 per location, primarily for facilitating the three workshops for TEIP committee members. UNICEF did not provide funding for implementation of TEIP. UNICEF staff involvement in TEIP included an Education Specialist and Project Manager, and a national Education Officer, as well as one national consultant hired in 2014. Contributions from other stakeholders included involvement of focal persons from central MOE in planning and organising activities, and who were also responsible for financial documentation. The DBE Head Office and State/Region Education Departments provided approximately 12 members of staff to take responsibility of facilitators. Two to four staff members from Township Education Departments (TEDs) and three to four head teachers per township were involved in the TEIP process as TEIP committee members. One member of the Township Development Assistant Committee (TDAC) participated in the TEIP process for some townships. Each TEIP committee consisted of between seven and 19 members per one location, and each committee spent between 10 to 35 days developing their township education improved plan. Derivation of the total human resources used for all 34 townships ranged from 238 to 646 days. Project expenditure QBEP's estimated expenditure on all 34 TEIP townships was US$85,000. In addition, TEDs other materials such as stationery, equipment and office space to development of TEIP. At the time of this report, a total of approximately US$108,178 was reportedly spent on TEIP over the course of QBEP.14

14 From UNICEF correspondence; evaluation team did not have access to budget data to confirm this.


6

TEIP manual A TEIP manual was first developed in 2013 by a consultant employed by UNICEF. In early 2013 the consultant conducted a situation and document analysis, which then led to a TEIP manual writing workshop held in Mon State, with the participation of MOE State and District Education Office officials from all ten townships. Prior to the workshop, the consultant also engaged with officials from the DBE and the DEPT. The manual outlines the specific TEIP objectives and also provides operational guidelines as to how to carry out project activities. It is reported to have been revised six times along the course of TEIP roll-out, although it should be noted that only Version 5 was provided to evaluators in spite of their requests to receive all versions. The 'organic' nature of the manual was intended to be responsive to changing conditions and to allow for adjustments of activities in order to incorporate lessons learned. While this is a valid intention, poor document management meant that changing iterations and their rationales were not recorded, resulting in ambiguous and confusing project objectives. Implementation status TEIP was implemented in a phased-manner. The 34 participating townships were allocated into 10 batches. At the time of evaluation, the project had been underway for two years and five months, during which 25 out of 34 townships had completed the development of a second draft TEIP and all had completed a first draft TEIP. The evaluation included townships that started the process as the fourth batch, the fifth batch, the eighth batch and the ninth batch to maximise lessons learned (See Table 1). Table 1: Townships that had completed development of TEIP at the time of evaluation (December, 2015)

Name of township, state/region Batch Status15 Kyaikto in Mon State First May 2013

Thaton, Bilin and Paung in Mon State Second August 2013

Mawlamyine, Kyaikmayaw and Chaungzon in Mon State Third November 2013

Mudon, Thanbyuzayat and Yae in Mon State Fourth November 2013

Myebon in Rakhine State Fifth August 2014

Kawkareik in Karen State Sixth October 2014

Yebyu, Dawei and Thayetchaung in Tanintharyi Region Seventh November 2014

Hlaing Thar Yar in Yangon Region, Pya Pon in Ayeyarwaddy Region and Htain Ta Bin in Bago Region

Eighth January 2015

Muse, Namtu, Namsang, Pinlaung, Kengtung, Monghpyak and Tachileik in Shan State

Ninth April 2015

1.2 Context of basic education in Myanmar

Myanmar has an estimated population of 53 million people16 comprising 135 officially-recognised ethnic groups speaking a total of 111 languages. The majority ethnic group are Bamars, with seven other main ethnic minority groups: Chin, Kachin, Kayah, Kayin, Mon, Rakhine and Shan. The most recent Census results suggest that 70 per cent of the population live in rural areas and 30 per cent in urban areas. Ethnic states tend to have higher proportions of rural dwellers and many of the smaller ethnic groups are concentrated in the country's most remote and economically marginalised border areas, where livelihood strategies involve migration, agriculture and dependence on children's contribution to household incomes. These patterns, which are particularly pronounced in remote ethnic regions, impact on children's opportunities to access and benefit from quality education. There is also is deep inequity in access to different types of education provision and children of the poorest and most remote

15 Month and year indicated were not exact time as a key informant revealed that he could not recall them exactly. 16 2014 National Census data.


7

communities, especially in ethnic areas, are more likely to be served by multi-grade schools and have less opportunity to progress to secondary education.17 Since the previous Myanmar government formed in March 2011, education reform has been a national priority. Expenditure on education increased from 0.7 per cent of gross domestic product in the 2010/11 fiscal year to 2.1 per cent of GDP in the 2013/14 fiscal year, with approximately two thirds spent on basic education.18 A number of advances in policy and strategic priorities have been underway including a Comprehensive Education Sector Review (CESR), which began in July 2012 in collaboration between the Government of Myanmar and development partners, in order to identify the challenges, gaps and strategic options of the education sector more effectively. The Framework for Economic and Social Reform (FESR) is another key national planning document part of the educational reform and was released in December 2012. In this document, four policy priorities were outlined: (1) sustained industrial development to catch up with global economies; (2) equitable sharing of resources among regions and states and promoting foreign and local investments for regional development; (3) effective implementation of people-centred development to improve education, health and living standards; and (4) reliable and accurate gathering of statistical data and other information for better public policy decisions.19 Of these, policy priorities two and three promote a decentralised planning and implementation process. Details on how decentralisation was to be accomplished and within what timeframe were not clear in the FESR or other documents, although one move was to create a new district level of administration in 2012. In theory, this should have shifted decision-making responsibilities up from the township level and down from the Union level.20 However, the distinction of responsibilities between state/region-level administration and union ministries remains ambiguous. At the time of QBEP and TEIP development in

17 UNICEF, QBEP Design Document, 2012, p. 5. 18 MOE, CESR Phase 2 Report, MOE, Nay Pyi Daw, Myanmar, 2014b. 19 MOE, National EFA Review Report, MOE, Nay Pyi Daw, Myanmar, 2014a. 20 Zobrist, B. and P. McCormick, A Preliminary Assessment of Decentralisation in Education: Experiences in Mon State and Yangon Region, The Asia Foundation, 2013.


8

2012, the exact nature, direction and future of decentralisation was unknown but it was nonetheless considered an important government strategy to support.

Another indication that education is recognised as a national priority is Section 28 of the Constitution of the Government of Myanmar, which guarantees free and compulsory primary education. Myanmar became a State Party to the Convention on the Rights of the Child (CRC) in 1991 and to the Convention on the Elimination of All Forms of Discrimination (CEDAW) against Women in 1997 and has a body of national laws committing the State to the realisation of children’s and women’s rights. Myanmar is also committed to Millennium Development Goal 2 (now Sustainable Development Goals) and the Education for All (EFA) goals21 through its EFA Plan of Action (2003-2015). The table below outlines some basic education indicators. Table 2: Basic education indicators in Myanmar (Source: National Census, 2014)

Education data Number

Total population 54,246,096

Child population under 18 > 21 million

Literacy rate (persons aged 15 and over) 89.5%

Male literacy rate 92.6%

Female literacy rate 86.9%

Net primary school enrolment 87.7%

Attendance of children aged 5-9 71.2%



Number of primary schools 28,967

21 The six international EFA goals have been slightly re-worked into four Myanmar EFA Goals, by combining the literacy and

lifelong learning goals, combining the access and quality goals, addressing of gender as a cross-cutting issue rather than a

separate goal, and the addition of a specific goal on education management and EMIS. However, internationally Myanmar

reports on all six global goals.

Decentralisation structures in Myanmar Within the structure of Myanmar’s government there are numerous subnational levels including state/region governments, districts, townships, towns and village tracts/wards. Many Union-level ministries have officials at one or more of these levels. The township level of government is a key point of interaction for the majority of Myanmar people. Within the MOE, planning, budgeting and decision making are highly centralised. Following the subnational structure outlined above, the MOE administrative units are State/Region Education Departments in the 12 states and regions, District Education Offices in all districts and Township Education Offices in all 325 of the country's townships. The township level education offices are the focus of TEIP. As noted in a 2013 Asia Foundation report on decentralisation and education in Myanmar, despite the moves toward decentralisation, directives are still centralised and the authority to implement work is very much a flow from the Union level MOE down toward the township offices, with little communication or feedback flowing from the townships upwards. Decades of centralisation mean that without authorisation from the top levels of MOE, lower level administrative units are not well equipped to act. From the time of QBEP and TEIP conception until present, an exact interpretation of decentralisation, as envisaged by the Union Government, has not been clear. Globally, the three major forms of decentralisation are recognised as deconcentration, delegation, and devolution. From evidence of efforts so far, the form that appears most applicable to the Myanmar context is deconcentration, where broadly understood as a shift of responsibility away from the centre and developing administrative capacity beyond the central government (although still liable to be controlled by it).

Figure 1: Overview of decentralisation structures in Myanmar


9

2. EVALUATION PURPOSE AND OBJECTIVES

2.1 Evaluation purpose, objectives and intended audience

In 2015, UNICEF contracted Montrose to undertake an independent end of project evaluation of the TEIP project. Montrose undertook this contract in partnership with EMPOWER, a Myanmar consultancy group. The call for evaluation was in response to the midterm review, carried out in 2014, which found that QBEP would benefit from more independent evaluation of their projects and specifically stated that doing so would strengthen the credibility of QBEP successes.22 The review recommended that QBEP should review its monitoring, evaluation, and knowledge management strategy with a more, "rigorous, reflective, and critical approach" to M&E.23 Evaluation of the TEIP project should also technically feed into QBEP Output 2 of strengthened learning from assessments, evaluations, and studies, which in turn should support Outcome 2 of an improved evidence-base for advocating and delivering quality basic education. The midterm review recommended a cautious approach to further QBEP expansion and in response UNICEF stated that activities that do not show evidence of being in line with NESP will be phased out.24 Therefore, this evaluation was also intended to assess whether TEIP was suitable and ready for nationwide scale up. The stated purpose of this end of the project performance evaluation, per the TOR, was to assess TEIP as a QBEP funded activity, strengthen its ongoing implementation, and learn lessons for future application. The evaluation was intended to examine the relevance, effectiveness, efficiency and likely sustainability of QBEP’s TEIP activities to date, identify lessons learned, and formulate recommendations for future programming. The evaluation was therefore both summative and formative in nature. It is anticipated that the evaluation will be able to support the new government in deciding their education priorities as well contribute toward a decision on whether to continue QBEP support for decentralisation reform. Specifically, the evaluation had the following specific objectives:

To test programming assumptions and the underlying theory of change;

To identify areas where implementation can be strengthened under QBEP for possible expansion under the NESP;

To explore a finding from the QBEP midterm review questioning the relevance of its TEIP activities; and

To respond to an overarching mid-term review recommendation that QBEP assess its activities before scaling up.

The primary audiences of the evaluation are intended as UNICEF and MDEF partners. Relevant MOE officials at the central level and relevant MOE officials at state, district and township levels are also expected to use the evaluation findings.

2.2 Evaluation criteria and scope

The evaluation sought to provide an independent assessment of QBEP’s TEIP activities based on the Organization for Economic Cooperation and Development's Development Assistance Committee (OECD/DAC) criteria listed in the evaluation questions. Given the nature of TEIP as mainly a capacity building project in non-contentious areas and not in a period of disaster, OECD/DAC criteria were more suited to the object of evaluation than other human rights or humanitarian based criteria frameworks.

22 MDEF, MOE, UNICEF, QBEP Midterm Review, 2014, p. 21. 23 Ibid, p. 6. 24 UNICEF, JPIP, 2014, p. 10.


10

Per the TOR, the evaluation was not intended to examine impact or higher level results; this is because a two and half year of implementation was not sufficient to produce higher level results. Per DAC interpretation of 'impact' however, where possible, unintended consequences (both positive and negative) have been noted. The primary focus of the evaluation was to highlight lessons learned and draw on these for the modification of any future TEIP iteration or similar decentralised planning modality. The evaluation scope includes a review of the TEIP process throughout all steps of TEIP development, as outlined in the TEIP manual, including:

1. Plan preparation 5. Plan implementation and progress reporting; 2. Draft plan review and revision process; 6. Annual report and review; 3. Revised plan review and approval; 7. Annual work planning and budgeting 4. Plan dissemination and communication;

The evaluation's lines of enquiry were based on these objectives, which include implementation of TEIPs. However, UNICEF have since stated that the operational portion of the manual was only meant to be achieved once TEIP was institutionalised.25 As noted in the section on Education in Myanmar, an exact decentralisation pathway was not made explicit by the Government and over the course of QBEP it became apparent that the extent of decentralisation was not as developed as had initially been anticipated. This has been suggested by UNICEF as a reason for the shift away from the outlined objectives of TEIP implementation per the manual, toward a purely capacity building exercise without implementation. Evaluators recognise the changing context against which TEIP was rolled-out and recognise UNICEF's efforts to adapt to the needs and environment accordingly. However, as the changes in project objectives were not recorded and the manual objectives were not modified, evaluators had no verifiable source to confirm these statements or against which to base their methodology; therefore, all stated stages remained a part of the evaluation. As TEIP has been rolled out in various townships using a phased approach, the sampling strategy was requested to include townships at each phase of TEIP development. In addition, the TOR also required the team to conduct a comparative assessment of other subnational education planning activities through collecting data from both QBEP and non-QBEP townships, including townships considered ‘worst off’. The TOR of the evaluation set out three major evaluation questions and 11 sub questions:

1. Are TEIP activities relevant to the planning needs of Township Education Officers (TEOs) to

manage the delivery of basic education in their townships?

a. Did the activities respond to the capacity needs of TEOs? b. Are the QBEP-supported TEIPs responsive, equitable and inclusive of their respective

township education needs? c. Are there other systems or initiatives currently in place that are more relevant in meeting

this planning needs?

2. To what degree are TEIP activities effective and efficient in developing education plans that are inclusive, informed, and realistic?

a. Are the activities producing the results originally intended? b. Are there are unintended consequences (positive or negative) resulting from the

activities?

25 Feedback from UNICEF on first draft of TEIP Evaluation Report.


11

c. Are there any particular groups currently not being reached through the TEIP activity? d. Are resources dedicated to TEIP sufficient to reach intended groups and desired results? e. Are there any ways in which the activities can be improved, either through changes in

their current implementation or based on good practices from similar activities in Myanmar and/or elsewhere in the region?

3. What elements of the TEIP activities should be sustained as part of NESP following the end of the current QBEP funding?

a. Is capacity available for the TEIP activities to continue to be carried out at the township level following the end of QBEP?

b. To what extent can TEIP activities support the development of district or state level education improvement plans?

c. Is the current design for supporting TEIP activities appropriate and viable for replication in other townships – and which specific elements, if any, need to be strengthened? Are there any other Government of Myanmar or non-government partners that need to be involved?

There were certain constraints in being able to address these evaluation questions due to the lack of project data and disorganised document management systems within UNICEF. The first evaluation questions hinge on the basis that the needs of TEOs were known; however, UNICEF was not able to provide evaluators with any needs assessment report or other source that outlined what TEO needs were. In the absence of specified indicators from project documents, the criteria for evaluation were developed by the technical team lead and were based on lines of enquiry per OECD/DAC recommendations. Project documents were used as much as available to determine evaluation criteria, but it must be noted that that the only document specific to TEIP provided to the team at the start of the evaluation was the TEIP manual Version 5. Other documents were provided after first report submission, but these could not inform the development of criteria and this is recognised as a limitation. The TEIP manual did provide some benchmarks for determining indicators of effectiveness and relevance. For instance, the five objectives outlined in the manual were used as benchmarks against the project’s intended results and the intended stakeholders listed in the manual were used to measure participation of key stakeholders and linkages of the TEIP with other programmes and projects. Per UNEG report guidelines, evaluation criteria and subsequent data collection, analysis, and reporting included considerations of equity and gender equality for relevance, effectiveness, efficiency and sustainability of the TEIP project, as appropriate. This covered whether, how, and to what extent the specific education needs of different genders and marginalised groups (such as various ethnic groups, diverse religious groups, socially marginalised groups such as children infected and affected by HIV, children with four types of disability, children of women headed households and children from migrant families) were addressed by the TEIPs. Review of the QBEP design document and linked annexes suggests that human rights principles did not appear to explicitly inform the design of the QBEP programme or TEIP project; however, in keeping with UNICEF understandings of human rights principles,26 the evaluation criteria nonetheless included considerations of participation and inclusion. The extent to which the capacity needs of TEOs and other MOE staff members were fulfilled was examined by looking into disaggregate analysis by gender. It should be noted that QBEP as a whole only provides indicators disaggregated by gender for some of its activities, a point that was also recognised by the mid-term review; there were none provided for TEIP meaning that there were no programme or project benchmarks against which to assess findings on gender.

26 UNICEF, The State of the World's Children, Annex B The Human Rights-based Approach: Statement of Common Understanding, 2004, p. 91.


12

The lines of enquiry and criteria developed by evaluators were actively reviewed and revised by key staff members of UNICEF and MOE through a series of workshops and Reference Group meetings. The participation of these stakeholders, particularly the TEIP workshop facilitators and TEOs from the six sample townships, greatly strengthened the relevance and usefulness of the finalised evaluation questions and criteria. UNICEF guidelines on equity, equality and inclusivity as outlined in institutional material provided to evaluators were used to inform lines of enquiry. The evaluation matrix, along with full lines of enquiry is listed in Annex 3.

3. METHODOLOGY

3.1 Overall approach

In line with the TOR and above evaluation questions, the evaluation study used a summative process evaluation for verification approach to assess TEIP activities to date. This summative evaluation approach aimed at providing UNICEF and MDEF partners with an assessment of outcome and output level results of TEIP. A formative process evaluation for modification investigated the strengths and weaknesses of the activities implemented in order to identify lessons that can be used to improve any future replication of TEIP. The evaluation employed a quasi-experimental design known as ‘Static Group Comparison Design’27 in which data collection was undertaken at intervention sites and comparison sites only at the end of the project (post-test design). Given the absence of a control group in the original project design, a true experimental evaluation design showing causal effect of project activities was not possible; however, the selection of a quasi-experimental comparison site offered value by allowing observation of differences between intervention and non-intervention sites. These observations could then form the basis for assumptions of project influence and can also highlight areas that require further exploration. Ideally, this design requires selection of the same number of intervention sites and non-intervention sites; however, given the limitations in time and budget, six non-randomly selected intervention sites and only one non-randomly selected comparison site were selected. This study design still enabled the evaluation study to discern the Strengths, Weaknesses, Opportunities and Limitations (SWOLs) of the TEIP programme in order to comment on the commonalities and differences between TEIPs and alternative programmes, as required by the TOR. To correspond to the study design, a sequential mixed methods approach consisting of three phases was applied to the design and implementation of the study. The diagram below outlines how this approach uses qualitative and quantitative data collection.

27 Hawthorne, G. Introduction to Health Program Evaluation, Melbourne, Victoria, Australia, 2000.


13

Figure 2: Outline of sequential mixed methods approach28

Before designing the quantitative tools, an initial exploratory qualitative phase was conducted. This consisted of a document review as well as data collection through focus group discussions, in-depth interviews and key informant interviews with TEIP committees and teachers, parents and students of the government primary schools, middle schools and high schools in one sample catchment of Hlaing Thar Yar Township in Yangon Region. In the absence of baseline data, consultative meetings were organised throughout this phase to understand the central issues of the evaluation and, in particular, to determine the needs of TEOs. This initial phase informed the development of context-appropriate tools and approach for the second phase of data collection. This was especially important given the minimal level of M&E data available, as acknowledged by UNICEF.29 These preliminary activities allowed evaluators to identify key terminologies; formulate appropriate and accessible wording, phrasing and tone for the structured questionnaire; identify the languages required for subsequent interviews; as well as to recognise and prepare for operational challenges, recruitment of interviewers and logistical issues for the following phases. The second phase consisted of quantitative data collection aimed at gathering quantified evidence of relevance, effectiveness, efficiency and sustainability through structured questionnaires. These findings fed into the third phase of explanatory qualitative data collection, consisting of focus groups, in-depth interviews, and key informant interviews with key stakeholders of two TEIP townships (Tachileik and Yae) and one non-TEIP township (Lashio). This phase allowed evaluators to draw out explanations and elaborate on the quantitative data analysis and to formulate recommendations for future programme design and strategies.

3.2 Data collection

Qualitative data collection tools Qualitative tools were developed by the evaluation team through the inputs from the key UNICEF and MOE stakeholders who participated in the pre-fieldwork design workshops held by evaluators, in addition to the content of the TEIP manual, and other project documents identified in the course of the inception

28 Creswell, J.W., Research Design: Qualitative, Quantitative and Mixed Methods Approaches, 2nd Ed., SAGE, Thousand Oaks,

California, USA, 2003. 29 UNICEF, TEIP Terms of Reference, 2015, p. 3.

Quantitative Phase to determine

magnitude of REES


14

phase.30 Draft tools were shared with the key stakeholders, including UNICEF’s Evaluation Manager, the TEIP Reference Group, and TEOs during the inception phase. Feedback was incorporated into finalising these instruments. English tools were translated into Myanmar and then back-translated into English to ensure accuracy of questions. Quantitative tools During the inception phase, a structured self-administered questionnaire was developed by the evaluation team and shared with key stakeholders. Feedback from key stakeholders was incorporated into a revised second draft version. Findings from focus groups and interviews from the first sampled intervention township were used to refine a third draft. The third draft questionnaire was pre-tested and findings of the pre-test were used to finalise a structured questionnaire. The questionnaire covered outcome, output, and process level information needs. The rationale for using a self-administered in-person questionnaire was that:

This method is relatively less costly than using enumerators;

The education level of the intended respondents was of a sufficiently high level to elicit responses with a satisfactory level of quality and validity to a certain extent; and

Responses can be elicited within the relatively shorter time frame. However, it should be noted that the method itself has certain drawbacks as participation rates can be lower than face-to-face interviews and answers may be liable to being less complete, precise, or valid. To mitigate this, the evaluation team arranged a four-hour workshop at each of seven sampled townships to explain the questionnaire to representatives of randomly selected schools and to make necessary arrangements to complete the survey. For a full outline of the number of interviews and respondents that took part in qualitative and quantitative data collection please refer to the detailed methodology section under Annex 4.

3.3 Analysis methods

Qualitative data Qualitative data analysis followed a sequential and iterative process and sought to classify analysis by theme, content and question31. The field team kept a field record during data collection that contained key points of interest made by participants, comments raised, evaluators' spontaneous interpretations and impressions, emerging trends and themes, probing questions, and notes for new issues to be explored in further interviews. Qualitative data analysis was conducted in three phases: (1) preliminary analysis; (2) secondary analysis; and (3) final analysis (see details in Annex 4). Thematic analysis was applied and ATLAS-ti software was used to code the qualitative data in accord with themes and sort themes. From these categories of data, links, causes and consequences were drawn out. Quantitative data SPSS version 16 was used for data entry, cleaning, and analysis. Data were examined for distribution of continuous and categorical variables for verification and cleaning first. Summary measures were sought for each of the continuous variables to examine the mean, standard deviation, minimum and maximum values and percentiles. The tabulate command was used to examine whether categories were permissible, frequencies were reasonable, and data were consistent for categorical variables. Descriptive analysis using only one variable was performed between factors such as improvement of the needs of the education sector from two years prior to TEIP and during two years of TEIP.

30 It should be noted that project documents were provided to evaluators in an incremental fashion, some even after

submission of the first draft, meaning that some opportunities were lost to coordinate benchmarks and indicators of progress. 31 Morse, J. M. and Field, P. A. Qualitative research methods for health professionals, SAGE, Thousand Oaks, California, USA.

1995.


15

3.4 Data sources

A range of data sources were used in the course of the evaluation. During the Inception Phase secondary document sources included material provided by UNICEF, such as the TEIP manual, available QBEP programme documents, and reports by the TEIP manual design consultant. These were used to inform the evaluation design. The absence of key data meant that primary sources in the form of MOE officials, TEOs, and TEIP facilitators were used to fill in gaps in project knowledge. The limitations of using primary sources for this purpose are recognised as imperfect recall and the potential for differing interpretations. During the course of fieldwork, the team reviewed 11 TEIPs developed by township committees to assess their performance against the criteria stated in the manual. The findings were used to cross-check responses from interview participants. A mix of stakeholders were consulted and/or participated in the evaluation study and captured a diversity of perspectives. The aim of using this diversity of sources was to capture data from multiple identified TEIP stakeholders to ensure representative and accurate data and to allow for data triangulation of data across sources. The human sources used and the rationale for their selection are outlined in Annex 4. In summary, a total of n=939 participants took part in n=461 focus groups, interviews, participatory workshops, general interviews, as well as school and TED surveys. Table 3: Number of interview sessions and surveys conducted and number of participants involved in data collection (October to November 2015)

Method Sessions/surveys Participants

Focus group discussions 21 371

In-depth interviews 14 16

Participatory workshop 7 60

General interviews 35 108

School surveys 367 367

TED surveys 17 17

Total 461 939

3.5 Sampling strategy

Qualitative data collection First stage sampling of Townships Given the limited time and budget available, six TEIP sites and one comparison site were non-randomly selected for study. A maximum variation sampling method was applied to identify different townships with diverse operational contexts across five states and regions. Commonalities and differences of Strengths, Weaknesses, Opportunities and Limitations identified from these diverse locations were used to maximise the value and appropriateness of lessons learned for wider states and regions of Myanmar. From a sample frame of 25 QBEP townships that had completed TEIPs (at the time of evaluation), six townships from Shan, Rakhine, Mon States and Bago and Yangon Regions were chosen in line with UNICEF principles of equity and inclusivity. Accordingly, criteria such as hard to reach areas, areas with most vulnerable populations and areas offering ethnic and religious diversity were applied (for full details of selection rationale, please refer to Annex 4). For the comparison site, Lashio Township from Shan State was chosen because of its similar characteristics to Tachileik Township. Data collected from the comparison site was compared to the data from the corresponding intervention site.


16

Second stage sampling by catchment area In the second stage of sampling, purposive methods, using criterion-based sampling, were employed to select one catchment area in all TEIP townships and non-TEIP townships (Patton, 1987). A catchment area is a geographical area where students from its primary schools will continue to study in its middle schools, and where students from its middle schools will continue their education in its high schools. The number of schools per one catchment area in the seven evaluation townships ranged from four to 23. Criteria developed and agreed upon by key stakeholders during the planning phase comprised of difficulty in transportation, remoteness and poverty, which were used to categorise catchment areas of each township into three strata: (1) relatively well-off; (2) typical; and (3) relatively poor. In all seven sample sites, the majority of schools and students in a township were considered ‘typical’. Under each township, one of the areas considered ‘typical’ was chosen as a sample catchment area. To examine the issue of equity, a catchment area located in a rural setting was also selected. Students, parents and teachers from as many schools as possible in one catchment area were chosen, in order to capture their views and experiences of TEIP as well as the varied education needs amongst schools. Both men and women participated in focus groups. Sample selection and rationale is detailed in Annex 4. As the typical stratum covers the majority of schools and students of each of the sampled townships, if TEIP is found to be successfully implemented in a typical stratum, the evaluation will be able to draw the conclusion that TEIP can provide benefits to the majority of schools and students in each township. In addition, the education needs of community members and capacity needs of education staff members identified in the catchment area of the typical stratum would likely be similar to those of the broader populations in each township. Thus, the needs identified in catchment areas would form the basis for future TEIP programming or other similar decentralised planning modalities. Random sampling for quantitative data collection methods The sampling frame for the quantitative survey included all schools within the six intervention and one comparison sites. All schools were stratified by catchment area, which was a collection of primary, middle and high schools in one geographical location. The required sample size per township was divided by the number of catchment areas of each township to compute a required number of sample schools in each stratum. From a list of all schools within each stratum, sample schools were randomly selected by using Statistical Programme for Social Science (SPSS). Each of the seven townships had a list of all schools broken down by catchment area, which enabled the evaluation study to derive a complete sampling frame and an accordingly methodologically sound random sample of schools. Availability of the complete enlisting of schools of each township facilitated application of this stratified random sampling successfully. The only limitation was a non-response rate of 2.1 per cent of selected schools or eight out of 375 selected schools. Limitations Overall, the team acknowledges that a sample of six townships is too small to be considered entirely representative; however, the maximum variation approach adopted by the team ensured that diverse conditions were captured, meaning that the findings should be indicative enough to draw broad conclusions for future programme planning. The sample selection did not include conflict sites or those with known human rights violations and therefore methods were more geared toward the more contextually relevant issue of equity and gender.

3.6 Ethical considerations

Montrose and EMPOWER implemented an ethical protocol in accord with the UNEG Ethnical Guidelines for Evaluations and UNICEF's Procedure for Ethical Standards for Research, Evaluation, Data Collection and Analysis to ensure that the highest ethical standards were adhered to in the course of the


17

evaluation.32 33 Privacy, anonymity, and confidentiality were the major ethical issues considered. The following measures were undertaken:

The plain language statement was translated from English to Myanmar. This statement outlined the objectives of and risks involved in the study, types of information and reasons for seeking such information, where and to whom participants could report ethical misconduct and could seek more information on the study;

The evaluation team members were trained on the following issues: importance of ethical issues in evaluation; how to seek informed consent, to provide assistance to participants, and to document standards of ethical conduct; and how to safeguard anonymity and confidentiality;

To minimise potential compromise of anonymity and confidentiality through participants of focus groups, the facilitator explained the importance of keeping information confidential and requested that participants help to ensure confidentiality. Names and addresses of participants in focus groups and interviews were not sought, and participants were not followed up after their interviews;

When each school or TED completed a questionnaire, the completed questionnaire was put into a sealed envelope;

Questionnaires and voice records were kept under security conditions while fieldwork was being undertaken. Hard copies were kept in lockers and soft copies were protected with passwords;

When a report was produced, identifiable information (name, position/title, etc.) was not revealed as much as possible to prevent readers from identifying informants; and

Informed consents were obtained from all respondents, and by caretakers of students interviewed.

3.7 Limitations and mitigation strategies

This study was constrained by several limitations with regard to evaluation design, sample townships, use of self-administered questionnaires, and application of qualitative methods. The study took measures to mitigate these drawbacks to the extent possible, within the available resources and other priori limitations, and the conclusions drawn from this study should still be considered valid. Suboptimal document management The evaluation concurs with findings from the midterm review regarding QBEP's, "relaxed attitude to programme documentation."34 What project documents do exist were provided to evaluators in a piecemeal fashion, extending beyond the inception phase; not having the full range of documents at the start of the evaluation may have affected the focus and direction of evaluation as evaluators had to work with incomplete information. However, in order to strengthen the report, documents provided after the first evaluation draft were still taken into consideration and their content has been assessed and referenced where relevant. Lack of standalone project documents Apart from the TEIP manual, standalone TEIP design documents do not exist but rather, are imbedded within a range of broader QBEP documents. The lack of design and M&E documents, such as a standalone log frame, regular update reports, or adequate needs analysis, challenged evaluators. Having to extract TEIP-relevant data from multiple documents made it difficult to form a comprehensive picture of the project rationale and objectives; this process was time consuming and does not contribute to effective M&E. The lack of a specific log frame or indicators was addressed by developing evaluation criteria based on the TEIP manual, benchmarks from the MOE and QBEP, and the expertise of the technical team lead.

32 UNEG, UNEG Ethical Guidelines for Evaluation, United Nations Evaluation Group, 2008. 33 UNICEF, UNICEF Procedure for Ethnical Standards in Research, Evaluation, Data Collection and Analysis, UNICEF, 2015. 34 MDEF, MOE, UNICEF, QBEP Midterm Review, 2014, p. 21


18

Lack of baseline data This challenge was acknowledged by UNICEF at the start of the evaluation and the team was prepared to employ alternative means in response to the lack of baseline data. The lack of baseline quantitative and qualitative data on education needs and capacity needs of TEOs meant that these had to be established through contemporary workshops with the MOE. The weakness of this approach is that it relied on recall, and the needs identified during these workshops may not accurately reflect the needs at the project start-point. However, in the absence of alternative data, the evaluation team had to use these findings to assess the relevance of TEIP activities. Other baseline measurements were reconstructed by asking targeted respondents (please refer to the section on Data Sources) to recall the situation at the beginning of the TEIP project by both quantitative and qualitative data collection methods. The absence of the baseline measurement prevented the study from calculating the extent to which improvements were made from the baseline status at the end of the project. However, as the evaluation objectives did not require an assessment of programme impact, this limitation was not as problematic as might normally be expected. Self-administered questionnaire method Self-administered questionnaires were used to reconstruct baseline data, which presented some drawbacks. Using this method, participation rates of the survey and response rates to questions of the questionnaire were expected to be lower than those from face-to-face interviews. This is because interviewers would not be able to explain the study's aims or clarify the meaning of questions of the questionnaire, or use human interaction to prompt respondents and counter response fatigue. Responses might be expected to be less complete, precise or valid because of the absence of an interviewer to clarify questions. The study addressed these potential weaknesses by arranging a four-hour workshop with all participating schools of each of the seven townships, in which the objective of the study and all items were explained and queries of respondents were answered by the study team members. The participation rate of the school survey was 97.9 per cent, which was much higher than anticipated. The participation rate of the TED survey was 68.0 per cent, which was as low as anticipated. This lower participation rate is likely due to the fact that the team could not travel to all 25 townships where this questionnaire was distributed to provide explanations of follow up support. Participant recall limitations The self-administered questionnaire sought to capture factors that related to education sector improvement over two periods: two years prior to the TEIP; and two years during the TEIP programme. This approach was constrained by participants' recall ability, as well as by the influences of the present situation on participants' perceptions of the past situation. Qualitative methods The limitations of employing qualitative methods only were offset by applying both quantitative and qualitative methods. Surveys were conducted for 375 schools in seven sample townships and 25 TEIP townships to verify self-reported data from two sources. Both surveys showed consistent findings for most of the issues under examination, which indicated that the weakness of self-administered questionnaires was offset to certain extent. Although qualitative data were collected from six TEIP townships and one non-TEIP township, the TED survey could obtain data from 17 TEIP townships, which compensated for the limited samples of the qualitative method. Eliciting data from 15 data sources also enhanced the validity and reliability of data. Validation of data and report was carried out with key stakeholders. These measurements enhanced the validity and credibility of the evaluation findings to an acceptable level despite the presence of some weaknesses.


19

Limited sample size The selected sample of six intervention townships could not be considered representative and the value of selecting a single comparison site was limited. Nonetheless, given the limited time and other resources available, this was the maximum sample size that this evaluation could accomplish. The selection of these sample townships was agreed upon by MDEF partners and other key stakeholders, including the MOE. As stated above, the TED survey obtained data from 17 TEIP townships, which mitigated the limited sample townships for qualitative data collection. Potential sample bias Data collected from purposively selected participants might not reflect the situation of other target populations who were not included in the sampling.

4. FINDINGS

4.1 Relevance

Are TEIP activities relevant to the planning needs of TEOs to manage the delivery of basic education to their townships? Did the activities respond to the capacity needs of TEOs?

Key weaknesses in TEIP project design and management limited the evaluation team's ability to accurately answer the question of whether TEIP activities were relevant to capacity needs of TEOs in order to manage the delivery of basic education to their townships. At the start of the evaluation, there was no evidence that a needs assessment had taken place to establish TEO needs, a point that evaluators enquired about in detail. Given that the evaluation is based on responding to the capacity needs of TEOs, this was a crucial piece of missing data. To address this shortcoming, the evaluation team held a series of workshops with TEOs and MOE officials during the evaluation inception phase, and drew on the information provided by these participants to reconstruct the kinds of capacity needs required by TEOs to manage and deliver basic education in their townships. These capacity needs were identified as the following areas:

1. Needs assessment; 2. Planning; 3. Monitoring and evaluation; 4. Developing sustainability strategies; and 5. Resource allocation and management.

Summary of key findings:

TEO capacity needs, as reconstructed through workshops, were partially addressed by TEIP activities. The needs addressed were in planning and needs assessments.

Operational capacity skills were not directly addressed through training activities but broader organizational skills may speak to them indirectly.

Strengthened capacity was found beyond TEOs only – skills were extended to Deputy TEOs, Assistant TEOs and male and female head teachers.

Female and male head teachers' perspectives were broadened beyond their individual schools.

Widely reported changed attitudes among committee members in terms of better understanding the importance of valid data.

Examples of committees applying skills learnt toward application for school grants, thus demonstrating demonstrable relevance of capacity building activities.


TEO capacity needs, as reconstructed through workshops, were partially addressed by TEIP activities. The needs addressed were in planning and needs assessments.

Operational capacity skills were not directly addressed through training activities but broader organizational skills may speak to them indirectly.

Strengthened capacity was found beyond TEOs only – skills were extended to Deputy TEOs, Assistant TEOs and male and female head teachers.

Female and male head teachers' perspectives were broadened beyond their individual schools.

Widely reported changed attitudes among committee members in terms of better understanding the importance of valid data.

Examples of committees applying skills learnt toward application for school grants, thus demonstrating demonstrable relevance of capacity building activities.


20

The evaluation team found that three workshops for training, development and revision of TEIP had been conducted for 25 townships and TEIPs had been developed by these 25 townships. Such activities were found to respond to the first two needs identified above. As has been discussed, no TEIPs have yet been implemented. As such, TEIP activities were not able to fully respond to the last three needs outlined above. However, through the training activities provided, it is possible that underlying skills in document preparation and awareness of data collection procedures will indirectly respond to these needs. It should be noted that, after submission of the first evaluation report draft in early 2016, it was clarified that a consultant employed by UNICEF to develop the TEIP manual in 2013 had conducted an assessment of education and capacity needs at the township level education department in Mon State. In spite of this discovery, the results of this assessment were not provided to the evaluation team and the actual findings of capacity needs at the township level remain unknown. Strengths and opportunities of the activities in responding to the capacity needs of TEOs In order to ascertain the strengths and weaknesses of the TEIP activities that were able to respond to the capacity needs of TEOs, findings from qualitative data collection35 were analysed. Almost equal numbers of male and female TEIP committee members from the six sample townships underwent capacity building training in how to undertake needs assessments and develop township education plans. Committee members reported finding the activities useful in teaching them the importance of having valid and accurate data to develop plans and of the necessity of undertaking a situation analysis before developing a plan. It also made them aware of the benefits of having a proper plan in place, so that when opportunities to access resources arose they would be prepared for how to use them and would be able to act in a responsive and timely manner. Previously, plans were only created after resources were made available thereby resulting in delays in any actual action. Committee members also reported acquired capacity in analysing trends in access to and quality of education, as set forth in the TEIP guidelines; in collecting and analysing data to create a township profile with particular emphasis on its socio-economic situation; in developing plans that include strategic objectives; in calculating budgets; in discerning SWOLs of the education sector; in formulating solutions for both their own and other townships; and in realising what resources were needed for enhancing their township level education sector. TEIP activities were found to have improved capacity beyond the target beneficiaries of TEOs; benefits were found to extend to other TEIP committee members who were the second line leaders for the township level management of education services. Previously, only the TEO was responsible for township level planning but as there was a high level of transfer rate of TEOs, they had neither sufficient institutional knowledge of their township nor adequate time to understand their township's needs or broader situation. The result was that TEOs would draw up township education plans that did not accurately reflect the requirements of their township. As one interviewee explained,

35 Participatory Reflective Appraisal (PRA) workshops with TEIP committee members, interviews with DEOs and SEOs of six TEIP townships, interviews with three TEIP facilitators, and interviews with three UNICEF officials.

"TEOs were transferred in and out frequently. While they were trying to examine needs of the township, they were transferred out. So a township plan developed by TEO alone could not reflect the needs comprehensively. Moreover, a township plan was hinged on one person’s decisions, and others did not know what and how to do when that person was transferred. Now a group of people had a chance to analyse needs and to plan, it reflected township needs better."

"TEOs were transferred in and out frequently. While they were trying to examine needs of the township, they were transferred out. So a township plan developed by TEO alone could not reflect the needs comprehensively. Moreover, a township plan was hinged on one person’s decisions, and others did not know what and how to do when that person was transferred. Now a group of people had a chance to analyse needs and to plan, it reflected township needs better."


21

The TEIP process somewhat addressed this problem by encouraging the inclusion of Deputy TEOs, Assistant TEOs, and head teachers together with TEOs to develop TEIPs. This helped shift responsibility in developing a plan from a single TEO, to a group of people who better understood the contextual needs and situation. Adoption of this new practice appeared to make analysis and planning more relevant and offset the weakness of planning by a single official. All six TEIP townships participating in the evaluation revealed that they would implement their TEIPs if funding would be provided. Are the QBEP-supported TEIPs responsive, equitable and inclusive of their respective township education needs?

In order to assess whether TEIPs were responsive, equitable, and inclusive of their respective township needs, a document review of 11 TEIPs was undertaken. An indication of responsiveness was considered the ability of TEIPs to incorporate solutions to identified education needs at the township level. Indicators of equity and inclusiveness in a TEIP were the inclusion of strategies enabling access to basic education for girls and boys, different ethnic groups, diverse religions, socially marginalised groups such as children infected and affected by HIV, children with four types of disability, children of women headed households and children from migrant families. From this review, evaluators noted that three out of the 11 plans had conducted disaggregate analysis on gender with respect to some education indicators, while one had carried out disaggregate analysis on catchment areas that captured urban and rural discrepancy. This therefore shows uneven levels of equitable or inclusive TEIPs. This was further demonstrated through document review and analyses of data of Participatory Reflective Appraisal (PRA) workshops with TEIP committee members and interviews with DEOs and SEOs of six TEIP townships, interviews with three TEIP facilitators and interviews with three UNICEF officials. This illustrated that six township TEIPs had not included specific targets or intervention measures to improve access to and quality of education for their most vulnerable populations.36 A key reason for not including the needs of vulnerable populations was stated as the difficulty in obtaining specific disaggregated data. The following groups, identified as being of high vulnerability, were not included within TEIPs. Their exclusion from TEIPs points to gaps in responsiveness, equity, and inclusivity and highlights an area that needs strengthening in the future. It is possible that given the changing attitudes and capacity in terms of data collection, TEOs may now have the means to be able to fill these gaps.

36 Most vulnerable populations are classified in line with the criteria stipulated in the lines of inquiry such as poverty status,

rural and urban location, migrant status and ethnicity.


The situation analyses conducted as part of the TEIP development process did not capture the most pressing issues that were undermining access to and quality of education.

Consequently, the TEIPs developed did not incorporate objectives and strategies to address these factors.

QBEP supported TEIPs were not found to be fully responsive to township needs and demonstrated uneven levels of equity and inclusivity in their strategies.


The situation analyses conducted as part of the TEIP development process did not capture the most pressing issues that were undermining access to and quality of education.

Consequently, the TEIPs developed did not incorporate objectives and strategies to address these factors.

QBEP supported TEIPs were not found to be fully responsive to township needs and demonstrated uneven levels of equity and inclusivity in their strategies.


22

For further details on what areas were identified by situation analyses, please refer to Annex 5 Are there other current systems or initiatives currently in place that are more relevant in meeting these planning needs?

The line of enquiry for this question was first to ascertain the existence of alternative mechanisms or systems within intervention and non-intervention townships and then to compare these alternative

Students of poor families

This group had relatively more difficulty in accessing schools, learning while they were at school, and completing higher grades than the students of better off families. Students residing in rural areas encountered similar difficulties. Six TEIPs did not identify the specific needs of students of poor families and rural areas or provide tailored solutions to tackle their specific needs.

Domestic emigrants and immigrants and transnational emigrants to Thailand and Malaysia

TEIPs in these townships did not incorporate specific measures to address the needs of these groups. For example, students of domestic emigrant families working as unskilled labourers on rubber plantations, brick-baking, and paddy fields in Mon State had to discontinue their education when their parents either moved to worksites in Mon state or to other areas. Many parents from Htan Ta Bin township of Bago Region had migrated to Thailand leaving their children behind; when parents struggled to earn adequate money to support their children, their children had to drop out of school. Many parents from Myebon township of Rakhine state had migrated to Thailand and other parts of Myanmar leaving their children behind, and when parents struggled to earn adequate money to support their children, their children had to drop out of school.

Students of ethnic groups

These students had difficulties comprehending school lessons taught in Myanmar language and other languages used by teachers in four townships. TEIPs in these townships did not incorporate specific measures to address the needs of these groups. For instance, students belonging to Pa-O and Karen ethnic groups in Htan Ta Bin township of Bago Region had some degree of difficulty in communicating in Myanmar language. Students belonging to the Chin ethnic group in Myebon Township of Rakhine State had some degree of difficulty in communicating in the Rakhine dialect used by Rakhine teachers. Students belonging to Shan, Arkhar and Lahu ethnicities in Tachileik Township of Shan State experienced difficulty communicating in Myanmar language.


All systems identified were found to be relevant in identifying township education and school needs; however alternative systems were not found to be more relevant than TEIP as they each covered slightly different elements of planning and TEIP still covered a broader and longer remit.

Parents and headteachers in school planning initiatives were found to demonstrate genuine interest, motivation and ownership.

TEIP did not appear to seek resources from local or donor resources, whereas some plans developed through alternative systems could actually be implemented as a result of independent and local fund-raising initiatives.

Alternative initiatives were more well-known than TEIP and school survey respondents showed a slight preference for the SSA/SIP and school grants programmes to TEIP.

In spite of the other QBEP projects also focused on building capacity in education planning, linkages between SSA/SIP and TEIP activities were not found and the opportunity to integrate data and learn from findings was not exploited.


All systems identified were found to be relevant in identifying township education and school needs; however alternative systems were not found to be more relevant than TEIP as they each covered slightly different elements of planning and TEIP still covered a broader and longer remit.

Parents and headteachers in school planning initiatives were found to demonstrate genuine


23

planning activities' ability to address the criteria cited above against TEIP activities. Findings from interviews conducted37 show that a number of alternative current systems are in place such as the School Self-Assessment programme (SSA) / School Improvement Plans (SIP), the school grants programme funded by the World Bank and DFAT, and the Township Education Management Information System (TEMIS) which is also funded by QBEP. This section will briefly describe these systems before assessing their relative relevance to TEIP in meeting planning needs. There are two different mechanisms of SSA/SIP and school grants. The SSA/SIP is supported by QBEP and trains Parent Teacher Associations (PTA) in the development of needs assessments and planning. PTAs work with head teachers to develop school improvement plans and the process involves ten students, ten parents, and ten teachers from each school. SIPs are submitted to UNICEF, and two sets of criteria are used to assess them. One set of criteria is related to the ranking system of A, B, C, D and E by using education indicators, the presence of ethnic groups, and security and remoteness of each school. Another set of criteria encompasses the five dimensions of effective learning, a safe and secured school environment, gender, and community participation amongst other things. SIPs that had attained high scores were provided with grants worth up to US$1,000 per school. The school grant programme supported by the MOE through funding from the World Bank and DFAT started in 2014/15. It provides funds calculated by number of students and the level of school – primary, middle or high school. Each school has to draw up a SIP to access funds and has to use the funds in line with MOE budget category and regulations. Township level auditors examine how funds are expended every six months. Unlike the QBEP supported SSA/SIP, funds are given to all schools, including monastic schools, regardless of the quality of their SIP. Proposals submitted are not evaluated based on a set of criteria and funds are not given on a competitive basis. The SSA/SIP programme was implemented as part of QBEP in five of the TEIP sampled townships, excluding Mudon. Positive findings from these alternative systems were that SSA/SIP appeared to have assisted many schools in identifying their needs and in formulating plans. Some of these schools had received funding from QBEP through the MOE, following submission of proposals to the MOE. They had also been able to mobilise additional resources from community members and philanthropists, which collectively enabled these schools to accomplish some activities included within their SIPs. Similarly, the school grants programme helped schools to develop their own plans and receive financial support from the MOE. The SSA/SIP programme was widely known and was implemented in five of the sampled townships, while the school grants programme was implemented in all seven sampled townships. Students, parents, teachers, and relevant community members were involved in ascertaining needs and in drawing up a plan for each school under SSA/SIP. A management committee, with involvement of key stakeholders, oversaw the execution of each project, including financial management. Many parents and relevant community members were assessed to have genuine interest, motivation, and ownership in carrying out these initiatives. These initiatives were similar to the TEIP in that they all supported a decentralised planning process. Relevance in meeting planning needs Analysis of quantitative data from the school survey and the TED survey demonstrated that the majority of respondents of both surveys considered TEIP, SSA/SIP, and school grants appropriate activities for addressing their township's education and school needs. However, a slightly higher proportion of school survey respondents (by approximately 2 per cent) expressed preference for SSA/SIP and school grants compared to TEIP. The school survey found that the majority of respondents (94.9 per cent, n=130 out

37 Student, parent and teacher focus groups; PRA workshops with TEIP committee members; interviews with DEOs, SEOs,

officials from TGAD, TMC, DRD and TDAC of seven townships, as well as with three TEIP facilitators and three UNICEF officials.


24

of n=137 total) considered TEIP to be appropriate and relevant to addressing their school needs, while 96.7 per cent (n=118 out of n=122 total) of school survey respondents felt that TEIP was appropriate and relevant to addressing their township education needs. When asked about alternative systems, respondents provided slightly higher levels of approval. As such, 96.4 per cent (n=212 out of n=220) of school survey respondents found SSA/SIP appropriate and relevant to addressing their school needs and 96.2 per cent (n=228 out of n=237 total) found the school grants programme to be appropriate and relevant to their school needs. Similarly, high proportions of school survey respondents agreed that SSA/SIP (98.9 per cent, n=184 out of n=186 total) and school grant (98.6 per cent, n=204 out of n=207 total) were relevant to their township education needs. By comparison, 100 per cent (n=17) of TED survey respondents considered TEIP, SSA/SIP, and school grant appropriate and relevant to addressing their township education needs. Table 4: Opinions on relevance and appropriateness of TEIP and other current systems (October – December 2015)38

Opinions on relevance and appropriateness of TEIP and other current systems

Agree (%)

School survey

TED survey

TEIP was appropriate or relevant to addressing your school needs 94.9 (n=130)

Not asked

SSA/SIP was appropriate or relevant to addressing your school needs 96.4 (n=212)

Not asked

School grant programme was appropriate or relevant to addressing your school needs

96.2 (n=228)

Not asked

TEIP was appropriate or relevant to addressing your township education needs 96.7 (n=118)

100 (n=17)

SSA/SIP was appropriate or relevant to addressing your township education needs 98.9 (n=184)

100 (n=16)

School grant programme was appropriate or relevant to addressing your township education needs

98.6 (n=204)

100 (n=17)

Source: Survey of 376 schools and 17 TEDs

However, in spite of these positive findings, the team noted significant weaknesses with regards to these alternative systems and TEIP. There was no integration of TEIP and other programmes of QBEP, UNICEF, the MOE, and non-state education service providers, and according to interviews with UNICEF staff, there was never any intention of integrating these alternative projects within the design, implementation, or M&E stages of TEIP. This meant that relevant data that could have been used to support the design of TEIPs existed, but was not utilised. For example, although SSA/SIP and TEIP were also elements of QBEP, the TEIP committee members and head teachers of schools involved in both SSA/SIP and TEIP were not aware of any specific linkages among these initiatives. When a TEIP was being developed, none of the needs assessment findings or school improvement plans from schools that had completed SSAs or SIPs were incorporated into the formulation of a larger township level plan. Similarly, none of the TEIP committee members or head teachers involved in TEIP, nor any of the teachers participating in focus groups knew if the school grants programme was linked to TEIP. Moreover, the potential link between the Township Education Management Information System (TEMIS) and TEIP did not materialise. TEMIS was launched by the MOE in 15 townships – Nyaung Dong, Bogalay, Kutkhine, Pyi Gyi Dagun, Htan Ta Pin and the ten townships of Mon state – with technical and financial support from

38 Note that n-values vary according to the different response rates for each question.


25

QBEP but was halted because of software problems and other reasons. The cessation of the TEMIS project prevented TEIP from using data of TEMIS for developing TEIP. Lessons learned from each of these initiatives did not appear to feed into the evolution of other initiatives and nor was the capacity building provided, essential in each initiative, being implemented in an integrated manner. Although coordination between capacity building initiatives could have strengthened township education plans, the challenges in actually doing so are recognised. For instance, SSA/SIP and school grants were intended to facilitate the development of a one-year-plan whereas TEIP was a three-year-plan meaning that any shared objectives could have been misaligned. SSA/SIP primarily focussed on developing infrastructure while TEIP tended to be more comprehensive, whereby any activity or strategy that could address 19 indicators related to access to and quality of education could be included in TEIP. Therefore the two initiatives were not directly comparable. As one interviewee put it,

4.2 Effectiveness and efficiency

To what degree are TEIP activities effective and efficient in developing education plans that are inclusive, informed, and realistic? Are the activities producing the results originally intended?

The benchmarks used to assess this question were taken from the stated objectives in the TEIP manual, namely: (1) Greater ownership and leadership of townships in education planning and development process; (2) Greater equity in education provision within and between townships; (3) Improved effectiveness and efficiency in addressing the diverse needs of townships in their education service

"SSA/SIP, school grant and TEIP were not integrated intentionally… Since TEMIS was halted, TEMIS could not feed analysis of data for development of TEIP…There were significant differences between SSA/SIP/school grant and TEIP that, I think, would make it difficult for proper integration. On one hand, SSA/SIP largely focussed on infrastructure development. On the other hand, TEIP was more comprehensive because strategies and activities were formulated based on analysis of indicators of access to and quality of basic education. Moreover, SSA/SIP and school grant were short-term of 1 year while TEIP was 3-year."

"SSA/SIP, school grant and TEIP were not integrated intentionally… Since TEMIS was halted, TEMIS could not feed analysis of data for development of TEIP…There were significant differences between SSA/SIP/school grant and TEIP that, I think, would make it difficult for proper integration. On one hand, SSA/SIP largely focussed on infrastructure development. On the other hand, TEIP was more comprehensive because strategies and activities were formulated based on analysis of indicators of access to and quality of basic education. Moreover, SSA/SIP and school grant were short-term of 1 year while TEIP was 3-year."


Overall, the findings suggest that there were mixed achievements in meeting the intended results:

o At the time of evaluation, all 34 townships had completed a first draft and a total of 25 out of 34 townships had completed the development of a second draft TEIP;

o All the TEIPs reviewed did include three-year budgets, as required; o There was evidence of some townships having developed specific interventions for

their particular township education needs; o There was evidence of some plans, but by no means all, addressing the needs of

vulnerable populations; and o Capacity development workshops were carried out that addressed three of the

identified TEO needs.

However, responses from individuals of all categories interviewed consistently pointed out that intended results as outlined in the TEIP manual were not met adequately.

The key reason for this, and the most important limitation of TEIP effectiveness, was the non-implementation of the TEIPs developed.


Overall, the findings suggest that there were mixed achievements in meeting the intended results:

o At the time of evaluation, all 34 townships had completed a first draft and a total of 25 out of 34 townships had completed the development of a second draft TEIP;

o All the TEIPs reviewed did include three-year budgets, as required; o There was evidence of some townships having developed specific interventions for

their particular township education needs;


26

delivery; (4) Greater budget predictability through multi-year planning and budgeting processes; and (5) Enhanced accountability for education sector performance at all levels of the system. In order to address this question, the evaluation team undertook analyses of data from PRA workshops with TEIP committee members as well as from interviews with DEOs and SEOs of six TEIP townships, interviews with three TEIP facilitators, and three UNICEF officials. At the time of evaluation, a total of 25 out of 34 townships had completed the development of a second draft TEIP. The first intended result of TEIP, as stated in the TEIP manual, was for greater ownership and leadership of townships in education planning and development. In order to address this, a series of three workshops were carried out in each township, starting from the third batch. In addition, TEIP committees and working groups were formed to address planning, budget calculation, supervision and support, capacity building, and coordination of township education needs. After the second workshop training39 and formation of the TEIP committee and working groups, a core group of members, supported by 'secondary members' worked to develop the TEIPs and perform trend analyses on a series of education indicators from the past three to five years. One township undertook a sub-group analysis of some of these indicators by urban and rural areas. Some townships were able to begin the planning process immediately after training, while others could only begin the process up to three months after training due to other competing tasks. A first draft of TEIP was submitted to workshop facilitators from State/Region, and District Education Departments and the central office in Nay Pyi Daw. Some townships shared a first draft to UNICEF officers. Working groups met at least three times to revise their first draft. A core group consisting of either Deputy TEOs or Assistant TEOs, as well as one cluster head or one computer staff spent approximately 11 to 20 days on this process. The table below provides an overview of activities and is presented by capacity need rather than chronological order of implementation. Table 5: Overview of activities to address TEO capacity needs

Intended result

TEO capacity need Activity Participants

Greater ownership and leadership of townships in education planning and development

Data collection and analysis

2 day workshop on data collection and analysis

2-3 delegates from each township

Data collection at school level and from population data

Cluster head

Planning Formation of TEIP committee and working groups

7-19 members

TEIP development 3-5 day workshop on how to draft TEIPs

7 members per township including TEO, Deputy TEO/Assistant TEO, and head teachers from State high schools, middle schools, primary schools

15-35 days / 10-15 days spent on TEIP development

Core group / Secondary group

11-20 days spent on revision of first draft

Core group of Deputy TEO/Assistant TEO, cluster head, computer staff

39 Areas included within the second workshop include inter alia, the role of Township, District and State Education Officers in TEIP, TEIP principles and concepts, and how to set up TEIP committees. Technical training in stakeholder analysis, budget calculation, developing strategic objectives, targets and activities was also provided.


27

Third workshop 2 TEIP committee members

A document review was conducted 11 TEIPs from: Hlaing Thar Yar, Htan Ta Bin, Myebon, Tachileik, Yae, Mudon, Mong Phat, Kyaing Tong, Namtu, Muse, and Namsang. The review found that the second intended result of greater equity in education provision within and between townships was not achieved with uniformity across townships. As already outlined, only three plans conducted disaggregate analysis on gender and over half of the plans reviewed did not include specific targets or intervention measures to improve access to or quality of education for their most vulnerable populations. It is, however, likely that equity in education provision has improved since TEIP's inception, and both qualitative and quantitative data gathered attribute improvements partly to QBEP and TEIP influence but also because of overall increased government investment in education and initiatives supported by other donors, such as the World Bank and DFAT. Results from the school survey and TED survey suggest that any activity to improve equity in education provision is likely to face a number of facilitating and impeding factors; these are thoroughly outlined in Annex 5. The review of TEIPs also found that all had followed the structure suggested by the manual40 and that some had incorporated interventions to enhance the quality of teaching and learning of students. This goes some way to contributing toward the third anticipated result of improved effectiveness and efficiency in addressing the diverse needs of townships in their education service delivery. Some examples of interventions suggested by TEIPs to address the education needs of townships are as follows:

The Yae TEIP outlined improved mechanisms for sharing information between MNEC and government schools and for facilitating the transfer of students from MNEC to government schools. There were 68 MNEC schools in the township and convergence between the government school system and MNEC system was one of the most important needs of the township. Thus, the Yae TEIP was found to have incorporated specific interventions to address an important need within this township.

The Mudon TEIP suggested interventions that would involve a series of interactions between different stakeholders with the aim of addressing teacher needs through provision of teaching aids, additional teaching, and increased access to electricity and telephone. It also suggested interventions such as night-time study periods for 10th and 11th grade students and use of child centred approaches, which address the learning needs of students. The TEIP outlined how this could be achieved through engaging with parents, mobilising donors, enhancing capacity of teachers through group discussion and organising discussions at the school cluster level.

These are promising findings and suggest that some townships at least have sought to address their education needs within their plans, and have considered the means to deliver improved education service. However, as none of the TEIPs have been implemented, it cannot be said that TEIP project activities have yet produced these intended results. The fourth intended result was for greater budget predictability through multi-year planning and budgeting processes. The review of TEIPs found that all had developed a three-year budget in accord with the two tables outlined in the guidelines: a table showing budget by strategic objective and a table showing budget by budget category. Some townships, namely Htan Ta Bin, Myebon, Tachileik, Yae and Mudon, also mentioned contributions from donors other than the MOE, as suggested by the guidelines. These results suggest that achieving this intended result should be possible.

40 These included Executive Summary; Chapter 1: Township Overview; Chapter 2: Situation Analysis; Chapter 3: Township Education Vision and Goals; Chapter 4: Strategic Objectives and Targets; Chapter 5: Activity Plan; and Chapter 6: Financing Plan.


28

Overall, the data captured suggest mixed findings in the achievement of intended results. Aside from the specific examples noted above, responses from focus groups, PRA workshops, and interviews41 consistently pointed out that the intended results as outlined in the TEIP manual were not met adequately. The key reason for this, and the most important limitation of TEIP effectiveness, was the non-implementation of the TEIPs developed. This crucial point has been touched upon in previous sections, but a more detailed exploration of the context, the reasons why they were not implemented, and the ensuing consequences are outlined below. Exploration into why TEIPs were not implemented A fundamental lack of clarity regarding project objectives has affected how this report reviews the effectiveness of TEIP. The only available TEIP-specific project documents state that the township education plans were meant to be implemented; however, UNICEF has stated that the original intent to implement was changed so that the objective of the project became purely the development of a township education improvement plan. As has been noted, this change was not recorded in any official document made available to evaluators and it is not known when this stated change took place, who made the decision to do so, or how this change was communicated to other stakeholders. One stated reason for non-implementation was a lack of funding. The evaluation did not find evidence of the involvement of MOE decision makers in the development of the TEIP project and this is likely to have been a reason for lack of essential government support in funding.42 As such, there was no money available from either MOE or QBEP for the implementation of plans, and this highlights a significant weakness in the project conception and evidences why basic assumptions should have been tested before the project was allowed to go ahead. Evidence suggests that the lack of clarity in these areas led to mixed understandings of TEIP objectives amongst TEIP workshop facilitators, committee members, the MOE, and other stakeholders. During the course of fieldwork it became clear that there were very different understandings among workshop facilitators and committee members, some of whom perceived the project as a capacity building project only, while others thought the plans were meant to be implemented. The lack of MOE involvement also meant that there was no clear instruction from the central level to state and township education offices regarding the objectives and process of TEIP. Neither was there instruction for how the decentralised process (involving Township, District, and State Education Departments) would actually work, or indeed that the project objective was to implement a decentralised plan. Many TEIP committee members disclosed that the working practice of MOE staff members was shaped by the orders imposed from their supervisors. Confusion surrounding the decentralised process was expressed by one interviewee, who explained that,

41 Focus groups with parents and teachers, PRA workshops with TEIP committee members, interviews with DEOs and SEOs and group interviews with officials of TGAD, TMC, DRD and TDAC of six townships, interviews with three TEIP facilitators and interviews with three UNICEF officials. 42 A workshop held in Mon State for the development of the TEIP manual did involve some government officials, but these

were only from Mon State and did not include decision makers from the Government.

"It {progress of TEIP process} depended on attention of the leaders. We, as government servants, had to pay more attention to the tasks for which more pressure was given by our supervisors. Although we knew that TEIP was important, we were under the pressure of other competing tasks, which were instructed as more urgent and important ones. So we had to give more priority to these competing tasks. If there had been a clear instruction from our superiors that it {TEIP} was a priority and must do task, we would have paid more attention to it."

"It {progress of TEIP process} depended on attention of the leaders. We, as government servants, had to pay more attention to the tasks for which more pressure was given by our supervisors. Although we knew that TEIP was important, we were under the pressure of other competing tasks, which were instructed as more urgent and important ones. So we had to give more priority to these competing tasks. If there had been a clear instruction from our superiors that it {TEIP} was a priority and must do task, we would have paid more attention to it."


29

Findings from school surveys and TED surveys corroborated the above qualitative findings. Table 6: Reasons for not implementing TEIP (October – December 2015)

Reasons for not implementing TEIP Agree (%)

School survey

TED survey

Insufficient resources invested for TEIP inhibited implementation of TEIP 90.7 (n=49) 100 (n=6)

Inadequate capacity in Project Cycle Management (PCM) such as need assessment, planning and monitoring and evaluation

77.8 (n=49) 100 (n=6)

Understood that the objective of TEIP was to build capacity, and thus, an end product was a plan

62.1 (n=41) 66.7 (n=4)

There was no clear instruction or order from the MOE that TEIP was a decentralised plan and an objective was implementation of TEIP

54.7 (n=29) 100 (n=6)

Although TEIP was developed, the townships could not implement TEIP because they had to carry out the tasks imposed from the central office

63.0 (n=34) 83.3 (n=5)

Although TEIP was developed, the townships could not implement TEIP because numerous tasks imposed by the central office prevented them from sparing time for TEIP

81.2 (n=56) 100 (n=6)


Other factors that adversely affected the effectiveness and efficiency of the project were linked to the absence of expected project design documents or M&E instruments. This meant that there were no baseline measurements, no detailed implementation plan, insufficient documentation and monitoring reports, and no record of changed objectives in terms of TEIP as a capacity building or decentralisation process. The 2012 QBEP design document, which includes a high level overall QBEP log frame, was insufficient in providing necessary detailed descriptions of input, process, output and outcome level targets, and indicators for TEIP. The TEIP manual described three processes that were related to monitoring, namely plan implementation and progress reporting, annual report and review, and annual work planning and budgeting. However, the manual did not cover how progress against targets would be tracked, nor did it describe the frequency, methods and data sources to be used for data collection, management, analysis, reporting and utilisation. The team could not identify any specific reports on TEIP progress; TEIP facilitators, committee members, and UNICEF officials had to recall the input, process, and output of TEIP from their memories, which was a clear indication of insufficient documentation of the project. In addition to which, there were considerable turnover rates of TEOs, Deputy TEOs, Assistant TEOs, District Education Officers (DEOs) and State Education Officers (SEOs) and a heavy workload on the part of key workshop facilitators, which prevented them from effectively and efficiently fulfilling their responsibilities to TEIP.


30

Are there any unintended consequences (positive or negative) resulting from the activities?

The lines of enquiry taken in order to address this question included whether: (1) greater time consumption hampered other stakeholder tasks; (2) higher resource utilisation could have been invested for other tasks with greater benefits – i.e., forgone opportunities; (3) overburdening human resources resulted in low quality TEIPs; (4) TEIP undermined access to and quality of education; and (5) there was opposition to TEIP by stakeholders, intentionally or unintentionally. Analysis of results from school and TED surveys, document review and analyses of data of PRA workshops, interviews, and focus groups43 collectively and consistently uncovered several findings that point to a range of unintended consequences of TEIP activities. Positive consequences Although TEIP activities were primarily targeted at TEOs, qualitative data analysis made it clear that several other beneficiaries received unintended benefits from the training. Of particular importance were TEIP workshop facilitators (two-thirds female and one-third male) who reported gaining significant benefits from TEIP. One of the key influences of TEIP activities was in their recognition of the importance of and need for developing a state/region level education improvement plan in addition to a TEIP. Another area developed was their ability to analyse the content of state or regional parliamentarians' comments and suggestions regarding the education sector; some facilitators reported being able to better assess how appropriate these suggestions were in terms of further implementation. From their role as teachers, workshop facilitators were already familiar with teacher-centred methodologies, but they reported that since being involved in TEIP facilitation, they had become more acquainted with participatory methods. If true, this could lead to positive outcomes in the classroom and potentially, improved teaching and learning outcomes. TEIP activities were also reported to have encouraged proactive behaviour among some facilitators in terms of self-learning to broaden and deepen their education knowledge and skills through the internet and available literature. Sustained capacity development was evidenced through staff members of District and State Education Departments, who initially attended workshops as observers but became facilitators when their townships entered the TEIP process. In addition a group of TEIP facilitators were produced at the head office of the DBE and district, state, and region education departments.

43 Focus groups with TEIP committee members, interviews with DEOs and SEOs and group interviews with officials of TGAD, TMC, DRD and TDAC of six townships, interviews with three TEIP facilitators and IDIs with three UNICEF officials.

Summary of key findings: Unintended but positive consequences of TEIP activities were found to be:

Beneficiaries other than those originally intended received unanticipated benefits from the training, including significant proportions of female workshop facilitators.

TEIP activities were also reported to have encouraged proactive behaviour among some facilitators in terms of self-learning.

The whole state approach appears to have strengthened relations between UNICEF, the national government, and ethnic education bodies, thereby contributing to peace building within Mon state.

Unintended but negative consequences of TEIP activities were found to be:

By not submitting TEIPs to the intended government bodies, there was little knowledge of TEIP aims and therefore little inclination for district or state administrations to support TEIPs.

Opportunities to provide support to townships from higher levels were therefore lost.

Summary of key findings Unintended but positive consequences of TEIP activities were found to be:

Beneficiaries other than those originally intended received unanticipated benefits from the training, including significant proportions of female workshop facilitators.

TEIP activities were also reported to have encouraged proactive behaviour among some facilitators in terms of self-learning.

The whole state approach appears to have strengthened relations between UNICEF, the national government, and ethnic education bodies, thereby contributing to peace building within Mon state.

Unintended but negative consequences of TEIP activities were found to be:

By not submitting TEIPs to the intended government bodies, there was little knowledge of TEIP aims and therefore little inclination for district or state administrations to support TEIPs.

Opportunities to provide support to townships from higher levels were therefore lost.


31

Other unintended benefits were related to the whole state approach, which covered all ten Mon townships, whereby relations between UNICEF, the national government, and ethnic education bodies were reportedly strengthened, thereby contributing to peace building within the state. In addition to reports of enhanced capacity in needs assessments and planning, respondents from these townships claimed that TEIP activities had led to better relationships and collaboration between UNICEF and the MOE as well as stronger trust between the MOE and the MNEC, which oversees the education programme of the New Mon State Party. Impoverished MNEC schools benefitted from capacity building delivered through TEIP and SITE activities, as well as from material and financial support related to TEIP. Mon townships benefited from access to the UNICEF school grants scheme, whereby UNICEF assisted the MOE in training MNEC schools to develop SIPs, compiling SIPs, and granting funds, as well as from the development of an ethnic language policy and Mon curriculum to fulfil the rights to learn education in mother tongue prescribed in the National Constitution. Although TEIP created a heavy workload for committee members (between 15 to 35 days for core members and between 10 to 15 days for other members over a period of two to 13 months), evaluators found that this burden did not adversely affect the committee members' other responsibilities. Similarly, while committee members paid out of pocket to attend TEIP meetings (not workshops) and TEDs had to use their own resources for TEIP, the costs incurred were not felt to have drained their resources significantly or to have been so great as to prevent them continuing their involvement. Negative consequences None of the six townships had sought the involvement of relevant government departments (such as the Township General Administration Department, Department of Rural Development, or Township Municipality Committee), relevant NGOs, or other education service providers such as private schools, monastic schools, or education projects run by NGOs. Several factors prevented TEIP committees from expanding the involvement of key stakeholders, but some notable ones were the fact that existing stakeholders were too occupied with their primary tasks, and that although TEIP workshop facilitators gave advice on how to seek the involvement of these stakeholders, many TEIP committees did not understand this advice sufficiently. TEIP committee members also assumed that other potential stakeholders would not be able to spare their time, which inhibited them from enquiring further. Finally, and significantly, MOE officials have been strongly influenced by the engrained environment of centralised planning that does not promote a culture of working with other stakeholders effectively. As mentioned above, none of the six townships submitted their first draft TEIPs to district, state, or regional education departments as outlined in the manual. Only after finishing the second draft version did they submit these documents to district, state, or regional education departments. An unintended consequence of not submitting TEIPs to the stipulated body meant that departments were unaware of TEIP aims and therefore opportunities to provide political, technical, or managerial support to townships from the regional or state level were lost.


32

Are there any particular groups currently not reached through the TEIP activity?

The line of enquiry taken to address this question looked at the extent to which TEIPs include or exclude one or more of the following groups: (1) men and women; (2) various ethnic groups; (3) various religious groups; and (4) socially marginalised groups such as children infected and affected by HIV, children with four types of disability, children of women headed households and children from migrant families. Findings from data collection44 indicated that TEIPs did not set out specific intervention measures for many disadvantaged groups of students as outlined in the section on relevance above. These findings highlighted that TEIP activities were not responsive to the needs of many ethnic minority groups, and that this was a more significant omission than those of gender or disability. The groups outlined in the above Relevance section had more difficulties in getting access to education and encountered a higher degree of drop outs from schools. Their learning requirements were for more appropriate forms of skills more pertinent to their lives, such as livelihood skills and life skills, because they were more likely to stop their education before university education. They also had more problems in learning school lessons effectively. Current intervention measures could not address these unique needs sufficiently, and although specific measures are needed, they were not found in the TEIPs studied.

By not seeking the involvement of relevant government departments, relevant NGOs, and other education service providers, as discussed under the unintended results sub-section, TEIPs were unable to get a broad picture of their township populations and prevented them from discerning the needs of the most vulnerable groups of students. Neither were they able to formulate specific intervention measures relevant to these students' needs. Education service providers for the most vulnerable students, such as monastic schools, were precluded from TEIP resulting in a lost opportunity to address their problems. TEIP situation analyses did not tease out the most pressing issues related to quality of education and consequently, TEIP did not incorporate strategies to address the following issues: the examination system, the quality of teaching and competency of students, quantity versus quality, the utility of education in real life, and tuition (for more details, please refer to Annex 5).

44 From document review and analyses of data of PRA workshops with TEIP committee members, interviews with DEOs and

SEOs and group interviews with officials of TGAD, TMC, DRD and TDAC of six TEIP townships, interviews with three TEIP

facilitators and interviews with three UNICEF officials.


TEIPs did not set out specific intervention measures for many disadvantaged groups of students.

The lack of response to ethnic minority groups was more significant to communities than the lack of attention paid to gender or disability.

Learning needs amongst populations are for practical skills that can be applied directly to their livelihoods - measures to address such needs were not found in the TEIPs studied.

Opportunities to address the needs of the most vulnerable populations were lost as synergies with other education providers who had access to such populations were not created.


TEIPs did not set out specific intervention measures for many disadvantaged groups of students.

The lack of response to ethnic minority groups was more significant to communities than the lack of attention paid to gender or disability.

Learning needs amongst populations are for practical skills that can be applied directly to their livelihoods - measures to address such needs were not found in the TEIPs studied.

Opportunities to address the needs of the most vulnerable populations were lost as synergies with other education providers who had access to such populations were not created.


33

Are resources dedicated to TEIP sufficient to reach intended groups and desired results?

The indicators developed to address this question included the following questions: (1) How much in the way of human resources is allocated to TEIP and sufficiency of these resources to reach intended groups and to trigger intended results? (2) How much capacity is acquired to TEIP and sufficiency of capacity to reach intended groups and to trigger intended results? (3) How much ownership of responsible persons is present for TEIP and sufficiency of ownership to reach intended groups and to trigger intended results? (4) How much accountability of responsible persons is present for TEIP and sufficiency of ownership to reach intended groups and to trigger intended results? (5) How much material is allocated to TEIP and sufficiency of material to reach intended groups and to trigger intended results? (6) How much time is allocated to TEIP and sufficiency of time to reach intended groups and to trigger intended results? Evaluators found that resources invested for TEIP were sufficient for finishing the development of TEIP but insufficient for implementation of TEIP, for which no funding was available. Each township received approximately US$2,500 of support from QBEP. While QBEP covered the costs of participant and facilitator travel and food, as well as other expenses for the organization of three workshops for TEIP committee members, out of pocket expenditure was still common. Participants were accommodated by the schools in which workshops were held. However, TEDs had to utilise their own resources to convene TEIP development meetings and workshops, as well as to type and print TEIPs. Members of TEIP committees paid out of pocket to shoulder travel costs and food costs for participating in additional TEIP meetings/workshops. The number of human resources utilised for the development of TEIPs ranged from seven to 19, of whom nearly half were female. Evaluators found varying attitudes toward TEIP among those involved at the ground-level, the most half-hearted of which could impact significantly in the degree to which TEIP reaches the intended groups or results. Leadership attributes of TEIP committee members were found to differ from one township to another significantly with varying degrees of enthusiasm, commitment, diligence and determination to accomplish what they ought to achieve. Such mixed attitudes suggest that human resources could be improved in order to achieve desired results. Similarly, the strength of ownership of TEIPs by TEIP committee members varied widely, with evidence of limited belief in their role in the project. Although some TEIP committee members perceived TEIP as a genuine means to engage in improving the education status of their township, others perceived it as something implemented by UNICEF and felt that the task was only a capacity building activity that was not realistic because they understood the exercise as simply following instructions from the top level of the MOE. Numerous routine tasks and ad hoc pieces of work allocated to TEIP committee members disrupted the process of developing TEIPs. As a result, TEIP committee members had to take a sporadic breaks drawing


Financial resources invested in TEIP were sufficient to develop TEIPs but insufficient to implement them.

Attitudes and levels of ownership amongst TEIP committee members vary – some committee members showed enthusiasm and belief in their roles whilst others felt that it was an imposed top-down activity.

Clarifying TEIP objectives and strengthening committee member ownership would promote sustainability.

Summary of key findings

Financial resources invested in TEIP were sufficient to develop TEIPs but insufficient to implement them.

Attitudes and levels of ownership amongst TEIP committee members vary – some committee members showed enthusiasm and belief in their roles whilst others felt that it was an imposed top-down activity.

Clarifying TEIP objectives and strengthening committee member ownership would promote sustainability.


34

out the development of TEIP from two up to 13 months. Overall, the organizational management, financial management, and M&E systems of each school and each township were inadequate to facilitate efficient implementation of the TEIP (see details in Annex 5) and while the resources provided by QBEP and the MOE enabled the TEIP committees to produce a plan, these were insufficient to allow committees to actually implement them. Are there any ways in which the activities can be improved, either through changes in their current implementation or based on good practices from similar activities in Myanmar and/or elsewhere in the region?

This element was assessed through measuring the SWOLs of TEIP and alternative programmes administered in comparison sites. Findings from the data collected45 revealed that there were several examples of good practice amongst private and monastic schools as well as CBO and NGOs in terms of school structure, resource mobilisation, and inclusion of vulnerable populations. One of the key areas of TEIP weakness was found to be linked to decentralisation, and in consequence the evaluation team has focused on identifying where improvements can be made based on their own findings, as well as from drawing on examples of international good practice. Analysis of interviews with five private schools and one NGO/CBO shed light on promising practices and lessons that can be learned for a future TEIP programme (for further details refer to ‘Lessons learned from other education service providers of six townships’ in Annex 5). The owners of the five private schools were either former government school teachers or private tuition teachers who remained committed to improving the education sector. Three out of the five charged approximately Kyats 30,000 to 40,000 per month for 8 to 10 grades, which is slightly higher than the cost of Government schools if tuition fees are included. These schools made use of limited resources by employing retired MOE teachers and officials. They developed teaching techniques based on lessons learned from their past experience, such as extending one teaching session to one hour (from 45 minutes) for the 6th grade to the 11th grade, as shorter lessons did not allow time for teachers to revisit areas that students had not understood. Regular assessments of lessons were undertaken daily or weekly by either guides or students. All schools set aside ample time for self-learning with coaching. These two practices helped students to digest lessons thoroughly. Teachers were assigned to teach one subject for primary or middle or high grades regularly, which helped teachers to learn and improve his or her skills better. One private school created a feedback mechanism for students, through which anonymous letters could be submitted to the school. Once a week, anonymous letters were reviewed, and issues raised and solutions were discussed in a student meeting. This feedback mechanism could address many issues that would otherwise not have

45 From focus groups with parents and teachers, PRA workshops with TEIP committee members, interviews with DEOs and

SEOs and group interviews with officials of TGAD, TMC, DRD and TDAC of six townships, interviews with three TEIP facilitators

and interviews with three UNICEF officials.


There are several good examples from schools that could be incorporated into TEIPs for example: drawing on the experience of human resources in the face of limited financial resources, implementing feedback mechanisms, developing income generating activities and resource mobilization mechanisms.

The decentralisation process is not yet functioning well and there needs to be more support and clearer instructions from the national to subnational levels for this to improve.

Attitudes have not yet moved away from centralised systems and this is restricting stakeholders from having ownership or accountability over education activities; changing these attitudes would help create a more enabling environment for TEIP to operate.


There are several good examples from schools that could be incorporated into TEIPs for example: drawing on the experience of human resources in the face of limited financial resources, implementing feedback mechanisms, developing income generating activities and resource mobilization mechanisms.

The decentralisation process is not yet functioning well and there needs to be more support and clearer instructions from the national to subnational levels for this to improve.

Attitudes have not yet moved away from centralised systems and this is restricting stakeholders from having ownership or accountability over education activities; changing these attitudes would help create a more enabling environment for TEIP to operate.


35

been identified and solved. These initiatives provide examples of good school practice that could be incorporated within TEIPs. The Hlaing Thar Yar Development Network was established with 23 CBOs and self-help groups. They have been implementing non-formal education classes, community-based early childhood care and development (ECCD), night group study, and livelihood skills teaching. The Hlaing Thar Yar Development Network collaborated with township Departments for Social Welfare and TEDs for ECCD, livelihood skills and non-formal education. A resource mobilisation mechanism was established via running various forms of income generating activities. Portions of the profit were used to run activities for education, health, and other sectors. At the time of the evaluation, approximately Kyats 2,500,000 or 60 per cent of the total cost of Kyats 4,000,000 for the various development activities of one CBO could be supported by the profit from its income generating activities. An exit strategy was developed and implemented by World Vision Myanmar and the Hlaing Thar Yar Development Network, which helped them to continue their activities after a phase-out of the World Vision project in 2015. Good practice in terms of including vulnerable populations was demonstrated by monastic schools that accommodated students from very difficult circumstances such as the poorest of the poor, ethnic and religious minorities (including Christian and Muslim children), as well as orphans and vulnerable children, migrants, and children with disabilities. The school survey and the TED survey examined the factors that could influence improvement or deterioration of the education sector. These analyses can be used as a baseline situation of TEIP and should be taken into consideration for the formulation of future iterations of TEIP and other related programmes (see details in Annex 5). Findings demonstrated that the aspect of decentralisation has posed several challenges to TEIP. In order to overcome these challenges, a number of issues need to be imbedded within the TEIP process. Firstly, there are good practices that have been identified by the evaluation team from their observations within Myanmar and that can be built on. However, there needs to be more valid and reliable data available to discern the needs and contexts of each township more accurately. The management of TEIP could be improved through more participation from key stakeholders (including local community members) and education service providers, through a delegated and autonomous committee with clearly delineated roles and responsibilities. Management systems at the subnational level could be improved through clear and concise policies and instruction from the national level as well as through the provision of more financial, managerial, and technical support. It will also be necessary to move away from the entrenched centralised organizational culture and attitudes embedded within individuals. Increasing the ownership and accountability of key stakeholders, integrating the education sector with other sectors in each township, and addressing the root causes of education needs would also serve to improve the context in which TEIP could operate. Secondly, findings from a document review of the decentralised education programmes across various settings highlight good practice from beyond Myanmar but that are complementary to analyses of the above primary data. Key findings from document review "Educational Quality in the Developing World," published by USAID46, reported that most decentralisation initiatives fall into three categories. The first being models that are based on devolution of service delivery responsibilities from national to regional governments, which have to provide education services, and which shoulder some or all costs. The second being models that delegate service delivery decisions and functions to regional government and school levels, with funding supported by the

46 USAID, Decentralisation in Education, Educational Quality in the Developing World 3, No.4: 1-4, 2005.


36

national government. In this model, school management committees are formed with elected community representatives, teachers, and a school director who are given autonomy, authority and responsibilities. The third being models that adopt both types of decentralisation simultaneously. The review highlighted several common challenges encountered by decentralised projects. One that is particularly relevant to the Myanmar context regards the design of decentralisation, as specified in legislation and decrees, which creates uncertainty as to which level of government or which decision-maker is responsible for what. This may also be highlighted by legislation that creates conflicts with other existing laws. Moreover, the Myanmar context echoes the study findings in that the education ministry continues to regulate the curriculum and determine teacher employment and pay scale, leaving little room for local control. Other shared characteristics relevant to the Myanmar context were found to be that the capacities of school boards, directors and teachers to work collectively to reform the school are often weak and need development. In addition, system support to the newly decentralised authorities often does not exist or is weak. Subnational governments, school boards, and parents have limited access to information about their schools’ academic and financial performance relative to other jurisdictions or schools. School inspectors or local offices of education ministries do not possess the culture and capacity to provide guidance or assistance. Teacher in-service education remained supply-driven by education ministries. Little training and other support was given to school directors to manage their new tasks. A final point was that decentralisation was often not accompanied by increased discretionary funding required for schools to exercise their new responsibilities for self-improvement47. A study that aimed at drawing lessons learned from decentralisation in education governance illustrated that there was direct association between improved education performance and an increase in autonomous school management in some developed countries. Some commonalities included the increased role of local stakeholders, new forms of accountability and competition at each locality, and a shift from input orientated to output orientated management. The study also suggested that three dimensions of decentralisation - administrative, fiscal, and political - should be considered in formulating decentralised governance and that deconcentration, deregulation, and privatisation were key elements of decentralisation. The reported advantages were:

A better fit between schools and local demands; more choices for parents and students; local level knew needs better;

The fact that decentralisation promoted participation and democracy at the local level and could prevent control of national institutions by special interest groups; and

That decentralisation enhanced the efficiency of service provision because localities competed against each other.

By contrast, the reported drawbacks were that decentralisation could create more bureaucracy; local institutions could also be controlled by special interest groups; there is the risk of increased administrative overload on the local level; competition might have deleterious consequences in terms of costs; and that the impact on educational performance remained unclear48. A cross country study used a panel dataset from a sample of developed and developing countries for five-year intervals during the period 1970-2004 to examine impact of decentralisation on education output. This study discovered that education decentralisation had significantly improved repetition, dropout, and completion rates, as well as test scores in science at the primary school level when other confounders

47 Ibid. 48 Busemeyer, M.R., Two decades of decentralisation in education governance: Lessons learned and future outlook for local

stakeholders, OECD Conference on Effective Local Governance in Education, Warsaw, Poland, April 16, 2012, Warsaw, Poland,

2012.


37

were controlled. Moreover, with respect to the influence of decentralised decision-making in education, the study found that both decision-making at the intermediate and school levels significantly improved education output. Specifically, these findings were suggestive that decisions on planning and personnel management had a greater influence on education output when taken at the intermediate level of government such as states and provinces, and that decisions at the school level could significantly improve education output. In addition, it revealed that allocating a portion of decisions for a sole category of education decisions might not lead to improvements in education output. Besides, the study findings supported the efforts of international financial institutions, bilateral donors and governments for subnational level decentralisation of education. Finally, the study suggested that since educational attainment and human capital are considered to be a key determinant in reducing poverty and improving economic performance, policy-makers may want to consider education decentralisation as a tool to influence education indicators to ultimately fight poverty and achieve economic growth49. A review of 14 modalities from 12 countries ranging from the United States of America (US), Netherlands, Zimbabwe, Mexico, the Philippines and Indonesia shed light on common important features that underpinned these decentralised modalities. Future TEIP and other decentralised modalities in Myanmar could consider embedding these elements (see ‘Experiences of decentralisation of education sector from other countries’ in Annex 5).

49 Heredia-Ortiz, E., The Impact of Education Decentralisation on Education Output: A Cross-Country Study, Dissertation for

PhD, Georgia State University, USA, 2007.


38

Market-based competition and school choice: This modality provided parents with the power and opportunity to choose the school their child will attend. Parents were provided with financial support by the national governments or subnational governments and allowed them to make a choice of school for their children from public or private schools. Schools received government fund in accord with the number of students enrolled. This system promoted schools to improve the quality to compete for students and accompanied fund (Brennan and Buchanan, 1980; Hanson, 1997; McGinn and Welsh, 1999; Parry, 1997; Vegas, 1999).

Managing schools by an independent body formed with key stakeholders: Public schools were managed by an independent body consisting of educators, parents, community leaders, educational entrepreneurs and others to develop and deliver services tailored to educational excellence and community needs. In some modalities, boards or independent bodies were elected. Some boards included elected students over the age of sixteen (Fiske, 1996; Lobo et al, 1995). Decentralisation of decision-making authority: Decision-making power was granted to local communities for administrative, financial and pedagogical autonomy: construction of primary schools; authority to hire and fire teachers; how to spend grant funds and locally raised education revenues; curriculum; pedagogy of teaching; school calendar; etc. (Fiske, 1996; Lobo et al, 1995). In some models, schools were administered and governed by private school boards, which could exert greater freedom to allocate their resources and manage their own affairs, although they still were held accountable to the central government for their performance and policies (Patrinos, 2000). Provision of grant and other support to schools by national or subnational governments: Many modalities rendered grant or voucher per student to schools. The amount of money received by each school depended on the number of students enrolled. In some modalities, teacher’s salaries were supported by the MOE. Both private and public schools were provided with funding by national governments, and additional funding was provided for schools in regions with high numbers of underprivileged families in some cases (Patrinos, 2000). All independent schools approved by the National Agency for Education were entitled to public funding, and gaining access to public fund enabled 90 per cent of private schools to waive school fees to students in one country (Heredia-Ortiz, 2007). Role of the national level MOE or equivalent body: Some modalities retained the authority of designing the curriculum, conducting examinations, and training teachers for the MOE. Teachers’ union bargaining was maintained at the national or subnational level (Fiske, 1996). The national level monitored quality, standard and integrity but allowed schools to manage independently in some modalities. The MOE imposed a number of statutory quality standards and its role was confined to broad policy-making and to creating the right conditions for the provision of quality education in some counties. Private and public schools had to follow the national curriculum (Patrinos, 2000). Role of parents to make an active choice and to increase parents’ accountability: In one modality, parents were given a catalogue profiling all the local schools plus a voucher that parents presented for their school choice. Parents who did not choose a school were contacted by some of the closest schools to encourage an active choice and parents were also required to present a new voucher before the first, fourth, and seventh grade even if the child was attending the same school (Heredia-Ortiz, 2007).


39

A comprehensive reform in Armenia: A reform strategy was introduced to place more responsibility at school level by establishing a framework for managing education reform, including development of detailed implementation plans and capacity building for reforms of school finance and governance. Schools received a lump sum fund on a capitation basis, and were free to allocate these funds between different inputs within specified limits such as minimum salary rates. Newly established school boards managed the budget. Pilot implementation was carried out in 10 per cent of the country's schools. The project funded technical assistance to define details of the new funding formula, legal and regulatory framework, accountability and reporting requirements, and supported training of school principals, board members and accountants. The Pilot School Improvement Programme was designed to build management capacity at the school level to match the autonomy reforms by providing grants up to US$10,000 to schools for self-identified projects. These were for investment projects, and not for normal recurrent costs or reconstruction. Its typical components included purchase of equipment and teaching materials and teacher training in new subjects/methods – geared either to teaching the core curriculum better or to introducing extra-curricular classes. Projects had to show a strategy for sustainability and to include providing paid services to the community – for example computer or language training -- or selling product of extracurricular vocational activities – for instance, agricultural/food products. Schools had to be autonomous and to finance 10 per cent of costs (Heredia-Ortiz, 2007). Education Decentralisation Efforts in Mexico: The education decentralisation process was implemented in three stages. Between 1978 and 1982, the MOE deconcentrated management of the education system to 31 states by giving responsibility on budgeting, managing schools, writing of curriculum and textbook choice. Revenue generation, core curriculum design and labour policy remained at the central level. During this first stage, enrolments of preschool, primary and secondary increased, especially in rural areas. During the second phase from 1983 to 1988, although the national government intended to transfer additional control to the states, it failed because of teacher unions’ opposition of negotiating with 31 states. Moreover, central government staff members resisted due to their interests in the centralised system and their long-standing cooperative arrangements with teachers. In 1988, a new government negotiated an agreement with the national teachers’ union to adopt a new law, which transferred most educational decision-making authority for primary and secondary schools to the state governments. However, the central government’s role in financing education through negotiated transfers to the states resulted in de facto continued centralisation. It was not until 1998 that decentralisation was in place when education transfers became automatic (Fiske, 1996; Hanson, 1997).

A comprehensive reform in Armenia: A reform strategy was introduced to place more responsibility at school level by establishing a framework for managing education reform, including development of detailed implementation plans and capacity building for reforms of school finance and governance. Schools received a lump sum fund on a capitation basis, and were free to allocate these funds between different inputs within specified limits such as minimum salary rates. Newly established school boards managed the budget. Pilot implementation was carried out in 10 per cent of the country's schools. The project funded technical assistance to define details of the new funding formula, legal and regulatory framework, accountability and reporting requirements, and supported training of school principals, board members and accountants. The Pilot School Improvement Programme was designed to build management capacity at the school level to match the autonomy reforms by providing grants up to US$10,000 to schools for self-identified projects. These were for investment projects, and not for normal recurrent costs or reconstruction. Its typical components included purchase of equipment and teaching materials and teacher training in new subjects/methods – geared either to teaching the core curriculum better or to introducing extra-curricular classes. Projects had to show a strategy for sustainability and to include providing paid services to the community – for example computer or language training -- or selling product of extracurricular vocational activities – for instance, agricultural/food products. Schools had to be


40

4.4 Sustainability

What elements of the TEIP activities should be sustained as part of NESP following the end of the current QBEP funding? Is capacity available for the TEIP activities to continue to be carried out at the township level following the end of QBEP?

Evaluators used the following lines of enquiry to answer this question: (1) Do TEOs have sufficient skills in examining education needs at the township level; allocating financial and other resources needed to develop and implement a TEIP; monitoring and evaluating a TEIP and using findings appropriately? (2) Do TEOs report being likely to actually implement their TEIPs? (3) Is there an exit strategy? (4) What means are there for incorporating lessons learned and alternative planning approaches? Analyses of qualitative data50 consistently demonstrated that the capacity of the two or three core TEIP members of each committee was sufficient in terms of needs assessment and planning, due to the longer time they spent on developing TEIPs than other members. During this period, they utilised knowledge gained from the three TEIP workshops to analyse their township's situation and draw up an entire plan. In comparison, the capacity of the five to 15 non-core TEIP committee members was found to be weaker, as they spent less time working on TEIP, during which their roles were limited to writing only single chapters of TEIP or contributing to individual elements of the TEIP. The capacity of TEIP committee members in implementation, M&E and sustainability strategies remains untested because plans have not yet been implemented. There is no known exit strategy for TEIP although the overall QBEP exit strategy is cited as being designed whereby townships will 'graduate' from QBEP once they have achieved minimum quality standards against key standards. These were meant to be assessed in 2015 in order to judge which townships could ‘safely’ be allowed to exit.51 However, the strategy described is vague and does not address individual project components such as TEIP. It assumes that there will be funding available for those schools that do not meet indicators and cannot graduate, and to the evaluators' knowledge, no data collection was carried out in 2015 to make the exit assessment. Qualitative findings triangulated between a range of different sources between TEIP committee members, UNICEF staff and government officials (amongst others)52 suggest a varying degree of political support between townships, highly dependent on the interest and concern of individuals in government departments and state/region governments. Despite the variation, it was assessed that a considerable

50 From PRA workshops with TEIP committee members of six TEIP townships, interviews with three TEIP facilitators and interviews with three UNICEF officials. 51 UNICEF, QBEP Design Document 2012, p. 38, Section 2.8.2, 2012. 52 PRA workshops with TEIP committee members, interviews with DEOs and SEOs and group interviews with officials of TGAD, TMC, DRD and TDAC of six TEIP townships, group interviews with one township level education foundation, interviews with three TEIP facilitators and 3 UNICEF officials.


Positive indications of sustainability were found in terms of skills and behaviours whereby capacity levels of core TEIP committee members and commitment and interest among broader committee members.

Financial sustainability was found to be less certain and dependent on the resourcefulness of individual townships in setting up independent foundations to support education provision.

Less positive was the lack of a TEIP specific exit strategy and an insufficiently detailed overall QBEP exit strategy; the plans themselves were not found to include sufficient exit strategies either.


Positive indications of sustainability were found in terms of skills and behaviours whereby capacity levels of core TEIP committee members and commitment and interest among broader committee members.

Financial sustainability was found to be less certain and dependent on the resourcefulness of individual townships in setting up independent foundations to support education provision.

Less positive was the lack of a TEIP specific exit strategy and an insufficiently detailed overall QBEP exit strategy; the plans themselves were not found to include sufficient exit strategies either.


41

degree of political support was provided to TEIP by township, district, state and regional government departments as well as by state/region governments. Members of the Township Development Assistance Committee (TDAC) of six townships were found to have shown genuine interest, commitment, and financial support. Three out of six TEIP townships and one non-TEIP township had established a functioning township level education fund or foundation that provided financial support to poor but outstanding students and allowed them to continue their university education. The education foundation of one township also provided stipends to poor but outstanding students enrolled in primary, middle and high schools and who had plans to deliver livelihoods skills and provide job opportunities for children and youth. These initiatives indicate that for some areas, there is an enabling financial environment to sustain TEIP activities. Analyses of data collected from multiple programme and government sources53 suggest that almost all of interviewed stakeholders thought that decentralised planning and implementation at the township level was more appropriate than a centralised process to improve the education status of their townships. Furthermore, they felt that the interest, commitment and support shown by stakeholders was considered adequate to sustain a future decentralised planning and implementation process. By contrast, willingness and support from parents and community members to improve the education sector is still insufficient. One interviewee felt that, The school survey and TED survey uncovered complementary quantitative findings to the above qualitative findings. The majority of schools (96.7 per cent, n=147) and TEDs (94.1 per cent, n=16) surveyed felt that a decentralised school improvement plan and a decentralised township education improvement plan would be more suited to addressing needs of their schools and townships than a centralised one. Table 7: Opinions on decentralised school plans and township education plan (October – December, 2015)

Opinions on decentralised school plans and township education plan Agree (per cent)

School survey

TED survey

In the future, your school should use a decentralised school improvement plan rather than a centralised school improvement plan

95.1 (n=174)

Not asked

In the future, your township should use a decentralised township education improvement plan rather than a centralised township education improvement plan

96.7 (n=147)

94.1 (n=16)


Further analysis revealed that TEIPs did not include a proper exit strategy or mechanisms for incorporating lessons learned.

53 Focus groups with parents and teachers of six TEIP townships and one non-TEIP township, PRA workshops with TEIP committee members, interviews with DEOs and SEOs and group interviews with officials of TGAD, TMC, DRD and TDAC of six TEIP townships, interviews with private schools and monastic schools of four townships, group interviews with NGO and one township level education foundation, interviews with three TEIP facilitators and three UNICEF officials.

"As you {the evaluation team} did with us {had a group discussion with parents}, a needs assessment should have been undertaken. Needs elicited from various sources should be combined. This compilation of needs and corresponding solutions should be submitted to those at the higher levels. The higher levels should also support the lower levels to fulfil the needs. This would have been a better approach."

"As you {the evaluation team} did with us {had a group discussion with parents}, a needs assessment should have been undertaken. Needs elicited from various sources should be combined. This compilation of needs and corresponding solutions should be submitted to those at the higher levels. The higher levels should also support the lower levels to fulfil the needs. This would have been a better approach."


42

To what extent can TEIP activities support the development of district or state level education improvement plans?

The indicators used to address this question were: (1) involvement of district and state or regional stakeholders in a TEIP process; (2) presence of a mechanism to compile and consolidate township level education needs at the district and/or state/regional level; (3) tailored solutions of diverse and unique needs of individual township by a district and/or state/region plan; (4) receiving political, managerial and technical support from relevant governmental organizations, NGOs and state or regional governments; and (5) presence of appropriate organization and financial management capacity in place to facilitate incorporation of township level needs into a district and/or state/region plan.

Analysis and triangulation of data collected from teachers, TEIP committees, MOE officials and UNICEF staff indicates that the existing human resources, capacity, and management systems of schools and township, district, state, and regional education departments were insufficient to sustain development and implementation of TEIP in the future. For example, state primary schools did not have any staff members to assist the head teachers in executing organizational management, financial management, or M&E systems. Although head teachers of state primary schools received training in these areas and in ways of computing indictors, the high degree of turnover, inadequate handover between outgoing and incoming head teachers, and sole dependency of these management systems on a head teacher collectively contributed to sub-optimal functioning of management systems. Although state middle schools and state high schools had some administrative staff members such as two clerical staff structurally, many of these schools were under-staffed. Similarly, township, district, state, and regional education departments are currently in a transitional period from an old structure to a new structure, and as a result, delineation of roles and responsibilities are still unclear or unsettled and many positions are vacant. Additionally, staff members who had to take the responsibility of financial management and information management were not found to have received specific training on these issues. Nor did they have formal educational qualifications in areas such as accounting. Findings from the TED survey corroborated the above qualitative analyses. Around half (54.1 per cent) of the positions of TEDs, 21.3 per cent of positions of primary school, 19.1 per cent of positions of middle school, and 13.4 per cent of positions of high schools were vacant in 17 TEIP townships. Analyses of data from PRA workshops with TEIP committee members as well as interviews with DEOs, SEOs, TEIP facilitators and UNICEF officials indicated that a system to consolidate information from the township to district and from district to state/region education departments was insufficiently functioning. State, district, and township levels had to seek data from the level below when Nay Pyi Daw requested it. Similarly, there was no system for using school plans as a basis to develop a township plan. Nor were there similar systems at the district and state/region levels to utilise plans of lower levels when they developed their plans. For example, when a TEIP was developed, SSAs/SIPs were not consolidated


Overall, it is unlikely that TEIP activities will support the development of district or state education improvement plans due to limited involvement of this level of education department in the TEIP process.

This is underscored by a lack of understanding amongst District and State Education Department staff of their roles under the decentralised system a lack of overall management capacity.


Overall, it is unlikely that TEIP activities will support the development of district or state education improvement plans due to limited involvement of this level of education department in the TEIP process.

This is underscored by a lack of understanding amongst District and State Education Department staff of their roles under the decentralised system a lack of overall management capacity.


43

to use as a basis for TEIP. Moreover, officials from district and state/region education departments were not well involved in planning, reviewing TEIPs, and providing feedback to TEIP committees. Further data analysis disclosed that although core group TEIP committee members were likely to gain capacity in indicator derivation, SWOL analysis, planning, among other things, the remaining TEIP committee members, despite their participation in a workshop, were unlikely to acquire adequate capacity in these skills as discussed above. These analyses, also supported by the fact the organizational culture of township, district, state/region, central education departments has been influenced by the centralised planning and management approach, have been in place for more than 50 years. Although the majority of stakeholders expressed a preference for a decentralised planning and management process, in the face of such entrenched centralisation, the evaluation team considers that they will continue to face considerable challenges of breaking down the organizational culture. The school survey also uncovered that 25.0 per cent of schools thought that there was a high likelihood of sustainability of TEIP, 63.9 per cent perceived there was a moderate likelihood while 11.1 per cent thought that there was a low likelihood. The findings in the table below corroborated the qualitative findings by pointing out insufficient practices being adopted and there was inadequate Project Cycle Management (PCM) skills and management capacity to sustain TEIP independently by TEDs.

Table 8: Opinions on foundations needed for sustainability of TEIP (October – December 2015)

Foundations needed for sustainability of TEIP Few Moderate High Did not know

% % % % TEIP received support from relevant government departments, non-governmental organizations and state/region governments

School (n=246)

15.0 22.0 2.8 60.2

TED (n=17)

29.4 23.5 5.9 41.2

Key stakeholders already adopted promising practices of TEIP

School (n=249)

16.5 26.9 6.8 49.8

TED (n=17)

35.3 47.1 11.8 5.9

Township Education Departments already possessed adequate Project Cycle Management (PCM) skills

School (n=249)

16.1 31.7 4.4 47.8

TED (n=17)

29.4

35.3 29.4 5.9

Township Education Departments already possessed adequate system and capacity regarding organizational management, financial management and M&E

School (n=247)

17.4 27.1 4.9 50.6

TED (n=16)

25 37.5 25 12.5



44

Is the current design for supporting TEIP activities appropriate and viable for replication in other townships – and which specific elements, if any, need to be strengthened? Are there any other Government of Myanmar or non-government partners that need to be involved?

To assess how appropriate and viable TEIP is for replication and how to strengthen it, the following areas were investigated: (1) organizational structure; (2) financial resources; (3) operational efficiency; and (4) operational capacity.

Only two to three TEIP committee members per township were found to have acquired enough capacity in needs assessment and planning. The failure to implement TEIPs prevented TEOs and other committee members from gaining experience and capacity in implementation, M&E, and sustainability strategies. Such suboptimal operational capacity is unlikely to support the roll-out of a fully-fledged decentralised planning and implementation modality in the future. A review of the budgets calculated by the 11 TEIPs provided to the evaluation team indicated TEIP budgets were significantly under-calculated and that a large amount of additional resources were needed. This suggests that even if funds had been made available from the MOE or QBEP, the financial needs stated in the TEIPs would have been inaccurate and too low for proper or full implementation of plans. This indicates that activities concerning budget creation (either through training activities or through review and feedback mechanisms) need improvement. Competing tasks imposed on key officials, weak organizational, financial and monitoring & evaluation systems, and insufficient human resources capacity of TEDs, DEDs and SEDs collectively undermined the operational efficiency and capacity for further replication in the same townships or in additional townships.

5. CONCLUSIONS

This evaluation had four key objectives, which are outlined below before addressing each research area more specifically.

1. Test programming assumptions and the underlying theory of change: A key assumption underlying TEIP was that there was funding available to carry out the intended project activities of training and building capacity of TEOs in creating TEIPs and implementing them. Funding was not available to undertake the implementation stage of TEIPs. A second key assumption was that the operating environment was ready for activities supporting decentralisation at the subnational level. This was not found to be the case. The theory of change for TEIP was developed retrospectively and reflects the changed objective of TEIP to being a capacity building exercise only, rather than for implementation as was originally conceived.


The current design of TEIP activities is limited to building capacity for the development of township education improvement plans and the training activities in place to enable this have only been sufficiently effective in producing lasting skills in two to three members of each township committee.

There is no funding for the implementation of TEIPs and TEOs and other stakeholders involved in TEIP development have not had the chance to develop the operational or M&E skills that would provide more depth to TEIP activities.

Budget training and constructive feedback on TEIP budgets created appears in need of strengthening.

Other non-government entities have not been well-involved in TEIP development and this has been identified throughout the report as an overall weakness.


The current design of TEIP activities is limited to building capacity for the development of township education improvement plans and the training activities in place to enable this have only been sufficiently effective in producing lasting skills in two to three members of each township committee.

There is no funding for the implementation of TEIPs and TEOs and other stakeholders involved in TEIP development have not had the chance to develop the operational or M&E skills that would provide more depth to TEIP activities.

Budget training and constructive feedback on TEIP budgets created appears in need of strengthening.

Other non-government entities have not been well-involved in TEIP development and this has been identified throughout the report as an overall weakness.


45

2. Identify areas where implementation can be strengthened under QBEP for possible expansion under the NESP: Analysis showed that all essential steps of the TEIP can be strengthened, and indeed that these steps can be redesigned. Strengthening implantation can be done by:

o Obtaining strong and consistent political support for the programme, before any expansion takes place;

o Using evaluation findings and undertaking additional needs assessments for a new programme design;

o Clarifying unclear objectives and ensuring they are informed by a series of robust preparatory activities such as conducting a risk analysis and stakeholder analysis, developing mitigation plans and plans on how to work with stakeholders, developing logical and clear linkages between impact, outcome, output, process, and input;

o Undertaking baseline measurements before any expansion takes place; o Developing a detailed implementation plan using the baseline findings; and o Developing an M&E system during the design stage of any expanded programme and

revising, with clear records and justifications, in the course of implementation.

3. Explore a finding from the QBEP midterm review questioning the relevance of its TEIP activities: The idea of TEIP was relevant even in the transitional period of democracy of Myanmar because this could encourage effective transition and strengthen foundations for a future decentralised system. However, numerous weaknesses of the TEIP project made the project less relevant when it was implemented.

4. Respond to an overarching midterm review recommendation that QBEP assess its activities before scaling up: Evaluators found the midterm recommendation to be highly relevant. The evaluation found that the current design of TEIP will not suitable for scale up at all unless significant changes are made to the design, implementation and monitoring of TEIP. Unless all these stages are reconsidered and unless rigorous planning activities are undertaken, finite resources that Myanmar needs for alternative programmes and priority areas will be wasted on the expansion of TEIP. Failure to invest considerable time in consulting with the new government and understanding the political climate will likely result in any future iteration of TEIP repeating the same problems and challenges outlined in this report.

Are TEIP activities relevant to the planning needs of TEOs to manage the delivery of basic education in their townships?

TEIP was launched at a time when Myanmar was undergoing a transition to genuine democracy and decentralisation, which made the concept of TEIP relevant to the country’s political and administrative context. TEIP activities, as a support to the decentralised planning and management of education services, could have offered an opportunity to capture the specific needs of individual townships and respond to them through designing and delivering tailored solutions. It could have strengthened management capacity to deliver these tailored education services at the subnational level, both conceptually and in practice. In sum, the concept of developing skills normally confined to the central MOE levels, in support of decentralisation was relevant. The evaluation found that the MOE, as implementer, and UNICEF, as a supporting agency for technical and financial assistance, accomplished the development of second draft TEIPs in 25 out of 34 target townships. TEIP training activities led to promising practices that could be replicated in potential future decentralised programmes/projects. The key achievements of note were:

• Producing a pool of capable and committed TEIP workshop facilitators and TEIP committee members who could continue to facilitate future iterations of TEIP;

• Strong participation of women in the role of workshop facilitators contributing to improved gender balances in MOE;


46

• Initiating the practice of undertaking needs assessments and planning by a group of first line and second line leaders at the township level; and

• Attitudinal change in recognising the importance of collecting valid data and utilising proper data analysis for planning at school and township levels.

However, the project was judged to have met only part of the criteria used for assessment of relevance, effectiveness, efficiency and sustainability. This under-achievement was found to be due to a number of key factors, mostly related to the limitation of TEIPs not being implemented in any townships, although other impeding factors were also found. That capacity was developed through TEIP is evident, and it is very positive that unintended people were able to benefit from participation in the TEIP process. However, the extent to which the activities were relevant to TEO needs is less conclusive. The reconstructed needs identified at the beginning of the evaluation (through workshops) are not certain to be reflective of the needs at the start of TEIP; and of those that were identified, half were not directly addressed through activities because of the lack of plan implementation. It is, however, conceivable that the attitudinal changes evidenced by TEIP involvement may still indirectly contribute to the skills related to implementation. The second draft plans developed by committees were not approved at the district or state/region levels. Even if funding had been available, this would have posed an obstacle to implementation of the plans. In spite of decentralised policies, the persistent reality remains of a deeply entrenched centralised system that restricts subnational education offices from making decisions without higher authorisation. It appears that TEIP was developed with weak input from decision makers in the MOE, resulting in unclear directives from the central level to lower administrative units regarding their role in TEIP. The absence of clear instructions to state/region and district education offices is a partial reason why these bodies did not act according to the planned TEIP stages. This is an indication that TEIP activities were not wholly relevant to the realities of the government context. In addition, the basic planning and management of TEIP, from internal documentation and usage of needs assessment findings, to documentation of project changes, and absence of M&E documents meant that the project's overall objectives remained unclear. TEIP stakeholders demonstrated differing understandings of the project, broadly split into those who considered it a purely capacity building exercise and others who expected implementation of the plans to be integral to the project.

Again, the concept of TEIP was very relevant to improving the education sector at the subnational level because it was intended to train TEOs in discerning the specific needs of diverse populations and to imbed tailored intervention measures to address these needs. However, in practice, the township education plans that were developed were not found to capture the needs of their most vulnerable populations, which meant that the plans did not include measures to address inclusiveness and equity concerning access to and quality of education. Targeting vulnerable population needs is highly necessary; it is therefore commendable that the project sought to address this area, but findings suggest that further improvement and focus would be needed to make this more meaningful in any future TEIP iteration.

The unknown specifics of decentralisation at the time of TEIP conception and the inability to predict how fast, or otherwise, the progress of meaningful decentralisation in Myanmar would take may account for the misalignment of activities to needs. However, it is arguable that such a risk could have been highlighted and mitigated if a TEIP-specific political economy or risk analysis had been conducted at the start of the project. Similarly, with more pre-project preparation, the uncertain sources of funding could have been identified at an earlier stage before the project was allowed to run ahead on untested assumptions and be required to halt half of its intended activities.


47

To what degree are TEIP activities effective and efficient in developing plans that are inclusive, informed and realistic?

Effectiveness The TEIP project achieved partial fulfilment of its effectiveness by meeting one intended result. At the time of evaluation, almost three-quarters of the townships or 25 of 34 townships had completed a second draft TEIP – a first draft of TEIP was reviewed and revised to produce a second draft, but none of these had yet been approved officially. During the course of the evaluation, the team found several promising effective practices that could be replicated in the future. Notable accomplishments were the participation of lower level MOE staff members and the consideration taken of basic township level needs and context in the course of developing the first draft of a manual, as well as the creation of a pool of TEIP facilitators. The latter became operational and able to assist TEIP committee members in acquiring knowledge and skills and in completing second versions of TEIP, which was an unintended but notable achievement. Both TEIP committee members and facilitators (comprising approximately equal numbers of men and women) gained a certain degree of capacity in needs assessment and planning. However, a key barrier to achieving the rest of the expected results stipulated in the manual was the non-implementation of TEIPs. Evaluators have noted UNICEF's response that these results are not relevant to the changed objectives of TEIP, whereby implementation was reportedly not intended until a later date. Being responsive and adaptable to changed understandings of conditions is positive and commendable; however, as these changes were not recorded and therefore not verifiable, the original expected results remained the basis for assessment. The whole state approach was found to have produced several benefits: (a) strengthening needs assessment and planning capacity of township, district and state level staff; (b) strengthened relationships and collaboration between UNICEF and MOE at different levels; (c) trust building between the MOE and the MNEC; (d) extended provision of capacity building such as TEIP and SITE and financial and material support to impoverished MNEC schools; (e) access to school grant scheme by MNEC schools; and (f) gradual convergence of two parallel education systems of the MOE and MNEC. It should be noted that some of these benefits were also related to relevance and sustainability. However, inadequate investigation into the needs of the most vulnerable populations and insufficient in-depth analysis of root causes of education sector prohibited TEIP from including tailored solutions for the most disadvantaged groups and root causes. This thereby minimised the potential for greater effectiveness, efficiency, equity, inclusiveness and ensuing impact. The evaluation team discovered several lessons learned from TEIP townships and from international decentralised processes for potential replication within Myanmar. Significant ones highlighted the importance of involving key stakeholders and key education service providers and of managing the decentralisation process through committees that included the participation of key stakeholders and local community members. Related, was the importance of clearly delineated roles and responsibilities, appropriate delegation of authority, and autonomy of such committees, in addition to adequate capacity building. Other lessons included the need to strengthen management systems at the subnational level, having a clear and concise policy, instructions and political support from the national level, and mobilisation and allocation of adequate resources. Finally, the evaluation measured the key factors that have influenced the education sector and key indicators of access to and quality of education for two periods: before the TEIP project began; and during the implementation of the TEIP project. This analysis indicated that trends of these factors had improved


48

and that QBEP funded activities and the recent increased investment in education by the MOE could claim contribution to these improvements. Efficiency Evaluators found that several factors undermined the efficiency of the project, including (inter-alia) varying degrees of ownership, accountability and leadership attributes, insufficient time spent on TEIP, inadequate capacity acquired by TEIP committee members, insufficient resources invested and numerous competing tasks of TEIP committee members. Positively, the overall human resources, TED materials, and out of pocket money spent by TEIP committee members used in the course of TEIP were not found to have been a significant barrier to effectively carrying out TEIPs or to have been an undue burden on individuals and the system. Thus, TEIP did not adversely exhaust resources that could have been exploited for better opportunities or routine work. Low intensity intervention with utilisation of a low level of resources could, understandably, produce less significant unintended consequences concerning resource utilisation.

What elements of the TEIP activities should be sustained as part of NESP following the end of the current QBEP funding?

Three elements that could support future implementation of TEIP and other similar decentralisation initiatives were found to be in place. Firstly, considerable levels of political support, concern and interest were shown and/or provided to the education sector by government departments and state/region governments. Secondly, an overwhelming proportion of study participants expressed their support for decentralised planning and implementation of education services, as well as for establishing and operating school funds and township level education funds. This is important, as support from key stakeholders was found to be one of the major elements of sustainability of a decentralised process. Thirdly, half of the TEIP townships sampled and the one non-TEIP township had established and were running-township level education funds or foundations, and many schools had their own school fund mechanism. Having a fund mechanism was found to be one of the key enabling factors for sustainability. Aside from these three facilitating factors, other essential elements for sustaining TEIP and other similar decentralised initiatives were not found to be sufficiently functional. As such, the current capacity of schools, township, district, and state/region levels regarding results based management, was still found to be low as discussed under relevance and effectiveness sections. In addition, shortage of human resources, particularly for management systems of schools and township, district, and state/region education departments, could not support sustainability. Besides, inadequate capacity and systems regarding organization management, financial management, and M&E of schools, and township, district and state/region education departments was not conducive for longer term sustainability. The evaluation found that TEIPs did not include a proper exit strategy or a mechanism for incorporating lessons learned. Nor did TEIPs incorporate a system to consolidate need analyses and solutions from school to township, district, and state/region education departments as currently needs analysis and planning were undertaken at different levels separately. In addition, there was the presence of a significant impediment by the centralised organizational culture for future decentralised process. Of particular importance was insufficient and unclear delegation of authority and autonomy while responsibility was being handed out to subnational levels. Assigning responsibility without delegating authority would deter full functioning of decentralisation. Finally, the lack of mechanisms to mobilise and provide adequate resources for TEIP implementation was another crucial restriction to project sustainability.


49

6. LESSONS LEARNED

Several lessons can be taken from this evaluation, primarily in terms of the importance of rigorous project design and management. These lessons can be used to inform better practice through relatively basic processes, and they should be noted by UNICEF for the improvement of all future projects as well as for any potential iteration of TEIP.

The importance of rigorous planning and project-specific design activities (as opposed to overall programme design activities): The evaluation appreciates that the key responsible persons of the MOE and UNICEF made efforts to plan and implement this project of decentralised planning and management in a political, economic, social and cultural environment that was not entirely conducive for accomplishing the results that were originally intended. The challenges imposed by such a context appear to have encouraged key players to alter both expected results and processes of TEIP throughout the course of the project. The evaluation considers this one plausible explanation for not achieving all the objectives the project aimed to have achieved. If the above explanation is indeed founded, the evaluation assesses that the designers, managers, and implementers of the project failed to take essential measures at all stages of design, implementation, and M&E. Missing elements of project design such as lack of TEIP-specific theory of change or results chain developed led to unclear project objectives, which were exacerbated by changes made in the course of the project. In addition, during the design stage a risk assessment and plan to mitigate the risks should have been properly drawn up. This measure could have captured the absence of an enabling environment for decentralised planning and management of education services and consequently, there would have been scope to completely reconsider the feasibility of the TEIP project as a whole. While there are sections relating to TEIP and decentralisation in overall QBEP documents such as the 2012 design document and MDEF 2 linked annexes, these were not detailed enough to provide sufficient contextual analysis or guidance for the rollout of this project. These examples have shown the importance of the foundation stages of project design, and how their absence can lead to projects with unclear relevance and contextual feasibility.

The necessity of systematic project documentation and communication: During the implementation stage, challenges should have been documented as they occurred; in particular, the alteration of the project design should have taken place in consultation with key stakeholders and with their approval. The change should have been thoroughly documented with a description of the rationale for change and who the decision makers were. This should then have been communicated to key stakeholders to ensure uniform and consistent understanding of the project's changed objectives. In addition, what project documents exist should be labelled and stored clearly for ready access; this would support internal institutional organization as well as aid external evaluation. These examples have shown how document management can have a serious impact on a project's implementation, success and evaluation.

The need to facilitate better conditions for monitoring and evaluation: During the monitoring and evaluation stage, challenges and progresses of the project should have been recorded through regular progress reports, which could then have been taken into account when undertaking the end of the project evaluation. This would have helped the evaluation set more exact evaluation questions and criteria, lines of inquiry, and evaluation methods. The evaluation did not discover any evidence that the project managers and implementers had taken these measures. This is an especially important lesson to learn from, considering that one of QBEP's stated outputs is for improved M&E capacity in the MOE.

The impact of weak stakeholder engagement: It is clear that weak involvement of decision makers from the MOE had a detrimental effect on the project's overall relevance and this is a key lesson for any future design. Adequate involvement of decision makers at the offset could have


50

prevented TEIP from derailing. For example, decision makers may well have rejected the idea of decentralised planning and management at the outset given the unfavourable political, social, economic and cultural environments.

Table 9: Lessons learned and suggestions for incorporation into future projects

Lesson learned How to incorporate lesson learnt

Why?

Importance of project-specific design activities and planning

Needs assessment Ensure relevance of project activities

Political economy analysis Test assumptions of operating environment

Risk analysis Identify challenges in order to provide mitigating action or even to reconsider project feasibility

Theory of change prior to start of activities

Ensure clear project objectives

Importance of assessing funding sources

Financial assessment Ensure the plausibility of implementing project activities

Importance of systematic document management

Documentation of any project changes

Ensure project objectives remain clear for stakeholders and assist evaluation process

Proper filing system and regular progress reports

Support institutional project knowledge and assist evaluation process

Importance of inclusive stakeholder engagement

Involve government decision makers at design stage

Ensure project is relevant and feasible in country context

Involve beneficiary stakeholders

Access to on-the-ground input and feedback on project relevance and activities

7. RECOMMENDATIONS

The following recommendations are organised in order of priority.

Clearer decision-making for decentralised planning and implementation: For policy makers and decision makers of MOE and UNICEF and donors It is recommended that the objective of any future replication of TEIP is made clearer. Specifically, it is recommended that the objective should be decentralised planning and implementation of tailored intervention measures that address specific needs of the education sector. This should be complementary to the development of other sectors at the national and subnational level. The timeframe is dependent on whether a future iteration or expansion of TEIP is agreed upon; if so, this should be an immediate priority and be addressed in the first stages of design. An example situation being: if, in a given township that relies on agriculture and is in need of skilled workers, school lessons do not currently provide students with the essential skills needed to pursue a livelihood in this sector, this shows that the education system is not fulfilling the needs of its students nor the township's other sectors. A decentralised education system in any township should take into consideration the needs of broader, non-education sectors. To support this undertaking, key policy and decision makers from the MOE, UNICEF and the donors should make a clear decision regarding the actual roll-out of decentralised planning and implementation of the education sector. They should be fully aware of and prepared to support essential elements of the decentralisation process: by producing clear policies and instruction at the Union level; by strengthening organizational and financial management systems as well as M&E systems at the subnational level; by delegating appropriate authority and autonomy to the subnational level; by making sufficient resources available; by making efforts to break down the organizational culture influenced by the centralised management system; by encouraging changed attitudes of MOE staff members to make them more open


51

to working with key stakeholders and community members; and by making the education sector become more community based. UNICEF can play a crucial role in facilitating the decentralisation of government systems. First, it can implement evidence-based advocacy targeted at decision makers of the national government using lessons learned from appropriate modalities from other similar contexts to adopt appropriate forms of decentralisation. UNICEF can strengthen the capacity of key stakeholders at the national level and the subnational level regarding key elements of decentralisation such as devolution, deconcentration, deregulation, and privatisation and regarding management systems. Third, necessary resources should be made available for designing, implementing, evaluating and revising a pilot modality if further scaling up takes place.

Need to address root causes of bottlenecks for greater effectiveness and efficient use of resources: For policy makers and decision makers of the MOE and UNICEF and donors It is paramount that the future decentralised planning and implementation of the education sector be conceptualised and implemented to address fundamental root causes of education bottlenecks and their consequences such as the examination system, quality of teaching and competency of students, utility of education in real life, and quantity versus quality. Addressing these fundamental issues will enhance the effectiveness and efficiency of activities and maximise the value of finite, precious resources invested. As above, the timeframe for this recommendation is dependent on whether a future iteration or expansion of TEIP is agreed upon; if so, this should be addressed in the immediate design period. Integration of different sectors and education service providers to promote mutually complementary development and inclusiveness and equity of education: For schools and Township Education Departments, District Education Departments, and State Education Departments Future decentralised projects or programmes should ensure that at each important stage of the project or programme, relevant stakeholders from other sectors such as Township Government Administrative Departments and the Department of Rural Development, private schools, monastic schools and NGOs that provide education services or funds are involved. This approach will help TEIP integrate with economic, health, and other sectors so that joined up actions will occur to support broader township level development. The timeframe for this recommendation is ongoing throughout the project lifespan. Monastic schools and NGO education projects are the best actors at reaching the most marginalised groups, so further interaction with them could help TEIP provide more inclusive and equitable education services. Future TEIP iterations should identify the needs of the most vulnerable groups of students, of targeted areas, and of schools within each township and develop specific intervention measures that are tailored to their unique needs and vulnerabilities. Disaggregate analysis of quantitative and qualitative data should be undertaken to examine the needs of different genders, poor families, various types of migrants, people with disabilities, and different ethnic groups. An example of a tailored intervention is the implementation of community-based non-formal education, in conjunction with formal schooling, to transfer basic literacy and numeracy, life skills and livelihood skills for students of poor families who are likely to drop out from formal schools to earn an income. This approach will distinguish a decentralised TEIP from a centralised TEIP in that it can tackle local needs rather than offering generic education services that do not address specific township level needs. This evaluation provided information that can be used as basis for designing future programmes to address the needs of the most vulnerable groups and root causes.

Strengthening the entire process of results based management for greater effectiveness, efficiency, and sustainability: For facilitators and programme managers of TEIP from MOE and UNICEF


52

Any future TEIP design should encompass the entire results based management cycle with the involvement of key stakeholders. This recommendation should commence during the design stage and the timeframe should be considered the lifespan of the programme. Each step of the results based management cycle should be designed and implemented fully and rigorously. The conceptualisation stage should be carried out with meaningful consultation of key stakeholders and embed an appropriate evaluation design. Next, appropriate baseline measurements should be undertaken. Then, baseline measurements should be utilised to draw up a detailed implementation plan. Following this, an M&E system should be designed at the beginning of the project and then be implemented regularly and consistently. Adequate documentation is also essential to enhance the utility of an M&E system. Embedding logical links and three essential attributes of programme impact: For policy makers and decision makers of MOE and UNICEF, and facilitators, programme managers and implementers of MOE Impact and outcome are a function of coverage, quality, and duration of implementation of any activity, strategy, project, or programme. Only when these three attributes are sufficient within each type of activity, will outcome and impact level changes be attained. Furthermore, there should be logical links among input, process and output. Only when a sufficient amount of inputs is invested in, can desired activities be implemented, and similarly, adequate implementation of processes will result in achieving targeted outputs. Step-wise gradual expansion and whole district approach: For policy makers and decision makers of MOE and UNICEF and donors Before scaling up TEIP, it is essential that all the above recommendations are addressed and the current modality is modified. Once these areas have been covered, a step-wise gradual expansion approach should be applied via two phases. Each phase should be implemented over a three to five year project life-span to ensure that the duration of implementation is sufficient to trigger outcome level results and follow a thorough results based management process. Only when the entire project cycle of the first phase is completed, should a rigorous evaluation be completed to learn lessons that will inform a refined modality for scaling up to the second phase. During the first phase, it is recommended that the whole district approach should be applied for two primary reasons. First, the newly created district level education office does not have a great deal of decision-making power and has been bypassed by many projects and programmes. If this continues, the district office will never have the opportunity to be utilised and will remain a peripheral entity, thereby undermining the subnational system that the project is intended to support. Second, an expansion into the whole state/region is likely to encounter numerous, challenging barriers such as diversity of ethnic groups and needs, disparity between capacity and management systems, heavy workload, and difficult transportation and communication. In this first phase, the existing capacity as well as quality and utility of township education improvement plans in the current TEIP townships should be consolidated by covering all different levels of the education system such as schools, cluster, catchment area, and township level first. After that, the capacity of other townships within a same district should be strengthened to develop and implement TEIPs. Management systems of school, township, and district levels should also be strengthened. A system to consolidate the needs and solutions of individual schools to develop a cluster level plan, followed by a catchment area level and a township level, should be set in place. An exit strategy should be designed and put into place from the moment of project inception. During the second phase, after consolidation has meaningfully taken place at the district level, expansion can move onto the state/regional level. Adequate supervision and support to all levels of programme For policy makers and decision makers of MOE and UNICEF and programme managers of MOE


53

To make the TEIP process more effective and efficient, experienced individuals or institutions should provide mentoring to TEOs for the entire process of the results based management cycle. This should initially be implemented in one district or one batch of districts. Receiving mentoring from a group of skilled coaches will strengthen the capacity of the existing TEIP facilitators from the MOE; they will then be able to take the lead in training successive batches of facilitators. As and when mentoring expands beyond the initial district or batch of districts, close supervision and support should be regularly and consistently provided to state/region, district, township, catchment, cluster, and school level stakeholders. Building capacity along the management line of the MOE both vertically and across key stakeholders at each level horizontally is essential.


54

BIBLIOGRAPHY

Brennan, Geoffrey and James M. Buchanan, The Power to Tax: Analytical Foundations of a Fiscal Constitution, Cambridge: Cambridge University Press, 1980.

Busemeyer, M.R, Two decades of decentralisation in education governance: Lessons learned and future outlook for local stakeholders, OECD Conference on Effective Local Governance in Education, Warsaw, Poland, April 16, 2012, Warsaw, Poland, 2012.

Creswell, J. W., Qualitative inquiry and research design: choosing among five traditions, SAGE, Thousand Oaks, California, USA, 1998.

Creswell, J.W., Research Design: Qualitative, Quantitative and Mixed Methods Approaches, 2nd Ed., SAGE, Thousand Oaks, California, USA, 2003.

Fiske, Edward B., Decentralisation of Education: Politics and Consensus. World Bank, Washington, D.C., 1996.

Hanson, Mark, Education Decentralisation: Issues and Challenges. Partnership for the Educational Revitalisation in the Americas, No. 9, 1997.

Hawthorne, G., Introduction to Health Program Evaluation, Melbourne, Victoria, Australia, 2000.

Heredia-Ortiz, E., The Impact of Education Decentralisation on Education Output: A Cross-Country Study, Dissertation for PhD, Georgia State University, USA, 2007.

Lobo, T., Guedes, A., Amaral, A.L. and Walker, R., Decentralised Management of Education in Minas Gerais, Brazil, In World Bank, Seminar on Education Decentralisation, Washington D.C: World Bank, 1995.

McGinn, N. and Welsh, T., Decentralisation of Education: Why, When, What and How?, Edited by Fundamentals of Educational Planning Series: UNESCO, 1999.

MDEF, MOE, UNICEF, QBEP Midterm Review, 2014.

MDRI and Asia Foundation, State and Region Governments in Myanmar, MDRI and Asia Foundation, Yangon, Myanmar, 2013.

MOE, Township Education Improvement Plan Version 5, MOE, Nay Pyi Daw, Myanmar, 2013.

MOE, National EFA Review Report, MOE, Nay Pyi Daw, Myanmar, 2014a.

MOE, CESR Phase-2 Report, MOE, Nay Pyi Daw, Myanmar, 2014b.

Morse, J. M. and Field, P. A., Qualitative research methods for health professionals, SAGE, Thousand Oaks, California, USA, 1995.

Parry, T.R., Decentralisation and Privatisation: Education Policy in Chile, Journal of Public Policy 17, No. 1: 107-33, 1997.

Patrinos, H.A., Market Forces in Education, European Journal of Education 35, No. 1: 61-80, 2000.

Patton, Q. M., How to use qualitative methods in evaluation, SAGE, Newsbury Park, California, USA, 1987.

UNEG, UNEG Ethical Guidelines for Evaluation, United Nations Evaluation Group, 2008.

UNICEF, Annual Report on Quality Basic Education Program, UNICEF, Yangon, Myanmar, 2012.

UNICEF, 2012 QBEP Design Document, UNICEF, Yangon, Myanmar, 2012.

UNICEF, MDEF 2 Linked Annexes, UNICEF, Yangon, Myanmar, 2012.

UNICEF, Annual Report on Quality Basic Education Program, UNICEF, Yangon, Myanmar, 2014.

UNICEF, Joint Performance Improvement Plan, UNICEF, Yangon, Myanmar, 2014.

UNICEF, UNICEF Procedure for Ethnical Standards in Research, Evaluation, Data Collection and Analysis, UNICEF, 2015.

USAID, Decentralisation in Education, Educational Quality in the Developing World 3, No.4: 1-4, 2005.

Vegas, E., School Choice, Student Performance, and Teacher and School Characteristics: The Chilean Case: The World Bank, Policy Research Working Paper Series: 2833, 1999.

Zobrist, Brooke and P. McCormick, A Preliminary Assessment of Decentralisation and Education, The Asia Foundation, Yangon, Myanmar, 2013.


55

ANNEXES

Annex 1: Terms of reference

Terms of Reference: Evaluation of QBEPs Township Education Improvement Plan (TEIP) Activities Section in Charge: Education How does the consultancy relate to education portfolio: Demonstrates renewed focus to more systematically and rigorously assess the performance of our activities and using that assessment for evidence-based decision-making. Objective reference: QBEP Objective 1: Systems Supporting Quality Basic Education Strengthened. Sub-Objective reference: MOE Capacity for Planning, Monitoring, and Evaluation Improved. 105: Basic Education and Gender Equality Programme Output 4: Planning and Management

1. Background: As a part of its larger, four-year Quality Basic Education Programme (QBEP), UNICEF and its Myanmar Multi-Donor Education Fund (MDEF) partners (Australia, Denmark European Union, Norway, and United Kingdom) have since May 2013 been working with select Township Education Offices (TEOs) in select states/regions to strengthen their ability to plan, monitor, and manage education activities within their respective townships. A 2011 UNICEF situational analysis identified that even though the TEOs occupy a “nodal point” in basic education delivery at the subnational level, the managerial capacity of TEOs are limited, particularly in terms of strategic planning and monitoring. In response, QBEP activities focused on providing TEOs with training and technical support to undertake their own situational analyses of education in their townships. TEOs are then guided on how to develop a three-year Township Education Improvement Plan (TEIP) to respond to any weaknesses identified and build upon identified priorities. The purpose of the TEIP is to serve as an official document that sets out the township’s long term vision, goals and objectives and programs of activities for the improve education services in the township. It is intended to set out a phased and rolling program of priority actions identified by the township education offices and guided by the government’s 20-year education plan and Education for All (EFA) goals and targets. All township plans will share and strive towards the same national goals and priorities, especially on implementation of compulsory primary education. In order to do so, all townships are expected to embark on their own education strategic planning process, with support from the state and district education authorities and other sources. In summary, the TEIP activities are designed to better equip TEOs, the key stakeholders of the TEIP activity, to engage in a more inclusive and transparent planning process responsive to local needs and priorities and to strengthen the management of primary education at the township level. These activities are assumed to be needed because the current limited capacity, coordination, and collaboration within TEOs and vertically to higher level education governance structures is restricting the level of effective planning, management, and delivery of quality basic education at the subnational level. To develop their TEIPs, TEOs are provided with training and technical guidance on how to: solicit input from a variety of local and national stakeholders on the primary education needs of their townships; utilise that information to develop a situational analysis of the education needs of their townships; and in turn develop a three-year TEIP for responding to those needs and priorities. The TEIP activities fall under Objective 1: Systems Supporting Quality Basic Education Strengthened of QBEP’s overarching logical framework. The specific Theory of Change guiding the activities can be conceptualised as follows: IF TEOs are provided with training and technical guidance on how to: (1) solicit input from a variety of local and national stakeholders on the primary education needs of their townships; (2) utilise that information to develop a situational analysis of the education needs of their townships; and in turn (3) develop a three-year TEIP for responding to those needs and priorities;


56

THEN TEOs will be better equipped to engage in a more inclusive and transparent planning process responsive to local needs and priorities and to strengthen the management of primary education at the township level; BECAUSE the current limited capacity, coordination, and collaboration within TEOs and vertically to higher level education governance structures is restricting the level of effective planning, management, and delivery of quality basic education at the subnational level.

2. Objectives of the Evaluation: Purpose: The purpose of this evaluation is assess QBEP’s TEIP and to strengthen its ongoing implementation, and learn lessons for future application. To this end UNICEF is seeking an external evaluation team to conduct a performance evaluation of its TEIP activities that is both formative and summative in nature as UNICEF is both interested in assessing the current TEIP models, as well as soliciting input in how that model can be strengthened for future implementation. The team, comprised of a mix of international and national consultants, will be expected to provide an assessment of the relevance, effectiveness, efficiency, and likely sustainability of QBEP’s TEIP activities to date, as well as identify lessons and formulate recommendations for UNICEF’s consideration. Specifically, the evaluation will be used by the UNICEF program management team and its MDEF partners to: (1) test programming assumptions and its underlying ToC; (2) identify areas where implementation can be strengthened under QBEP for possible expansion under the NESP; (3) explore a finding from an August 2013 QBEP Midterm Review (MTR) questioning the relevance of its TEIP activities; and (4) respond to an overarching MTR recommendation that QBEP assess its activities before scaling up. The primary users of the evaluation are relevant Ministry of Education (MOE) officials and TEOs, as well as the MDEF partners. Evaluation Scope: The evaluation will provide an independent assessment of QBEP’s TEIP activities based on the OECD/DAC criteria listed in the evaluation questions below. The evaluation should include review of the TEIP process throughout all five steps of TEIP development: (1) data collection; (2) TEIP drafting; (3) Follow-Up; (4) TEIP Finalisation; and (5) verification and follow-up. As TEIP has been rolled in various townships using a phased approach, the evaluation team must ensure that their sampling includes townships at each phase of TEIP development. In addition, as the evaluation team will be expected to provide a comparative assessment of other subnational education planning activities, the team should plan to travel to both QBEP and non-QBEP townships, including townships considered “worst off.” The team will be asked to provide a draft sampling plan drawn from a frame of all 34 QBEP townships and other comparison townships with their proposals. Primary Questions: The final evaluation report will be expected to provide complete, evidence-based answers to following evaluation questions, based on the OECD/DAC criteria: Are TEIP activities relevant to the planning needs of TEOs to manage the delivery of basic education in their townships?

Did the activities respond to the capacity needs of TEOs?

Are the QBEP-supported TEIPs responsive, equitable, and inclusive of their respective township education needs?

Are there other current systems or initiatives currently in place that are more relevant in meeting these planning needs?

To what degree are TEIP activities effective and efficient in developing education plans that are inclusive, informed, and realistic?

Are the activities producing the results originally intended?



Are there any ways in which the activities can be improved, either through changes in their current implementation or based on good practices from similar activities in Myanmar and/or elsewhere in the region?

What elements of the TEIP activities should be sustained as part of NESP following the end of the current QBEP funding?

Is capacity available for the TEIP activities to continue to be carried out at the township level following the end of QBEP?



57

Is the current design for supporting TEIP activities appropriate and viable for replication in other townships – and which specific elements need strengthening?

Evaluation Design: While the offeror is encouraged to propose their own evaluation methods, it is expected that the evaluation will follow a qualitative-dominant, participatory, but still mixed-methods design. It is expected that the team will use some or all of the following methods:

Field observations;

Key informant interviews;

Qualitative baseline interviews;

Quantitative beneficiary surveys;

Data analysis and documentary research;

Use appropriate sampling methods. The offeror should assume that a minimal level of M&E data presently exists and is encouraged to propose an approach that is responsive to such data limitations. Norms and Standards: These guidance documents will be part of the contract of the evaluation team:

United Nations Evaluation Group (UNEG) Standards for Evaluation in the UN System, 2005;

United Nations Evaluation Group (UNEG) Norms for Evaluation in the UN System, 2005 (including impartiality, independence, quality, transparency, consultative process).

UNICEF Ethical Standards for Research, Evaluation, Data Collection and Analysis will guide the overall process The evaluation should incorporate the human rights-based and gender perspective and be based on results based management principles.

3. Duration: The offeror is expected to submit a detailed work plan for how they envision the evaluation activities to be completed. The evaluation is tentatively planned to begin in in late July/early August and end in late September.

4. Management Arrangements: The evaluation team will be managed by the Education Section Evaluation Specialist (Evaluation Manager) that will provide day-to-day management and facilitation of the evaluation process, including day-to-day oversight of the evaluation team, under the guidance of the Evaluation Specialist (Cambodia, Malaysia and Myanmar) and the Education Chief. The evaluation team will be answerable to the Evaluation Manager. The team will decide its own fieldwork programme in consultation with the Evaluation Manager. It will inform the Evaluation Manager of any problems arising. The team will also immediately inform the Evaluation Manager of any issues regarding the integrity or effectiveness of UNICEF’s response encountered during the evaluation research. A Reference Group of immediate stakeholders as well as external experts will be established to ensure quality assurance. The Reference Group will be chaired by the Education Chief. A TOR outlining the roles and responsibilities of the Reference Group will be developed separately. UNICEF partners will be kept informed of the evaluation’s progress on a regular basis. They will be invited to the workshops and consulted on the evaluation outputs.

5. Type of Supervision/Support Required from UNICEF: The evaluation team will work closely with manager and larger UNICEF education team, MOE, and subnational level counterparts. Periodic reporting as agreed upon will be submitted in digital form.

6. Deliverables: The offeror will have ultimate responsibility for the timely and satisfactory submission of all deliverables, that will be reviewed by the Reference Group and Regional Office, and approved by the Evaluation Manager, including:

Tasks End Product/deliverables Duration/ deadline

Drafting and submitting an evaluation inception plan

Inception report to detail expected methodologies (including detailed work plan) to complete tasks required. The Inception Report should outline the overall approach, data collection and analysis plan, evaluation design matrix, sampling plan, tentative fieldwork

Within 10 working days of award


58

schedule, as well as present the draft data collection instruments and outline of the final report.

Participating in an In-Brief (Yangon) and leading data collection in Yangon, Naypyidaw, and first field site

Direct observation, KIIs, and group interviews. As agreed in Inception Report

Leading midpoint check-in Briefing with Evaluation Supervisor to provide an update on progress, any challenges faced, and preliminary impressions.

As agreed in Inception Report

Leading data collection in QBEP and non-QBEP field sites.

Direct observation, KIIs, and group interviews. As agreed in Inception Report

Leading end-of-fieldwork debrief Presentation of initial impressions regarding tentative findings and conclusions.

Final day of fieldwork

Submitting preliminary findings, conclusions, and recommendations matrix.

Presentation of a rough outline of major findings, conclusions, and recommendations and how they relate. A final outline will need to be approved before initiating the drafting of the evaluation report.

Within two weeks following conclusion of fieldwork

Submitting draft report Submission of draft 30-page evaluation report and annexes for QBEP Development Partners’ (DPs’) review and comment.

Within two weeks of approved matrix

Submitting final report Revised and final Evaluation Report Within one week of receipt of comments

Leading final oral presentation of report Presentation and discussion of final report (remote or in-person).

Within one week of submission of final report

7. Request for Proposal: The offeror is asked to submit a: A proposal outlining their draft technical approach, staffing structure, and management and quality assurance plan. A detailed financial bid demonstrating clear congruence with the technical approach submitted. Proposals should be received by 9:00am (Yangon time) on Monday, July 20, 2015.

8. Qualification and specialised knowledge/experience required for the assignment: The evaluation is expected to be conducted by a team of 2-3 experts, 1-2 international and one national evaluation expert. The team should be comprised of at least one male and one female team members. At a minimum, offerors are expected to meet the following qualifications. The international expert(s) will be expect to meet the following minimum qualifications:

At least seven (10) years of experience evaluating and/or implementing education-specific development programs in developing country settings;

Strong previous experience with decentralised educational planning and management;

Advanced university degree in Education or other social sciences; and

Demonstrated ability to work in a multi-cultural environment and establish harmonious and effective working relationships, both within and outside the work place.


59

The mid/senior national expert will be expected to meet the following minimum qualifications:

At least three (3) years of experience managing and/or evaluating educational planning activities in developing country settings;

Demonstrated experience providing written inputs into final evaluation reports;

Demonstrated ability to communicate effectively in both English and Myanmar;

Demonstrated ability to work in a multi-cultural environment and establish harmonious and effective working relationships, both within and outside the work place; and

Familiarity with township and state education planning processes strongly preferred.


60

Annex 2: Revised QBEP results hierarchy


61

Annex 3: Evaluation matrix

Evaluation questions Indicative lines of inquiry, standards or indicators Decisions to inform

Means of verification

Data source Location of data collection

Means of analysis

Did the activities respond to the capacity needs of TEOs?

Alternative lines of enquiry: Did activities enable TEOs: 1. To examine education needs at township level 2. To develop TEIP and understand its purpose and utility. 3. To allocate financial and other resources needed to develop and implement TEIP 4. To monitor and evaluate quality and effectiveness of TEIP

2011 UNICEF Situation Analysis (as mentioned in TOR but availability to be determined). Uncertain availability of situation analyses mentioned in TOR makes it necessary to use evaluation specialist’s parameters of needs.

Triangulation of self-reported needs and perceptions of improvement among different respondents

2011 Situation analysis, township situational analysis reports, focus group discussions (FGD) and in-depth interviews (IDI) with TEOs, TEIPs

5 QBEP intervention townships education offices

Qualitative

Are the QBEP-supported TEIPs responsive, equitable, and inclusive of their respective township education needs?

Indicative definitions to base lines of enquiry: Responsiveness - incorporating solutions to identified education needs of the township level. Equity and inclusiveness of a TEIP - enabling access to basic education of male and female, various ethnic groups, diverse religions, socially marginalised groups such as children infected and affected by HIV, children with four types of disability, children of women headed households and children from migrant families.

Evaluation and education expertise of technical team. UNICEF understanding of equity and inclusivity per PowerPoint.

Triangulation of self -reported perceptions of responsiveness, equity, inclusivity amongst different respondents with desk review of TEIPs

FGD and IDI with - TEOs - Community members - MOE TEIPs


Qualitative

Are there other current systems or initiatives currently in place that are

Ascertain existence of alternative mechanisms or systems in place within intervention and non-intervention townships. Compare alternative

Evaluation and education

Triangulation of self-reported perceptions of

FGD, IDI, self-administered questionnaire

5 QBEP intervention sites and 1

Qualitative and quantitative


62

more relevant in meeting these planning needs?

planning activities' ability to address the criteria cited above against TEIP activities.

expertise of technical team.

quality of alternative planning activities amongst the different groups interviewed against systems with document review of other initiatives

with TEOs, document review of alternative planning mechanisms

non-QBEP comparison site

Are the activities producing the results originally intended?

(1) Greater ownership and leadership of townships in education planning and development process; (2) Greater equity in education provision within and between townships; (3) Improved effectiveness and efficiency in addressing the diverse needs of townships in their education service delivery; (4) Greater budget predictability through multi-year planning and budgeting processes; and (5) Enhanced accountability for education sector performance at all levels of the system.

Intended results of TEIP per the TEIP manual (Version 5).

Self-reported perceptions of improved ability to produce effective and efficient township education plans that are inclusive, informed, and realistic. Attitude, sense of ownership, and leadership attributes of key stakeholders. Governance and check and balance mechanisms. Draft and validated education plans. Budget plan and actual resource allocation.

IDI, FGD, self-administered questionnaires with key stakeholders (more details on stakeholder identification below), TEIPs, programme budget documentation

5 QBEP intervention townships



Indicative lines of enquiry: (1) Greater time consumption that resulted in hampering of other tasks of stakeholders; (2) Higher resource utilisation that can be invested for other tasks with greater benefits – forgone opportunities;

Evaluation and education expertise of technical team.

Comparison of self-reported perceptions of unintended consequences amongst different groups interviewed.

Document review, FGD and IDI with township and DEOs and other key stakeholders, self-administered questionnaires




63

(3) Overburdening human resources with resultant low quality of a TEIP and other tasks; (4) Undermining access to and quality of education as a result of TEIP; and (5) Opposition of TEIP by stakeholders intentionally or unintentionally.

Evidence of unintended results.


Indicative lines of enquiry: To what extent did TEIP plan and actual roll out preclude and/or exclude one or more of the following groups: a. Men and women b. Various ethnic groups; c. Various religious groups; and d. Socially marginalised groups such as children infected and affected by HIV, children with four types of disability, children of women headed households and children from migrant families.

UNICEF guidelines of inclusivity, gender equality, and equity. Evaluation and education expertise of technical team.

Self-reported and actual occurrence of inclusion or exclusion, ascertained from interviews with facilitators, those involved in writing TEIP, review of workshop participant lists, content and language of TEIP.

Document review (including workshop participant lists), FGD and IDI with township and DEOs workshop facilitators, and other key stakeholders, TEIPs


Qualitative

Are resources dedicated to TEIP sufficient to reach intended groups and desired results?

(1) How much in the way of human resources is allocated to TEIP and sufficiency of these resources to reach intended groups and to trigger intended results? (2) How much capacity is acquired to TEIP and sufficiency of capacity to reach intended groups and to trigger intended results? (3) How much ownership of responsible persons is present for TEIP and sufficiency of ownership to reach intended groups and to trigger intended results? (4) How much accountability of responsible persons is present for TEIP and sufficiency of ownership to reach intended groups and to trigger intended results?


Examination of data regarding resources used for TEIP from UNICEF and MOE

FGDs and IDIs with TEIP development committee and working groups, DEOs, SEDs, MOE officials at Nay Pyi Daw, UNICEF staff members and financial report of UNICEF and MOE


Qualitative


64

(5) How much material is allocated to TEIP and sufficiency of material to reach intended groups and to trigger intended results? (6) How much time is allocated to TEIP and sufficiency of time to reach intended groups and to trigger intended results?

Are there any ways in which the activities can be improved, either through changes in their current implementation or based on good practices from similar activities in Myanmar and/or elsewhere in the region?

Measure SWOLs of TEIP and alternative programmes administered in comparison sites.

Per TOR

Self-reported and actual SWOLs indicated by the process and content of each TEIP.

Document review of TEIPs and alternative mechanisms. FGD, IDI, self-administered questionnaires

5 QBEP intervention townships; 1 non-intervention comparison site


Is capacity available for the TEIP activities to continue to be carried out at the township level following the end of QBEP?

Indicative lines of enquiry: Do TEOs have sufficient skills in examining education needs at the township level; allocating financial and other resources needed to develop and implement a TEIP; monitoring and evaluating a TEIP and using findings appropriately. If not, why not? Do TEOs report being likely to actually implement their TEIPs? Is there an exit strategy? What means are there for incorporating lessons learned and alternative planning approaches?

Expected results outlined in TEIP manual.

Triangulation of self-reported perceptions of capacity levels amongst different groups interviewed as well as with review of management systems within Township Education Offices, and available programme design documents.

TEIPs and QBEP design document, FGD, IDI, and self-administered questionnaires with TEOs and other key stakeholders




Indicative lines of enquiry: 1. Involvement of district and state or regional stakeholders in a TEIP process; 2. Presence of a mechanism to compile and consolidate township level education needs at the district and/or state/regional level;


Triangulation of different interviewed groups' responses and cross-check against review of available education plans.

National/State education plans, FGD, IDI, self-administered questionnaire with township and DEOs and




65

3. Tailored solutions of diverse and unique needs of individual township by a district and/or state/region plan; and 4. Receiving political, managerial and technical support from relevant governmental organizations, non –governmental-organizations and state or regional governments and 5. Presence of appropriate organization and financial management capacity in place to facilitate incorporation of township level needs into a district and/or state/region plan.

other key stakeholders.

Is the current design for supporting TEIP activities appropriate and viable for replication in other townships – and which specific elements, if any, need to be strengthened? Are there any other Government of Myanmar or non-government partners that need to be involved?

Strengths, Weaknesses, Opportunities, Limitations of the TEIP and alternative programs being implemented in the comparison sites and other countries. Indicative areas of enquiry: (1) Organizational structure; (2) Financial resources; (3) Operational efficiency; and (4) Operational capacity.

Evaluation and education expertise of technical team, TEIP manual.

Triangulation of self-reported perceptions amongst different groups interviewed with strengths and weaknesses determined by SWOL analysis.

Document review, FGD, IDI, and self-administered questionnaires with key stakeholders

5 QBEP intervention townships; 1 non-intervention comparison site



66

Annex 4: Detailed methodology

Annex 4.1 Sampling method

Qualitative data collection A. First stage sampling of townships A maximum variation case sampling method was employed to select six TEIP townships and one comparison township where a conventional programme was being implemented. A maximum variation case helped the evaluation study to identify a greater degree of variations of the TEIP across localities with diverse operational contexts as well as commonalities that cut across the TEIP programme with diverse operational contexts. Within the limited time and resources, selecting a range of diverse townships across five states and regions assisted the evaluation in examining the most valuable lessons across the different settings. Lessons drawn from a maximum operational context optimised the modification of the future programme. From a sampling frame of 25 QBEP townships that completed TEIP, six townships from within Shan, Rakhine, and Mon States and Bago and Yangon Regions were chosen. For the comparison site, Lashio township from Shan State was chosen because of its similar characteristics to Tachileik township, also in Shan State, namely:

Their proximity to an international border - Lashio to China and Tachileik to Thailand;

Both have relatively booming economic activities and lucrative border trading;

They both have a significant presence of migrants from other parts of Myanmar;

There are strong foreign cultural influences, of Chinese culture in Lashio and Thai culture in Tachileik, which is contributing to preferences among residents to learn in Chinese or Thai education systems rather than the Myanmar system; and

Both are central towns in Shan state that could receive closer supervision and support from higher level of the MOE.

The six intervention sites were chosen in line with UNICEF principles of equity and inclusivity. Accordingly, criteria such as hard to reach areas, areas with most vulnerable populations and areas offering ethnic and religious diversity were applied. The TOR required that sample townships at each of the different TEIP development stages be chosen. Consultation meetings with key stakeholders uncovered that 25 townships completed their plans but none of these townships implemented TEIP. As a result, key stakeholders agreed to select townships that started the process of formulating TEIP at different periods. As shown in the table below, Yae began the process as the fourth batch, Myebon was in the fifth batch, Hlaing Thar Yar and Htain Ta Bin were in the eight batch and Tachileik started as the ninth batch.

State Township Location type Rationale UNICEF criteria met

Shan Tachileik Intervention township

Sizable minority populations such as Shan, Akhar and Lahu; hard to reach communities; religious minority (Christian); influence of Thai culture; influence of Thai education; border area with Thailand; relatively booming economy and lucrative border trade with Thailand.

Hard to reach populations Inclusivity/equity Conflict area Vulnerable populations Worst off status UNICEF State of interest

Rakhine Myebon Intervention township

One of the poorest regions in Myanmar; difficult access to education; inter-communal conflict and high numbers of IDPs; the presence of key

Hard to reach populations Inclusivity/equity Conflict area


67

elements of QBEP and other projects of MOE.

Vulnerable populations Worst off status UNICEF State of interest

Mon Ye Intervention township

Ethnic minority; high numbers of emigrants (internationally) and immigrants (nationally); established alternative education systems of Mon National Education Committee of Mon New State Party in place; within Mon state where TEIP has taken an all-state approach.

Vulnerable populations Inclusivity/equity Alternative education system UNICEF all-state approach pilot Better off status

Mon Mudon Intervention township

Ethnic minority; high numbers of emigrants (internationally) and immigrants (nationally); established alternative education systems of Mon National Education Committee of Mon New State Party in place; within Mon state where TEIP has taken an all-state approach.

Vulnerable populations Inclusivity/equity Alternative education system UNICEF all-state approach pilot Better off status

Pago Htain Ta Bin

Intervention township

Characteristics resemble the central part of Myanmar; remote area; difficult transportation.

Equity/inclusivity

Yangon Hlaing Thar Yar

Intervention township

Within economic hub of Myanmar; peri-urban area with highest magnitude of poverty; highest number of migrants from within Myanmar; the presence of key elements of QBEP and other projects of MOE.

Equity/inclusivity Vulnerable populations Better off status

Shan Lashio Comparison township

(1) Lashio being close to the border of China as opposed to proximity of Tachileik to the border of Thailand; (2) relatively booming economic activities and lucrative border trading in both townships; (3) the presence of significant number of migrants from other parts of Myanmar; (4) influence of Chinese culture in Lashio in comparison to influence of Thai culture in Tachileik, which contributed to preference of residents to learn Chinese or Thai education to Myanmar education; and (5) being centre towns in Shan state that could receive closer supervision and support from higher level of the MOE.

B. Second stage sampling of catchment area Purposive sampling was also employed at the second stage. A criterion sampling method was used to select one catchment area in both the TEIP townships and non-TEIP townships. The catchment area is a geographical area where students from its primary schools will continue to study in its middle schools and students from its middle schools will continue their education in its high schools. The number of schools per one catchment area in six evaluation townships ranged from four to 23. Organising group discussions with students, parents and teachers from one catchment area helped the evaluators to capture education needs and how TEIP could address education needs (or not) at the second most basic unit of one TED.


68

Criteria developed and agreed upon by key stakeholders during the planning phase comprised of difficulty in transportation, remoteness and poverty. Catchment areas of each township were categorised into three strata by these three criteria: (1) relatively well in relation to three criteria; (2) Typical with respect to three criteria; and (3) relatively poor with regard to three criteria. In each of five intervention sites and one comparison site, the typical stratum covers the majority of schools and students of township. One of the catchment areas that falls under the typical stratum was chosen as a sample catchment area. The typical stratum covers schools in both rural and urban settings. To examine the issue of equity, a catchment area located in rural setting was also selected. Students, parents and teachers from as many schools as possible in one catchment area were chosen to discern views and experiences with the TEIP and education needs from schools with heterogeneous degree of experiences and needs. The theory underpinning this sampling method is that if TEIP can be implemented in a typical stratum, it has the potential to reach and benefit a majority of schools and students of each township. In addition, the education needs and capacity needs identified in this catchment area of the typical stratum would likely be similar to those of the broader population in each township. Thus, the needs identified in catchment areas would form the basis for future TEIP programming or other similar decentralised planning modalities. The unit of analysis was the entire township level. Study populations included were students, parents, teachers, TEIP committee and working group members, non-state education service providers such as monastic schools, private schools, non-formal education and NGOs, Township Administrative Department Officials and Township Development Committee members. Continuum of Education was the key issue to examine. The inclusion of non-state actors was in line with TEIP coverage as outlined in the TEIP manual. C. Random sampling for quantitative data collection method The sampling frame for the quantitative survey included all schools within the six intervention and one comparison sites. All schools were stratified by catchment area, which was a collection of primary, middle, and high schools in one geographical location. The required sample size per township was divided by the number of catchment areas of each township to compute a required number of sample schools in each stratum. From a list of all schools within each stratum, sample schools were randomly selected by using SPSS. Since TEIP is a programme intended to improve systems at the township level, the most appropriate respondent type for the survey was a school or the smallest unit of the township education system rather than an individual staff member from the township education system. Moreover, objective responses to questions could be elicited by examining the available data of each of the selected schools with a group of head of a school and relevant teachers of a school.

Annex 4.2 Evaluation team

Name Title Description of role Experience

Dr Naing Team Leader

and

evaluation

methodology

specialist

Study design, development of

guidelines for qualitative methods,

development of a structured

questionnaire, collection of

qualitative data, analysis of

qualitative data, reporting and

presentation of findings

Experience and formal qualification

in evaluation, considerable

experience in evaluation and design

of education programs, well versed

quantitative and qualitative methods,

ability to lead a large scale

evaluation/research studies and


69

knowledge of local context of all

study locations

Dr Ye Myint National

Education

Sector

Specialist

Provision of advice to development

of guidelines for qualitative

methods and a structured

questionnaire, analysis of qualitative

data, reporting and presentation of

findings

Experience and formal qualification

in education sector, knowledge of

local context of all study locations

and some experience of evaluation

methodology. Working relationships

at subnational level with township

education staff. Known to MOE.

Su Thanda Zaw Senior

Evaluator for

Quantitative

Methods

Study design, development of a

structured questionnaire, collection

of quantitative data, analysis of

quantitative data and reporting

Experience in evaluation, some

experience in evaluation of education

programs, well versed in quantitative

and qualitative methods, experience

in assisting the team leader in large

scale evaluation/research studies,

and knowledge of local context of

many of the study locations

Lwin Cho Junior

Evaluator for

Qualitative

Methods

Study design, development of

guidelines for qualitative methods,

collection of qualitative data,

analysis of qualitative data and

reporting

Experience in evaluation, well versed

qualitative methods, experience in

assisting the team leader in large

scale evaluation/research studies,

and knowledge of local context of

many of the study locations

Annex 4.3 Data sources

Type of stakeholders Ways of participating in the study

Rationale for level of participation

TEIP committee members of each of 6 TEIP townships and 1 non-TEIP township

1.PRA Workshop 2. Planning workshop held in the inception phase

TEIP role: Main implementers of TEIP activities. Evaluation questions covered: Relevance, effectiveness, efficiency and sustainability. Specific rationale: Participation of TEOs in the planning workshop was essential for designing appropriate and relevant evaluation study and tools. Assessing their ownership and leadership attributes would address questions of efficiency.

Students, parents and teachers from one catchment area of each of 6 TEIP townships and 1 non-TEIP township

Focus group discussions

TEIP role: Intended beneficiaries of TEIP implementation. Evaluation questions covered: Effectiveness, relevance, sustainability. Specific rationale: To determine the extent to which students and parents benefitted from TEIP. To assess how and to what extent students and parents were involved in the TEIP process and could answer the evaluation questions related to relevance and sustainability.

District Education Officers (DEOs) and State Education Officers (SEOs)

In-depth interviews

TEIP role: Main supervisors of implementation of TEIP activities. Evaluation questions covered: Relevance, effectiveness, efficiency and sustainability. Specific rationale: These stakeholders have knowledge of financial management systems and M&E systems of different levels of the education department.


70

Non-state education service providers such as monastic schools, private schools and NGOs

In-depth interviews

TEIP role: Target beneficiaries of TEIP (which aimed to cover both state and non-state education providers) and intended as planners and implementers of education related activities at the township level. Evaluation questions covered: Relevance and effectiveness. Specific rationale: Examining how TEIP could integrate planning and implementation of education services of different service providers aimed at answering relevance and effectiveness.

Township education foundation

In-depth interviews

TEIP role: These foundations provide resources to students to continue their education in formal schools or alternative forms of education and assist some township education office in acquiring resources. Evaluation questions addressed: Sustainability. Specific rationale: Assessing the amount of resources provided, to whom they provided resources and how they were integrated into TEIP was essential for understanding resource mobilisation and acquisition for sustainability.

Relevant officials from Township General Administrative Department (TGAD), Township Municipality Committee (TMC), Department of Rural Development (DRD) and Township Development Assistant Committee (TDAC)

In-depth interviews

TEIP role: These stakeholders were main supporters of the planning and implementation of TEIP activities. Evaluation questions addressed: Relevance, effectiveness and sustainability. Specific rationale: These stakeholders knew the needs of the education and other sectors and knew of the resources available across various sectors.

TEIP facilitators

1. In-Depth Interviews 2. Planning workshop held in the inception phase to review and revise evaluation questions, criteria and guidelines and to develop a detailed plan for the evaluation.

TEIP role: TEIP facilitators were key implementers and supervisors of TEIP activities through facilitating workshops. Evaluation questions addressed: Relevance, effectiveness, efficiency and sustainability. Specific rationale: To discuss their role, experiences and knowledge of TEIP, as well as how they were supervised and supported. Participation of TEIP facilitators in the planning workshop was essential for designing an appropriate, effective and efficient evaluation study because they were a key primary resource in the absence of project documents.

UNICEF staff 1. In-Depth Interviews 2. Planning workshop held in the inception phase to review and revise evaluation questions, criteria and guidelines and to develop a detailed plan for the evaluation.

TEIP role: UNICEF officials were supporters and supervisors of TEIP activities. Evaluation questions addressed: Relevance, effectiveness, efficiency and sustainability. Specific rationale: Examining how they were involved in the planning, implementation and M&E of TEIP activities would help the evaluation to identify Strengths, Weaknesses, Opportunities and Limitations of the TEIP project. Participation in the planning workshop was essential for designing the evaluation study in the absence of project documents and data.

Reference Group: Members of the MOE; Donor partners;

Participating in the planning workshop held in the inception phase to review and

TEIP role: Members were authorised persons of the MOE to support the accomplishment of the evaluation. Specific rationale: Their participation in the planning workshop was essential for designing the evaluation study as


71

UNICEF Evaluation Manager

revise evaluation questions, criteria and guidelines and to develop a detailed plan for the evaluation.

they facilitated approvals and delivery of key messages to higher level decision makers regarding the implementation of the evaluation and results of the evaluation.

Exploratory qualitative phase A. Document review The following documents were reviewed: NESR; one version of the TEIP manual available; QBEP midterm review and QBEP reports; and TEIPs of six townships included in the evaluation, five more townships of Shan state and five more townships of Mon state. Other education planning programmes designed and implemented in other countries were studied to draw lessons learned to compare with the findings of the TEIP programme. This approach enhanced the quality of the study as comparison could be made with the non-TEIP programmes and the programmes in settings other than Myanmar. B. Focus group discussions (FGDs), in-depth interviews (IDIs), key informant interviews (KIIs) and participatory reflective appraisal (PRA) workshops The evaluation team developed, reviewed and revised qualitative tools with involvement of UNICEF, the Reference Group, and TEOs during the inception phase and in accord with intended TEIP objectives outlined in the TEIP manual and other project documents identified in the course of the inception report development. On 8 September 2015, the evaluation team met with 13 officials from the DBE of the MOE in Nay Pyi Daw to present the study design and seek their inputs. Following that, a one-day planning meeting with a three-member Reference Group and relevant UNICEF officials was held in Nay Pyi Daw on 10 September 2015. During this meeting, participants reviewed and agreed upon:

An evaluation study design;

Key evaluation questions;

Data collection methods;

A first stage sampling strategy of six intervention townships and one comparison township,

A second stage sampling strategy that determined criteria for selecting catchment areas of each of the sampled townships;

Key stakeholders to be included as study’s participants; and

Key issues that should be included in the data collection instruments. In addition, a two-day planning workshop was organised in Yangon on the 14th and 15th of September 2015, in which one TEO from each of the six selected townships, two Deputy TEOs, and two members of the Reference Group. During the workshop, participants reviewed and finalised the elements outlined in bullets above, as well as agreed upon a detailed data collection plan for each location and the necessary logistical arrangements for data collection. These meetings and workshop also contributed to the capacity building of MOE officials by developing their understanding on how to manage an evaluation and how to utilise evaluation findings. In addition, the involvement of the MOE officials at the start of the evaluation strongly increased their sense of ownership over the evaluation and built trust and a working relationship between the evaluation team and key stakeholders. Data collection was undertaken in one intervention site, namely Hlaing Thar Yar Township in the Yangon Region, in this stage. Ways of constructing key terminologies, wording and phrasing of the structured questionnaire was also examined. This enabled the study team to develop a questionnaire with appropriate and accessible tone, register, and language. Languages that need to be used in interviews, operational challenges, recruitment of interviewers, and logistic issues were investigated and planned in this phase. Consultative meetings were organised throughout this phase to prepare for the quantitative phase and to understand the central issues of the evaluation.


72

An evaluation team consisting of a team leader, mid-level quantitative and qualitative evaluators, and a national education specialist facilitated FGDs, IDIs, KIIs and workshops. Before any interview, FGD, or workshop took place, a plain language statement was read to respondents and their informed consent was sought beforehand. Confidentiality was assured and respondents were able to opt out of participating at any time. Guidelines were used to facilitate FGDs, IDIs, KIIs and workshops by a team leader to ensure consistency, continuity and quality while allowing flexibility, progressive probing, and greater linkages of analysis across FGDs, IDIs, KIIs and workshops. In order to enhance verification and prevent bias as much as possible, a minimum of two evaluation team members were present during FGDs/IDIs. Having a gender balance amongst facilitators and an appropriate mix of evaluation and education specialists among the data collection team contributed to the effective facilitation of qualitative work.


73

Number of FGDs, Group Interviews (GIs), IDIs, PRA, school survey and TED survey and participants across seven locations

Type of participants Hlaing Thar Yar Htan Ta Bin Myebon Tachileik Yae Mudon Lashio Students of 5th, 9th, 10th and 11th grade from one catchment area

1 FGD and 16 participants







Parents 1 FGD and 14 participants







Teachers from schools belonged to one catchment area


1 FGD and16 participants





1 FGD and 18participants

TEIP committee members 1 PRA workshop and 12 participants

1 PRA workshop and 10 participants






Township Administrative Officers 1 IDI and 1 participant

1 GI and 5 participants


1 IDI and 1 participant




Municipality Committee, Township Development Assistant Committee, Department of Rural Development


Included in above GI DRD was included in above GI

2 IDI and 2 participants, 1 GI and 4 participants

2 IDIs and 2 participants

Included in above GI


Monastic schools 1 GI and 2 participants





Private schools 1 GI and 3 participants and observation





NGO and Community Based Organisations





DEO 1 GI and 5 participants



1 GI and 1 participant

1 GI and 1 participant

Included in GI of Yae


SEO 1 IDI and 1 participant


I GI and 2 participants





UNICEF staff members 1 GI and 2 participants



TEIP facilitators 1 IDI and 1 participant of SED of Yangon, 1 IDI and 1 participant of SED of Mawlamyine, 1 IDI and 1 participant of DBE

Survey of schools 45 schools 45 schools 40 schools 45 schools 45 schools 42 schools 105 schools

Survey of TEDs Muse, Namtu, Monghpyak, Tachileik, Nansang, Pinlaung, Myebon, Hlaing Thar Yar, Pyapon, Bilin, Thayetchaung, Mudon, Kyaikmaraw, Mawlamyine, Hpa-An, Chaungzon, Thanbyuzayat


74

Quantitative surveys During the inception phase, a self-administered questionnaire was developed by the evaluation team and shared with key stakeholders as described above. Feedback from key stakeholders was incorporated into a revised second draft version. Findings of FGDs and IDIs from the first sampled intervention township in Hlaing Thar Yar were used to refine a third draft, which was pre-tested. Findings of the pre-test were used to finalise a structured questionnaire. The self-administered questionnaire covered the following issues: (1) Factors related to the improvement of access to and quality of education at the township level; (2) Resources and capacity needs of townships regarding TEIP development, implementation, and M&E; (3) Awareness of the TEIP process; (4) Opinions on the relevance of a decentralised township level plan; (5) Self-assessment comparing the extent to which education needs and key indicators of the township were addressed in the two-year period prior to the TEIP and the two-year TEIP implementation period; (6) Self-assessment comparing the extent to which the capacity needs of the township were addressed between the two-year period prior to the TEIP and two-year TEIP implementation period; and (7) Self-assessment on the likelihood of the sustainability of the TEIP programme. The rationales for using a self-administered questionnaire were that:

This method was relatively less costly;

Education level of the intended respondents was likely to warrant the responses with satisfied level of quality and validity to certain extent; and

Responses could be elicited within the relatively shorter time frame. The parameters cited below resulted in a proposed sample size of 45 schools per township for TEIP townships and a proposed sample size of 105 for one non-TEIP township – a total of 330 schools.

Significance level of 5 per cent;

Study power of 80 per cent;

Outcome level target, such as effectiveness to address township level education needs at the time of the evaluation, of 50 per cent and the target set up to achieve at the end of the second phase of the programme at 80 per cent;

Outcome level target, such as effectiveness to address township level education needs at the time of the evaluation for the TEIP township, of 50 per cent and effectiveness to address township level education needs at the time of the evaluation of the non-TEIP township at 30 per cent; and

Estimated participation rate of 90 per cent. The questionnaire was translated from English into Myanmar, and then back-translated to English to ensure accuracy of questions. The evaluation team arranged a three-hour meeting at each of six sampled townships to explain the content and how to answer the questionnaire to head teachers of randomly selected schools. A plain language statement was provided to respondents and informed consent was sought before each participant responded to the self-administered questionnaire. The questionnaire was constructed so as to be self-explanatory. Heads of selected schools worked with relevant teachers in their respective schools to answer the self-administered questionnaire. Completed questionnaires were dispatched to their respective TEOs in a closed envelope and TEOs returned all completed questionnaires to the evaluation team. Explanatory qualitative phase After analysing quantitative data, the team conducted FGDs, IDIs and KIIs with key stakeholders of two TEIP townships, Tachileik and Yae, and one non-TEIP township, Lashio, to draw out explanations and elaborations for the quantitative data analysis and to formulate recommendations for future programme design and strategies. The team collected qualitative data in two TEIP townships, Htain Ta Bin and Myebon, alongside the quantitative data collection.


75

Annex 4.4 Special attention of evaluation

Investigation of alternative modalities The evaluation team identified alternative modalities to TEIP through consultative meetings with key stakeholders. School Self-Assessment (SSA) and School Improvement Plan (SIA) were discovered to be means of examining the needs of individual schools and of developing a plan to improve access to and quality of education through the involvement of stakeholders such as parents, Township Development Assistant Committees, and Township, Village, or Ward General Administrative Officials. The school grants initiative implemented with funding from the World Bank and the MOE was identified as supporting decentralised planning and implementation. These modalities were further examined along the course of evaluation. Investigation of the whole state approach Consultative meetings during the inception phase suggested that the whole state approach being applied in Mon state might present different inputs, processes and results from an approach that focussed primarily on few selected townships with higher degree of needs or disadvantaged townships. The evaluation added Mudon township of Mon State to Yae township to conduct a ‘sub-programme analysis’ whereby the two different approaches within the overall programme being evaluated were compared and contrasted. Reconstruction of data regarding baseline status, input and process of TEIP

Consultative meetings with key stakeholders revealed that proper documentation of the TEIP at the township and national levels did not take place, and as a result, it would not be possible to adequately examine how TEIP was implemented. The evaluation addressed this shortage of documentation, and resultant insufficient data, through alternative means and through reconstructing baselines. The evaluation team requested that all TEIP townships fill in a survey form, which helped the evaluation team ascertain what resources were used in terms of human resources, capacity, ownership, accountability, materials, money and time, in order to implement the TEIP programme in each township. Understanding which resources were needed for each TEIP Township would help inform MDEF, MOE and UNICEF decision-making for future programming. Obtaining information on the process from the 25 townships that completed the development of TEIP also offset the limitations posed by the purposive sampling of only six townships. Similarly, a lack of baseline data prevented the evaluation team from determining the extent to which access to and quality of education had improved since the baseline to the end of the TEIP programme. The evaluation team attempted to alleviate this weakness by asking participating schools and townships to compare the situation of access to and quality of education two years prior to the inception of the TEIP programme and during the two-year of TEIP programme.

Annex 4.5 Qualitative data management and analysis

Qualitative data analysis was conducted in three phases as below:

Preliminary analysis

Lead facilitator and note takers each recorded spontaneous impressions during the interview and catalogued emergent trends and themes.

Secondary analysis

Lead facilitator and note takers met for a debrief session after each interview to discuss impressions, recall points of interest, compare records of themes and trends. A written summary of each interview was required at the end of each day, or second day.


76

Final analysis Evaluators re-read field notes and familiarised themselves with the content and to gain an impression of overall themes. FGD and IDI guides and recall of emergent themes were used to develop codes. Evaluators used ATLAS-ti Version 5 to put codes, sort categories according to codes, define codes, and produce quotes of each category. The total number of codes did not exceed 15. Evaluators read each category thoroughly and wrote a detailed description (thick description). These descriptions were triangulated across FGDs or IDIs. Sub-categories were developed, and links between these sub-categories were identified. The team also reviewed own field notes to compare the first and the second phase analyses performed during and after FGDs and IDIs with the interpretation at this stage. Evaluators revisited original FGDs and IDIs transcripts to contextualise findings, through which interpretation were made. Evaluators described and explained findings together with quantitative findings. Evaluators translated direct quotes into English to illuminate findings and interpretations.


77

Annex 5: Additional findings

Annex 5.1 Relevance

1. As part of the TEIP development, TEIP committee members of six townships received capacity building regarding a needs assessment and development of TEIP. TEIP committee members, TEIP workshop facilitators and UNICEF officials thought that some TEIP committee members improved their capacity in the following issues: (1.1) Analysing trend of key indicators of access to and quality of education set forth in the TEIP guidelines; (1.2) Collecting and analysing data with regard to a township profile with particular emphasis on socio-economic situation; (1.3) Realising importance of having valid and accurate data to help them develop a proper plan; (1.4) Skills in planning through using data analysis such as setting up strategic objectives, drawing plans and calculating budget; (1.5) Awareness that having a proper plan would help the TEIP committee to present a plan when there is an opportunity to access to resources – previously they drew plan only when resources were available; (1.6) Acquiring a practice of developing a township education improvement plan through a group of TEOs and some head teachers. Previously, TEO was solely responsible for township level planning. Many TEOs were transferring in and out frequently and as a result, they did not have adequate time to understand the situation of the township well. A township education plan drawn by a TEO was unlikely to reflect the township needs and situation precisely. Now that the team analysed the needs of the township and developed a plan, the responsibility shifted away from one person to a group of people, who knew the needs and situation better than a TEO. Adoption of a new practice appeared to make analysis and planning more comprehensive and offset the weakness of planning by one official; (1.7) Some head-teacher TEIP-committee members reported that although they previously had concern on and knew about the needs and gaps of their own schools, they realised the needs and gaps of the education sector at the township level after being involved in the process; (1.8) Although head-teacher TEIP-committee members previously did not realise the importance of having valid and precise data adequately, they learned the importance of having valid and precise data and they started to gather data systematically; (1.9) Gaining ability to discern SWOL of education sector and to formulate solutions for own township and other townships; (1.10) Building better relationship with other stakeholders such as Township General Administrative Department, Township Health Department, Maternal and Child Welfare Association and Women Affair Association through data collection process; (1.11) Having desire to become a leader for township education development as opposed to a being a follower previously; (1.12) Realising importance and need of undertaking situation analysis before developing a plan; (1.13) Acquiring skills in planning of TEIP helped TEIP committee members develop school improvement plan for school grant project; (1.14) The committee knew resources needed for enhancing township level education sector; (1.15) One school head teacher applied the knowledge and skills acquired from the TEIP process to develop a school improvement plan. She used a school improvement plan to apply her school to become a member of an initiative of South East Asia Ministries of Education Offices (SEAMEO). Her school was selected to learn and share with other participating member schools of the initiative; (1.16) One Township Development Assistant Committee (TDAC) participated in a second workshop and ensuing development of TEIP from the 6th batch onwards. Involvement of a member of TDAC helped them understand needs and plans of TED.


78

Annex 5.2 Factors that could influence improvement or deterioration of the education section

The following discussions were prepared based on the analyses of qualitative data obtained from FGDs with students, parents, teachers, PRA of TEIP committee members, IDIs with DEOs and SEOs, GIs with relevant officials of TGAD, TMC, DRD and TDAC, IDIs with monastic schools, IDIs with private schools of six TEIP townships and similar study participants of one non-TEIP townships, survey of schools and survey of TEDs: 1. Transportation and rough terrain: Difficulty in transportation, terrain and disasters did not pose significant barrier to access to and quality of education in Hlaing Thar Yar, Htan Ta Bin, Myebon, Tachileik, Yae and Mudon Townships. However, traffic jams posed a significant barrier for teachers who resided in other townships to focus on their work to certain extent in Hlaing Thar Yar Township. Of six townships, Myebon Township has more than 170 islands and the only transportation mode is water way. Public passenger boats do not operate on a daily basis. These circumstances pose a considerable degree of difficulty to travel between school and home. Some village tracts such as Loidaw Kham and Kyaing Lap branch township in Tachileik township were relatively remote, and had more difficulties in transportation. Khaw Zar and Kyaing Ywa village tracts in Yae township were relatively remote, and had more difficulties in transportation. Upgrading schools and establishing new schools by the MOE provided most of the villages in these townships access to formal schooling. Difficulty in transportation and terrain did not create an impediment to access to formal education. 2. Examination system: An examination system introduced approximately ten years ago was intended to assess performance of students continuously by organising a Chapter End Test (CET). CET was designated to assess performance of students at the end of each chapter. It was anticipated that organising a CET regularly would encourage students to learn every lesson of each chapter thoroughly and could avoid rote memorisation of lessons. It was also anticipated that by having a regular assessment, students could avoid studying all lessons at certain points of the academic year when examinations were held. Teachers could not implement the continuous assessment system in accord with the concept and principles of the system. CET turned out to be an examination that was held every two to three months, and there were five CETs, a first semester examination and a second semester examination in one academic year. Students continued to study lessons by rote memorisation because teachers continued to prepare examination questions and make corrections that encouraged rote memorisation. The continuous assessment system also required that all students completed a grade at the end of academic year to minimise dropout rates with resultant waste of resources. Students who did not perform well were required to receive remedial teaching, in which teachers revisited previous lessons. After undergoing remedial teaching, students would complete a grade. However, teachers could not spare adequate time to coach students who could not digest lessons adequately to become well versed on these lessons as this system required. The reasons were heavy workload of teachers and a low teacher and student ratio. Unintended results of this system were that students did not revisit the previous lessons as they continued to learn new lessons and to prepare for subsequent examinations. In addition, students and parents who did not have complete understanding of this system perceived that they did not have to work hard because all students would complete a grade regardless of their performance. This misunderstanding discouraged students to learn lessons sufficiently. Consequently, many students completed a grade in spite of not digesting lessons of one grade well. When these students reached higher grades such as 9th, 10th and 11th grades, they encountered profound difficulties in learning lessons, failed examinations and many dropped out from schools.


79

3. Teacher student ratio: Teacher Student Ratio was low for some schools and some grades, particularly schools located in rural settings and State Primary Schools. Teachers were outnumbered by students, grades and classes could not pay adequate attention to quality of teaching. For example, two to three teachers had to manage five grades with one grade having 50 to 80 students in some primary schools. In addition, teachers had to complete the target set forth in a curriculum in one academic year. Some students could not follow lessons well at the pace that teachers needed to complete a curriculum. Teachers who had to handle 50 to 80 students per class could not revisit the lessons for these students who could not understand lessons well. Limited time availability of teachers also restricted teachers from correcting assessments of and/or led to incorrect assessment of assignments of students. A low teacher student ratio undermined the quality of teaching to certain extent. One mother revealed eloquently that “When I checked how the teacher gave corrections for the answers of my daughter, I found that some corrections were wrong. Then I told my daughter about these mistakes. My daughter did not accept my advice because she had more trust on her teacher than me. These kinds of mistakes created the circumstance where children mistakenly perceived the wrong answers as the right ones for many years”. 4. Quantity versus quality: In order to increase access to school, the MOE opened up new schools and upgraded the existing schools from primary to middle and from middle to high schools. This measure significantly increased access to schools for many students who would have otherwise dropped out from schools, particularly at a transition from primary to middle and from middle to high grades, when students had to go to schools in other villages or towns. Conversely, establishing new schools and upgrading the existing schools necessitated investing enormous resources for employing additional teachers, constructing new buildings, providing operation costs, etc. In spite of investing more resources by the MOE in the previous four years, enormous needs outweighed resources available, thereby leaving many needs unmet. Producing qualified teachers from various teacher colleges could not meet the number of new teachers needed, resulting in a low teacher student ratio. The MOE attempted to address this problem by employing teachers who were not graduates of teacher colleges. Although, these newly appointed teachers were trained in the service by projects such as School-based In-service Teacher Education Programme (SITE), quality of teaching of newly appointed teachers were assessed as insufficient. Increasing access to schools in the presence of limited, insufficient resources stretched the quality of education to certain extent at least for some years to come. 5. Utility of education for real life: Although completion rates and transition rates for primary, middle and high grades were improved in the recent years, many students did not complete the matriculation examination. Dropout rate from school was low during the 1st to 5th grade but started to rise when students transited from the 5th to the 6th grade. Afterwards, the higher the grade, the higher the dropout rate. The reasons for dropout were: (1) increase in cost of education, particularly cost incurred for tuition, as students reached higher grades; (2) perceived opportunity cost by parents – decisional balance or weighing up between cost of education and loss of income from jobs taken up by students or need to use students for household chores such as babysitting or need to use students for family business such as orchids and rubber plantations; (3) older children had difficulties in resisting their desire to get income that they could use to buy trendy materials such as cellular phone, apparel, etc.; (4) seasonal or permanent migration of parents to other areas domestically or internationally; (5) poverty level of parents forced students to work; (6) students who did not acquire adequate knowledge and skills on lower grades faced huge challenges in picking up lessons as the volume and complexity of lessons increased along the course of learning in higher grades, which made them depressed and reduced their interest in learning; and (7) some students were persuaded by


80

peers and environment to use game, internet, smoking and betel chewing, which collectively constituted an inhibitory environment for their study. Formal school education could not provide students who left the school before they reached the university education appropriate livelihood skills, life skills and general knowledge that could be used for their survival in real life. Many students and parents perceived that investment made for their children’s education did not pay off reasonable amount of economic and social returns or dividends. The amount of income earned from employment was more or less the same regardless of education status because students did not acquire essential skills for employment from formal schools. Thus, many parents and students did not perceive high value on formal schooling. 6. Inadequate use of teaching aid: A majority of teachers participating in FGDs disclosed that they had to create teaching aids by themselves. Heavy workload and unfavourable teacher student ratio left inadequate time for teachers to prepare teaching aids. A majority of students participating in FGDs also revealed that they had to learn lessons solely from the text book because of insufficient use of teaching aids and practicum. Science laboratories were not equipped with essential materials at the sufficient amount. Students could not undertake experiments adequately, as a result. Consequently, many students could not comprehend many lessons. 7. Number of projects and routine and ad hoc pieces of work versus quality of teaching: Many teachers and head teachers participating in FGDs disclosed that they had to participate in some projects of the MOE and other donors and some assignments to assist other government departments. These tasks created additional burden for already outstretched teachers and head teachers, particularly teachers and head teachers from those schools in rural setting and under-staffed primary schools. This circumstance took some time away from their primary teaching task, thereby negatively affecting quality of teaching. 8. Application of knowledge gained from various trainings: Many capacity building training/workshops were provided to teachers from seven townships. These training/workshops encompassed a range of different issues such as Child Centre Approach, Child Friendly School approach, Language Enrichment Programme, instructional leadership and management skills, etc. Many of teachers and head teachers who received training on these issues did not utilise knowledge gained to the full extent in their routine teaching and management tasks because of numerous tasks of teachers and a low teacher student ratio. In addition, insufficient availability of teaching aids and materials prohibited teachers from applying these approaches fully. A document review also indicated an organizational culture that had influence on behaviour change of teachers to adopt new teaching practices. The National EFA Review Report of 2014 revealed that teachers tended to teach how they had been taught at school. Since many teachers in Myanmar were taught at school and at the teacher education institutions through using traditional teacher-centred teaching. At the workplace, teachers trained in CCA were influenced by senior teachers and colleagues, most of whom applied their familiar ways of teaching. It is likely that school culture imbued with traditional views, beliefs and practices was conducive to the use of chalk and talk methods and rote learning (National EFA Review Report, 2014). 9. Tuition, cost implication and performance of students: Almost all teachers, parents, students and other study participants unanimously disclosed that a majority of students took tuition from either school teachers or private teachers. The amount of money that each student had to spend varied significantly depending on teachers, number of students involving in a tuition class and number of subjects that each student studied. While cost of tuition did not pose significant burden on the purse of affordable parents, it did create a cumbersome drain on poor parents. Many


81

parents and students did not consider that they actually needed additional tuition and followed a norm of other parents and students. Many students had to study in schools, tuitions and guidance from early in the morning to late evening or night, which deprived these students of time for family, sport, leisure, etc. Although these students took tuition, it did not appear that they were well versed with lessons. It was questionable that time and money spent for tuition was worth considering results and performance of students. 10. Additional teaching hours offered by schools versus tuition: Some schools in seven townships offered additional teaching to students. These extra teaching hours were designated specifically for students of the 5th, 9th and 11h grades because of stringent examinations organised by district/state/region/national levels. Schools in rural settings organised these additional teaching better than schools in urban settings. They offered more hours of additional teaching and guided study. In addition, teachers were more enthusiastic in teaching, many offered these services without asking money and students appeared to put more value on these extra teaching. These promising practices were assessed to be alternative ways for tuition because they offered the same degree of additional teaching and coaching with less costs. 11. Low salary and benefits of teachers: Most of the teachers, head teachers and TEIP committee members disclosed that a low salary and few benefits was one factor that led to the practice of giving tuition to students by some teachers and this factor also contributed to inadequate interest and commitment of some teachers. A review of the National EFA Review Report of 2014 corroborated this finding. The report revealed that the salary, increased recently, is still low, when compared to the cost of living, coupled with very few fringe benefits received by teachers meant there were few incentives for them to develop themselves or to take on challenging tasks.

Factors related to improvement or deterioration of education sector in TEIP townships and non-TEIP townships

None Few Moderate

High

% % % % Primary School Students

Long distance between home and school TEIP Townships (n=249) 69.1 24.1 6.4 0.4

Non-TEIP Township (n=100) 35.0 46.0 17.0 2.0

TED survey (n=17) 17.6 41.2 41.2 -

Lack of bicycle/motor cycle/bus/boat/ferry to go school or having necessary to go to school on foot

TEIP Townships (n=249) 68.3 18.5 6.0 7.2

Non-TEIP Township (n= 100) 40.0 28.0 17.0 15.0

TED survey (n=17) 5.9 41.2 52.9 -

Having necessary to go through stream/river

TEIP Townships (n=251) 86.5 12.0 0.8 0.8


TED survey (n=17) 29.4 58.8 11.8 -

Could not afford to pay transportation cost

TEIP Townships (n=250) 95.2 3.6 1.2 -

Non-TEIP Township (n=100) 88.0 8.0 4.0 -

TED survey (n=17) 58.8 35.3 5.9 -

Could not afford to buy text books and stationary

TEIP Townships (n=252) 54.4 36.1 7.5 2.0


TED survey (n=17) 47.1 52.9 - -

Flood/ landslide/arm conflict TEIP Townships (n=253) 83.8 14.2 1.6 0.4


TED survey (n=17) 35.3 64.7 - -

Migration of parents TEIP Townships (n=253) 17.8 67.2 10.7 4.0


TED survey (n=17) - 58.8 29.4 11.8

TEIP Townships (n=252) 13.5 51.6 25.0 9.9


82

Parents do not put adequate value on education


TED survey (n=17) 5.9 82.4 11.8 -

Divorce/remarriage/death of parents TEIP Townships (n=253) 16.6 68.8 9.9 4.7


TED survey (n=17) - 88.2 11.8 -

Parents only want to get completion of grade but do not care competency of students

TEIP Townships (n=253) 15.0 35.2 29.2 20.6


TED survey (n=17) 11.8 47.1 41.2 -

Students have to look after younger siblings because their parents have to work or there are no parents

TEIP Townships (n=252 19.8 67.5 9.1 3.6

Non-TEIP Township (n=100 29.0 60.0 9.0 2.0

TED survey (n=16) 6.3 81.3 12.5 -

Children could not study lessons well TEIP Townships (n=253 9.5 68.0 18.2 4.3

Non-TEIP Township (n=100 13.0 74.0 11.0 2.0

TED survey (n=16) - 81.3 18.8 -

School is located in other ward/village TEIP Townships (n=249) 80.3 17.3 2.0 0.4


TED survey (n=17) 35.3 52.9 11.8 -

Students are lured by human traffickers TEIP Townships (n=253) 98.0 1.6 0.4 -

Non-TEIP Township (n= 101) 97.0 2.0 1.0 -

TED survey (n=17) 58.8 41.2 - -

Students are beaten/robbed/sexually assaulted

TEIP Townships (n=254) 97.6 2.4 0.0 -


TED survey (n=17) 58..8 41.2 - -

Students encounter road traffic injury TEIP Townships (n=253) 88.5 11.1 0.4 -


TED survey (n=17) 29.4 64.7 94.1 -

Presence of animals with toxin such as snakes in the school premises

TEIP Townships (n=252) 70.6 27.0 1.6 0.8


TED survey (n=17) 52.9 41.2 5.9 -

Breeding of mosquitoes in the school premises result in mosquito borne diseases such as dengue haemorrhage fever

TEIP Townships (n=253) 62.8 33.6 2.4 1.2


TED survey (n=17) 47.1 35.3 17.6 -

Insanitary latrines and drinking water sources result in water borne diseases such as diarrhoea

TEIP Townships (n=254) 77.6 18.9 2.8 0.8


TED survey (n=17) 41.2 58.8 - -

Middle School Students



TED survey (n=17) 5.9 70.6 23.5


TEIP Townships (n=125) 88.8 10.4 0.8 -


TED survey (n=17) 52.9 41.2 5.9


TEIP Townships (n=126) 50.0 40.5 7.9 1.6


TED survey (n=17) 47.1 52.9

Could not afford to buy uniforms/school bags/umbrella/rain coat/sandals

TEIP Townships (n=126) 39.7 48.4 10.3 1.6


TED survey (n=16) 43.8 43.8 12.5

Do not have food to eat before coming to school and do not have food for lunch box or do not have money to buy lunch

TEIP Townships (n=126) 46.8 44.4 5.6 3.2


TED survey (n=16) 37.5 56.3 6.3

Could not afford to pay tuition fees TEIP Townships (n=126) 38.9 32.5 13.5 15.1



83

TED survey (n=15) 13.3 60 20 6.7



TED survey (n=17) 5.9 58.8 35.3


TEIP Townships (n=126) 16.7 51.6 22.2 9.5


TED survey (n=17) 11.8 70.6 17.6



TED survey (n=16) 6.3 68.8 25


TEIP Townships (n=126) 15.1 38.9 26.2 19.8


TED survey (n=17) 5.9 76.5 17.6


TEIP Townships (n=126) 27.8 60.3 7.9 4.0


TED survey (n=16) 12.5 81.3 6.3

Parents ask their children to work to earn income for family

TEIP Townships (n=121) 38.0 52.1 9.1 0.8


TED survey (n=16) 18.8 62.5 18.8

Middle graders could not follow schools’ lessons because they did not digest lessons in primary grades

TEIP Townships (n=125) 12.0 53.6 24.0 10.4


TED survey (n=17) 5.9 70.6 23.5

Students adopt practices such as using computer games, internet and cellular phones, gambling, snooker, smoking, betel nut chewing and drinking because of peer pressure

TEIP Townships (n=126) 41.3 49.2 6.3 3.2


TED survey (n=17) 17.6 64.7 17.6

School is located in other ward/village TEIP Townships (n=125) 72.0 21.6 6.4 0.0


TED survey (n=17) 17.6 70.6 11.8

Students themselves want to earn money for own use and thus leave schools to work

TEIP Townships (n=126) 54.0 42.9 2.4 0.8


TED survey (n=17) 41.2 58.8

Do not attend school regularly because of peer pressure or own desire

TEIP Townships (n=123) 37.4 56.1 6.5 -


TED survey (n=17) 17.6 82.4

Students are lured by human traffickers TEIP Townships (n=126) 95.2 4.8 - -

Non-TEIP Township (n=55) 96.4 3.6 - -

TED survey (n=17) 82.4 17,6


TEIP Townships (n=125) 96.0 4.0 - -


TED survey (n=17) 64.7 35.3

Students encounter road traffic injury TEIP Townships (n=120) 81.7 18.3 - -


TED survey (n=17) 29.4 64.7 5.9

High School Students



TED survey (n=16) 18.8 56.3 25




TED survey (n=16) 50 37.5 12.5


TEIP Townships (n=45) 55.6 37.8 4.4 2.2



84

TED survey (n=17) 41.2 58.8


TEIP Townships (n=45) 42.2 51.1 4.4 2.2


TED survey (n=16) 50.0 43.8 6.3


TEIP Townships (n=45) 51.1 37.8 8.9 2.2


TED survey (n=16) 31.3 68.8

Could not afford to pay tuition fees TEIP Townships (n=45) 40.0 31.1 17.8 11.1


TED survey (n=16) 12.5 43.8 37.5 6.3

Could not afford to pay costs collected by school

TEIP Townships (n=45) 46.7 48.9 2.2 2.2


TED survey (n=15) 40 46.7 13.3



TED survey (n=17) 76.5 23.5


TEIP Townships (n=45) 20.0 51.1 24.4 4.4


TED survey (n=179) 5.9 82.4 11.8



TED survey (n=16) 81.3 18.8


TEIP Townships (n=45) 13.3 35.6 35.6 15.6


TED survey (n=16) 12.5 87.5 0


TEIP Townships (n=45 22.2 66.7 8.9 22.2


TED survey (n=16) 18.8 75 6.3


TEIP Townships (n=44) 34.1 52.3 13.6 -


TED survey (n=17) 29.4 52.9 17.6

High graders could not follow school’s lessons because they did not digest lessons in primary and middle grades

TEIP Townships (n=45) 2.2 44.4 28.9 24.4


TED survey (n=17) 11.8 58.8 29.4


TEIP Townships (n=45) 26.7 53.3 17.8 2.2


TED survey (n=17) 11.8 70.6 17.6

School is located in other ward/village TEIP Townships (n=45) 51.1 35.6 13.3 -


TED survey (n=17) 17.6 64.7 17.6


TEIP Townships (n=45) 46.7 48.9 4.4 -


TED survey (n=16) 31.3 62.5 6.3


TEIP Townships (n=45) 15.6 75.6 8.9 -


TED survey (n=16) 6.3 81.3 12.5

Students are lured by human traffickers TEIP Townships (n=45) 91.1 8.9 - -


TED survey (n=17) 64.7 35.3




TED survey (n=17) 64.7 35.3

Students encounter road traffic injury TEIP Townships (n=45) 80.0 20.0 - -


85

Non-TEIP Township (n= 15) 60.0 40.0 - -

TED survey (n=17) 29.4 64.7 5.9

Teachers

Number of student outweigh number of teachers

TEIP Townships (n=239) 45.6 31.0 11.7 11.7


TED survey (n=17) 64.7 29.4 5.9

Some teachers have to teach more than one subject, and as a result, they do not have adequate skills

TEIP Townships (n=242) 43.8 33.9 12.0 10.3


TED survey (n=17) 5.9 64.7 29.4

Some teachers do not have adequate commitment to their work

TEIP Townships (n=241) 61.8 32.8 4.1 1.2


TED survey (n=17) 29.4 47.1 17.6 5.9

Some teachers use internet, computer game and cellular phone during teaching hours

TEIP Townships (n=243) 78.2 19.8 2.1 -


TED survey (n=17) 29.4 52.9 17.6

Some teachers do tuition that result in giving favours to students who take tuition from them or students fell like that

TEIP Townships (n=242) 84.3 13.2 1.7 0.8

Non-TEIP Township (n=99) 88.9 10.1 0.0 1

TED survey (n=16) 31.3 56.3 12.5

Some teachers only want to completion of students but do not care competency of students adequately

TEIP Townships (n=243) 64.6 26.7 6.2 2.5


TED survey (n=17) 29.4 58.8 11.8

Some teachers do not have adequate accountability for maintenance of furniture, school building, etc.

TEIP Townships (n=244) 52.9 36.5 9.4 1.2


TED survey (n=17) 35.3 52.9 5.9 5.9

Students did not revisit the previous lessons after each chapter end test that consequently undermined quality of education

TEIP Townships (n=244) 35.7 31.6 15.2 17.6


TED survey (n=17) 17.6 29.4 17.6 35.3

Students had lower value on education because all students passed examinations of all grades except 5th, 9th and 11th grades regardless of their competency

TEIP Townships (n=244) 15.6 30.3 23.0 31.1


TED survey (n=17) 11.8 29.4 29.4 29.4

Your school have many projects that result in cumbersome work load and drain on time of teachers and head teachers

TEIP Townships (n=244) 21.7 38.1 20.9 19.3


TED survey (n=17) 17.6 47.1 17.6 17.6

Teachers have to take non-teaching tasks that undermine their teaching tasks

TEIP Townships (n=245) 31.4 42.0 13.9 12.7


TED survey (n=17) 23.5 41.2 29.4 5.9

You school do not have adequate non-teaching staff

TEIP Townships (n=241) 40.2 30.7 12.0 17.0


TED survey (n=17) 17.6 41.2 29.4 11.8

Assessment of quality of teaching is insufficient

TEIP Townships (n=242) 42.1 37.2 16.1 4.5


TED survey (n=16) 18.8 43.8 37.5

Teachers who receive training do not apply their knowledge into their teaching adequately

TEIP Townships (n=236) 18.6 51.7 24.6 5.1


TED survey (n=17) 5.9 47.1 29.4 17.6

Your school does not have sufficient building

TEIP Townships (n=236) 18.6 51.7 24.6 5.1


TED survey (n=17) 5.9 52.9 35.3 5.9

Your school does not have sufficient furniture

TEIP Townships (n=244) 44.3 30.7 17.6 7.4


TED survey (n=17) 11.8 58.8 23.5 5.9

TEIP Townships (n=245) 88.6 8.6 1.2 1.6


86

Your school does not have sufficient text books


TED survey (n=17) 76.5 23.5 5.3

Your school does not have sufficient teaching aid

TEIP Townships (n=240) 21.7 50.4 18.3 9.6


TED survey (n=17) 11.8 41.2 35.3 11.8

Your school does not have sufficient latrine and dirking water or sufficient sanitation of them

TEIP Townships (n=246) 55.7 28.9 8.9 6.5


Your school does not have sufficient primary school teachers with accreditation

TEIP Townships (n=238) 45.0 32.8 11.3 10.9


Your school does not have sufficient middle school teachers with accreditation

TEIP Townships (n=181) 63.5 21.5 11.6 3.3


Your school does not have sufficient high school teachers with accreditation

TEIP Townships (n=97) 79.4 10.3 4.1 6.2


Your township does not have sufficient primary schools

TEIP Townships (n=149) 78.5 18.1 1.3 2.0


Your township does not have sufficient middle schools

TEIP Townships (n=126) 71.4 23.0 3.2 2.4


Your township does not have sufficient high schools

TEIP Townships (n=125) 64.8 21.6 8.8 4.8



Annex 5.3 Lessons learned from other education service providers of six townships

1. Private schools: This section depicts some promising practices or lessons learned that could be replicated in future decentralised planning and implementation process. Analyses of data of GIs and IDIs with head teachers and owners of five private schools from Hlaing Thar Yar, Htan Ta Bin, Tachileik, Lashio and Mudon and direct observations of these schools were used to prepare this section. (1) Owners of five private schools were either former government school teachers or private tuition teachers who were very committed to improving the education sector. Three of five schools charged approximately Kyats 30,000 to 40,000 per month for eight to ten grades, which were slightly higher than cost incurred for government schools if tuition fees were included. (2) They adopted the standards of the MOE by using the curriculum and examination system and providing necessary information. (3) They adhered to the accreditation requirements. (4) They made use of limited resources by employing retired teachers and officials from MOE. (5) They created some teaching techniques based on lessons learned: (a) 45 minutes allocated for one teaching session was assessed to be inadequate because teachers could not revisit lessons for students who could not comprehend adequately. Thus, some schools extended one teaching session to one hour for the 6th grade to the 11th grade; (b) one assistant teacher was assigned to a teacher to supervise students, and as a result, teachers could pay adequate attention to all students and students paid more attention to learning; (c) some students who could not perform well were identified by teachers. If these students had to learn the whole curriculum, they would learn less. These students were taught some selected lessons of the curriculum, which helped these students learn these lessons better; (d) a regular assessment of lessons was undertaken every day or once a week by either guides or students. Students who performed well had to listen to oral presentation of other students or students broken into five groups had to exchange assessment papers of each other for review and feedback. All schools set aside ample time for self-learning under coaching. These two practices helped students to digest lessons thoroughly; (e) teachers and a head teacher produced a variety of questions for each lesson, which were compiled and used by teachers and students. While teachers applied this compilation of questions for preparing lessons from different dimensions, students used this book to prepare for assessments; and (f) another teacher had to make corrections of homework or exercises of students of a teacher and corrected homework/exercises had to be submitted to a head teacher. In this


87

way, both performance of students and performance of teachers could be examined. (6) Private schools held regular meetings with parents, in which most parents participated and discussed. When a student had problems in learning or disciplines, his or her parents were invited to meet with a head teacher or a head teacher visited parents’ house. This action seemed to work because parents appreciated it and discussed behaviours of students openly. (7) Teachers were assigned to teach one subject for primary or middle or high grades regularly, which helped teachers to learn and improve his or her skills better. (8) Teachers from private schools could focus entirely on their primary task of teaching because their time was not detracted by other tasks assigned by the MOE or other government departments. (9) One private school created a feedback mechanism for students. Students could put an anonymous letter to express their opinions. Once a week, anonymous letters were reviewed, and issues and solutions to address issues raised were discussed in a student meeting. This feedback mechanism could address many issues that would otherwise not have been identified and solved. (10) One private school offered teaching of four skills of English, Chinese and Shan to students in addition to the school curriculum because acquiring these language skills assisted students in getting jobs with a higher salary and/or managing own businesses better. Many students attending formal government schools in Lashio also attended private language classes outside of the school hours. Teaching language skills in private school enabled students to learn languages in conjunction with the curriculum from one service provider. (11) Teachers with relatively fewer experience were coached by experienced teachers for at least one year, and a coached teacher observed teaching of a trainee teacher and gave feedback. 2. Community-based education service providers: Hlaing Thar Yar Development Network (HDN) was established with involvement of four self-help groups, one for people with disabilities, one for youth, five for education, and 12 for Community Based Organizations (CBO). Thirty four representatives from self-help groups and CBOs of the network participated in the management of the network. All of these CBOs and self-help groups were created and operated through financial, material, managerial and technical assistance from World Vision Myanmar (WVM), an international NGO operating in Myanmar for more than 20 years. CBOs and education self-help groups have been implementing many activities or initiatives to improve the education sector. First, they set up 20 non-formal education classes in various wards to provide basic literacy, numeracy, life skills and linkage with acquisition of livelihood skills to approximately 400 students per year. Non-formal education initiative also implemented parental education for parents/caretakers to help them manage their families better and understand the value of education. Outreach educators interacted with parents two times per month with one hour per time for six months. Second, approximately 50 community-based ECCD initiatives were run for about 400 students. 30 teachers received ECCD training from Department of Social Welfare (DSW). Third, night group study was set up to help students to study lessons under coaching of guides. This activity enabled poor students who could not afford to take tuition to revisit school lessons with assistance from guides. Fourth, livelihood skills such as dress-making, sandal-making, embroidery, etc., were taught and students who completed these courses were linked with business sectors for employment. CBOs and self-help groups under HDN collaborated with township DSW for running ECCD and learning livelihood skills for non-formal education students from a Kyaik Wine’ vocational schools of DSW. They also worked closely with TED to implement non-formal education and facilitate access to continued education in government schools of non-formal education students. Non-formal education students who completed two modules had to attend non-formal primary education run by government schools first. After finishing non-formal primary education, they were accepted by formal schooling. A resource mobilisation mechanism was established by all CBOs and self-help groups under HDN. Initial seed fund was provided to them by WVM, which was invested to set up various forms of income generating activities. Some of them were: (a) renting trishaws; (b) renting plates, spoons and other


88

utensils for wedding and other merit making ceremonies; (c) dress-making business; (d) sandal-making business; and (e) retailing of basic household food. Part of profits gained were reinvested while a portion of profit was used to run activities for education, health and other sectors. At the time of this evaluation, approximately Kyats 2,500,000 or 60 per cent of the total cost of Kyats 4,000,000 for various development activities of one CBO could be supported by profit from its income generating activities. At the time of this evaluation, WVM was phasing out from Hlaing Thar Yar. An exit strategy was developed and implemented by WVM and HDN for the previous three years, which helped HDN to continue their activities to certain extent. CBOs and self-help groups of HDN developed an annual plan in September of each year. When WVM was present, an annual plan of various CBOs and self-help groups were submitted to WVM for their review and subsequent funding.

Annex 5.4 Experiences of decentralisation of education sector from other countries

1. Market-based competition and school choice: A different type of education decentralisation reform currently being proposed involves improving education outputs through market-based competition and choice. The term ‘school choice’ means giving parents the power and opportunity to choose the school their child will attend. Traditionally, children in the US are assigned to a public school according to where they live. People of means are considered to have school choice, because they can afford to move to an area according to the schools available (i.e., where the quality of public schools is high), or they can choose to enrol their child in a private school. Parents without such means, until recently, generally had no choice of school, and had to send their child to the school assigned to them by the district, regardless of the school’s quality or appropriateness for their child. One hypothesis for school choice reforms is that competition between local governments promotes efficient use of resources and reduces the overall size of government. Moreover, residents will ‘vote with their feet’ by moving to another locality according to local taxes and the quality of education services. Another hypothesis states that under school choice technical efficiency improves through availability of better information at the local level. School choice means better educational opportunity, because it uses the dynamics of consumer opportunity and provider competition to drive service quality. There are different types of school choice programmes. According to the US for Education Reform there are full school choice programmes, private scholarship programmes, and charter schools. School Choice programmes have been implemented in countries such as Chile, Belgium, and the United States. A. Full school choice programmes, or voucher programs in the US, are government financed per-pupil subsidies given to parents allowing them to enrol the student in a public or private school of their choice. The rationale is that parents are given choice and schools are required to compete for students in order to survive and the quality of education is expected to improve. B. Private scholarship programmes in the US provide private funds to families of low socio-economic status giving them to opportunity to choose between schools. C. Charter schools in the US is a new form of choice, where schools are independent public schools, designed and operated by educators, parents, community leaders, educational entrepreneurs and others. Charter school designs differ according to the country of implementation, but in general they are sponsored by designated local or state educational organizations who monitor their quality and integrity, but allow them to operate free from the traditional bureaucratic system of public schools. Charter schools design and deliver programs tailored to educational excellence and community needs. D. Voucher Programme in Chile, introduced in 1980, was a modified voucher scheme, under which families could choose to send their children to free subsidised schools, either municipal or private, or they can choose fee-paying private schools if they could afford the tuition fees. A study by Parry (1997) discovered that public schools achieved higher performance with disadvantaged children while private-subsidised schools produce higher scores with ‘high quality’ students. A study by Vegas (1999) unveiled


89

that teacher education, decentralisation of decision-making authority, school’s schedule enforcement and teacher’s autonomy in designing teaching plans and implementing projects affected student outputs and that teacher autonomy was found to have a positive effect on student outputs only when decision-making authority is decentralised. 2. Increased local autonomy in Zimbabwe: In the late 1980’s the MOE of Zimbabwe launched an initiative that delegated decision-making power to local communities for construction of primary schools and authority to hire and fire teachers, and disbursed schools the per capita grants and teacher’s salaries paid to them by the education ministry. The MOE retained the authority of designing the curriculum, conducting examinations, and training teachers. Lessons learned included: (a) teachers were not paid on time, and some district councils retained some of per-pupil grants for non-educational activities instead of passing them to individual schools; (b) the central government discovered that numerous wages had to be paid to phantom teachers; and (c) the education decentralisation process failed and overall quality of education remained low. 3. Decentralisation reform in Minas Gerais, Brazil: In Minas Gerais, Brazil, in the early 1990s, only about 40 per cent of students completed all eight grades of primary school and low performance of education outputs was attributed to inadequate funding, poorly trained teachers, rigid pedagogies, and over regulated management. The state government enacted an educational reform to grant financial, administrative, and pedagogical autonomy to elected boards in each school, composed of teachers, parents, and students over the age of sixteen, which could decide how to spend grant funds and locally raised education revenues as well as on curriculum, pedagogy, school calendar, and other functions. However, teachers’ union bargaining was maintained at the state level. Lobo et al. (1995) discovered that school autonomy and greater transparency in decision-making led to increased operational efficiency. Early results of the 1994 student achievement tests of third graders also showed that in comparison with 1992, scores rose by 7 percent in science, 20 percent in Portuguese and 41 percent in mathematics. 4. School autonomy and decentralisation in The Netherlands: The Dutch education system has been decentralised and demand-driven since 1917. Almost 70 per cent of schools are administered and governed by private school boards. Public and private schools are funded by the government, and most parents have a choice of several schools near their homes, which spurred schools to develop a unique profile and to improve the education they offer. The MOE still imposes a number of statutory quality standards. In recent years, more central government powers have been transferred to the individual school level and the central government control is confined to broad policy-making and to creating the right conditions for the provision of quality education. Institutions are given greater freedom to allocate their resources and manage their own affairs, although they still are held accountable to the central government for their performance and policies. Additional funding is provided for schools in districts and regions with high numbers of underprivileged families. 5. School voucher system in Sweden: Sweden began a radical reform of its primary and secondary school system in the 1990s. All independent schools approved by the National Agency for Education are entitled to public funding. A voucher system replaced the centralised system, and a parental choice reform was instituted, which led to a significant rise in both the number of independent schools, and in the number of students enrolled. A new school enrolment rule allows money to follow students, and municipalities are required to provide capitation grants to each private school equal to 85 per cent of the public school cost, which enables nearly 90 per cent of the private schools to be free from charging fees. As a result, enrolments in private schools continue to grow, more than doubling in recent years to reach almost three percent of total enrolments. The Nacka municipality, outside Stockholm, created this voucher system. Each year, parents are given a


90

catalogue profiling all the local schools plus a voucher that is to be handed over to the school of their choice. Parents who do not choose a school are contacted by some of the closest schools to encourage an active choice and are also required to present a new voucher before the first, fourth, and seventh grade even if the child is attending the same school. Private and public schools alike follow the national curriculum. The competition of schools for students has resulted in more efficient allocations of funds and clearer institutional focus. 6. Capitation grants in Armenia: The government of Armenia, under its Model Charter for autonomous schools, embarked on a reform strategy to place more responsibility at school level by establishing a framework for managing education reform, including development of detailed implementation plans and capacity building for reforms of school finance and governance. Schools received lump sum funding from the Treasury on a capitation basis, and were free to allocate these funds between different inputs within specified limits such as minimum salary rates. Newly established school boards with principals selected by the board managed budget. Pilot implementation was carried out in 10 per cent of the country's schools. The project funded technical assistance to define details of the new funding formula, legal and regulatory framework, accountability and reporting requirements, and supported training of school principals, board members and accountants. The Pilot School Improvement Programme was designed to build management capacity at the school level to match the autonomy reforms by providing grants up to US$10,000 to schools for self-identified projects. These were for investment projects, and not for normal recurrent costs or reconstruction/civil works. Its typical components included purchase of equipment and teaching materials and teacher training in new subjects/methods – geared either to teaching the core curriculum better, or to introducing extra-curricular classes. Projects must show a strategy for sustainability and included providing paid services to the community – for example computer or language training -- or selling product of extracurricular vocational activities – for instance, agricultural/food products. Schools must be autonomous and finance 10 per cent of costs. 7. Spain’s democratisation and decentralisation reform: Spain has decentralised many aspects of its formerly centralised education system to 17 autonomous communities since late 1970s. The autonomous communities, established in 1978 to support the transition to democracy from the former dictatorship government, set up democratically elected parliaments and adopted a self-rule. The MOE established the “minimum academic requirements” on curriculum content to meet the goal of having one educational system composed of 17 integrated, semi-autonomous bodies rather than separate educational systems. The requirements formalised the MOE’s regulation of 55 to 65 per cent of the curriculum while still granting curricular freedom to reflect local and regional priorities. The central government established a block grant funding system that included funding for education, health, and transport. In addition, Inter-Territorial Compensation Funds (FCI) were established to achieve greater financial equity between wealthy and impoverished regions. By 1996, education spending had increased to over 5 per cent of GDP compared to 1.8 percent in 1975. In 1985, Spain enacted the Right to Education Law (LODE), reinforcing the decentralisation and democratisation of education. An administrative structure, known as State School Council (Consejo Escolar del Estado), which was an 80-member national level advisory body, was set up. It was required to meet at least once a year and to provide feedback on the state of education in Spain. The council and its members were encouraged to submit proposals for educational change. The CEE membership included representation from teachers, parents, administrative staff, trade unions, private schools, tertiary institutions, MOE administrators (10 per cent) and education scholars. The presence of ministry appointed delegates (10 per cent) often served as a disincentive for council members to critique ministry proposals and often challenged the effectiveness of the council. Conference of Education Counsellors brought the Minister of Education and the Chief Education Officers from each autonomous communities. It was required to meet at least once a year and comprised of 5 sub-committees that explored a range of educational issues. The Conference faced challenges due to the politics of the participants. The LODE established Consejos


91

Escolares del Centro (CEC) in each of the 17 autonomous communities. There were few CECs in effect during the first years of the reform. LODE also required the creation of Consejos Escolares (CE) in every public and private school receiving government funding. The primary focus of the principal was implementing the policies of the CE. The principal’s responsibilities also included managing the school budget and overseeing personnel issues. The principal also worked with the Chief of Academic studies to guide the teaching and learning processes. The school principal was elected by an absolute majority vote of the School Council members and could be fired by a two-thirds majority. After 20 years of decentralisation reform, Hanson and Ulrich (1994) stated that “School Based Management (SBM) was playing an important symbolic role in democratic participation at the local level, but did not yet demonstrate the anticipated improvement in administrative processes." While the three different levels of administration were designed to collaborate and work together, there was little or no evidence that this occurred. The authors also noted that "almost without exception, the interviews conducted for this study revealed that educators recognised that the practice of school-based management was not proving to be an effective mechanism for improving the quality of management and/or education in the schools.” While SBM might not yet have achieved all of its stated goals, it was successful in establishing a more widespread acceptance of the government’s commitment to democratic participation. The number of public schools increased from 1,100 in 1975 to approximately 3,000 in 1995, thus increasing access to education. Moreover, in 1975, only 70 per cent of 14 year olds were in school compared to approximately 100 percent of 15 year olds enrolled in 1995. Unfortunately, the long-term impact of these changes has yet to be observed and evaluated. 8. Education decentralisation efforts in Mexico: Before the Mexican educational reform in 1993, the education system was highly centralised and highly inefficient. One out of seven primary-age students lacked access to school, and in poor states such as Chiapas less than 20 per cent of students were in school. In addition, newly hired teachers waited over a year for their first pay-cheques, and any mistakes would have to be corrected by a costly and time-consuming trip to the capital. The education decentralisation process was implemented in three stages. Between 1978 and 1982, the MOE deconcentrated management of the education system to each of the thirty-one states of Mexico. Each state was given responsibility from budgeting and managing schools to the writing of curriculum and textbook choice. Revenue generation, core curriculum design and labour policy remained at the central level. During this first stage of reform, enrolments of preschool, primary and secondary increased, especially in rural areas. During the second phase of the reform, 1983 to 1988, the government intended to transfer additional control to the authority of the states. Nevertheless, it failed because of teacher unions’ opposition of negotiating with thirty-one states. Moreover, central government staff members resisted due to their interests in the centralised system and their long-standing cooperative arrangements with teachers (Fiske 1996). In 1988, a new government came to power and negotiated an agreement with the national teachers’ union which permitted the 1993 “Ley General de la Educación.” This new law transferred most educational decision-making authority for primary and secondary schools to the state governments. However, the central government’s role in financing education through negotiated transfers to the states resulted in de facto continued centralisation. It was not until 1998 that decentralisation was in place when education transfers became automatic. The central government continued to directly operate a system of rural schools called CONAFE (National Board for Educational Improvement), which ensured learning opportunities for remote rural areas, especially for indigenous children (Fiske 1996). While decentralisation efforts in Mexico have not been primarily focused on improving learning, some components of the reform may have a positive impact on learning such as changes in teacher evaluations and pay as well as additional resources for poor and indigenous rural children. While CONAFE schools gave parents a more important role than was found in the traditional public schools, teachers and parents were not yet actively engaged in leads to learning improvements at the level of the school. An empirical study would help to determine any learning output improvement from education decentralisation reforms.


92

9. From decentralisation to centralisation to decentralisation in Colombia: Colombia was an interesting case of a country that implemented centralisation reforms to correct a failed decentralised system. After twenty years, Colombia once again implemented decentralisation reforms as a medium of improving public services. Following World War II, Colombia implemented decentralisation reforms to break up an “oligarchical democracy” where political elites of the Conservative and Liberal Parties and the Roman Catholic Church controlled the country. Under the original decentralisation reform, local municipalities exercised considerable control over education but lacked the financial, administrative, and political capacity to generate revenues, manage schools, and deal with teacher strikes. The centralised system established in 1970 was created to correct the decentralised system. The MOE in Bogotá controlled all important decisions regarding curricula, textbooks, and other matters of educational policy. In addition, teachers were employees of the central government whose salaries were negotiated at the central level. This new system was successful at improving educational efficiency and at ending teacher strikes. However, after two decades, the centralised system developed into bureaucratic arteries which were unable to cope with growing demands for local autonomy. Moreover, during the centralised system in late 1980s, Colombia was at the edge of political, economic, and social collapse. Terrorist guerrillas and the corrupting influence of drug cartels had invaded the country. The government’s step to restore credibility was to give ordinary citizens a greater role in managing public institutions. The first step into greater decentralisation was in 1985, when popular elections of the mayors of Colombia’s 1,024 municipalities and thirty-three state governors were instituted. In 1989, Congress approved to hand municipalities a greater role in making decisions of the education and health sectors. The government’s new decentralisation reform was an effort to “municipalise” basic education and to increase the autonomy of local schools. Financial resources were transferred to municipalities and departments, and schools were given responsibility for managing personnel, design parts of the curriculum, and control aspects of finance. Moreover, parents and teachers were to gain greater voice in running schools and a voucher system for poor students was instituted at the secondary level. The legislation was adopted in 1993 and 1994 after continuous resistance from teacher unions. Local schools did not obtain autonomy to select, hire, and sanction personnel. A system of teacher evaluation was established, but measures of student output, such as test scores, were excluded. The effects of the new decentralisation reform were mixed. The 1994 budget for education increased to 3.65 per cent of gross domestic product, which was above the target figure of 3.5 per cent. In addition, parent and community groups were not well organised, nor were the mayors and governors who had been recently elected. In summary, the decentralisation effort in Colombia was successfully at improving legitimacy of the government, but the impact was diminished by failure of support from important players including governors, community members and teachers. 10. Indonesia and Philippine’s local funding: James, King and Suryadi (1996) investigated the impact of private versus public finance of education and private versus public management of schools on school cost and efficiency. The authors used a multi-product production function subject to a budget based on central government funding and local sources such as parental fees and contributions. A Cobb-Douglas variable cost function was then derived and empirically tested to estimate efficiency as the cost per student of achieving a given level of academic performance and a given level of enrolment. A key issue in the cost function estimation was the presence of endogeneity in the source of funding. Instruments for local share of funding were then used to solve the issue. James, King and Suryadi found that in Indonesia, where schools generally operated at very low funding levels, more money was likely to bring better school quality as measured by examination scores. Private management was found to be more efficient than public management in achieving academic quality. Moreover, this study demonstrated that local funding further enhanced efficiency regardless of whether the school was public or private. However, the incremental effect declined as the local funding share increased. In the study of the impact of local contributions on the efficiency of management and finance


93

in Indonesia, the authors measured only the fiscal dimension of community funding and not the decision-making community involvement. Since community and parental contributions played an important role in education funding (30 per cent of total education funding) in Indonesia, did the community and parental associations have a word in the decision-making process of education functions? If yes, how did it contribute to efficiency effect estimated? These were some of the questions that could be further explored about education decentralisation in Indonesia.

Annex 5.5 Perceived improvement in the education sector of Myanmar

Factors related to improvement or deterioration of education sector in TEIP townships and non-TEIP townships

Worse Indifferent

Better

% % % Primary school students

Long distance between home and school TEIP Townships (n=157) 1.9 73.9 24.2

Non-TEIP Township (n=83) 2.4 86.7 10.8

TED survey (n=17) 0.0 88.2 11.8

Lack of bicycle/motor cycle/bus/boat/ferry to go school or having necessary to go to school on foot

TEIP Townships (n=153) 1.3 70.6 28.1


TED survey (n=17) 0.0 94.1 5.9

Having necessary to go through stream/river

TEIP Townships (n=121) 4.1 79.3 16.5


TED survey (n=17) 0.0 94.1 5.9

Could not afford to pay transportation cost TEIP Townships (n=118) 4.2 78.0 17.8

Non-TEIP Township (n= 62) 0.0 82.3 17.7

TED survey (n=17) 0.0 100 0.0


TEIP Townships (n=179) 2.8 54.7 42.5


TED survey (n=17) 5.9 82.4 11.8

Flood/ landslide/arm conflict TEIP Townships (n=103) 8.7 78.6 12.6

Non-TEIP Township (n=54) 1.9 87.0

11.1

TED survey (n=17) 94.1 5.9

Migration of parents TEIP Townships (n=211) 10.9 74.9 14.2


TED survey (n=17) 5.9 70.6 23.5


TEIP Townships (n=219) 8.7 61.2 30.1


TED survey (n=17) 0.0 70.6 29.4

Divorce/remarriage/death of parents TEIP Townships (n=199) 13.1 75.4 11.6


TED survey (n=17) 0.0 82.4 17.6


TEIP Townships (n=224) 10.7 68.3 21.0


TED survey (n=16) 0.0 93.8 6.3


TEIP Townships (n=202) 7.9 72.8 19.3


TED survey (n=16) 0.0 87.5 12.5

Children could not study lessons well TEIP Townships (n=228) 7.9 58.3 33.8


TED survey (n=17) 0.0 100 0.0

School is located in other ward/village TEIP Townships (n=126) 3.2 86.5 10.3


TED survey (n=17) 0.0 94.1 5.9


94

Students are lured by human traffickers TEIP Townships (n=104) 4.8 76.0 19.2


TED survey (n=17) 0.0 94.1 5.9


TEIP Townships (n=102) 6.9 76.5 16.7


TED survey (n=17) 5.9 94.1 0.0

Students encounter road traffic injury TEIP Townships (n=111) 3.6 78.4 18.0


TED survey (n=17) 5.9 94.1 0.0

Presence of animals with toxin such as snakes in the school premises

TEIP Townships (n=133) 1.5 76.7 21.8


TED survey (n=17) 0.0 100 0.0

Breeding of mosquitoes in the school premises result in mosquito borne diseases such as dengue haemorrhage fever

TEIP Townships (n=174) 3.4 48.3 48.3


TED survey (n=17) 0.0 88.2 11.8

Insanitary latrines and drinking water sources result in water borne diseases such as diarrhoea

TEIP Townships (n=173) 2.9 48.6 48.6


TED survey (n=17) 0.0 94.1 5.9

Middle School Students



TED survey (n=17) 0.0 94.1 5.9

Could not afford to pay transportation cost TEIP Townships (n=69) - 82.6 17.4

Non-TEIP Township (n=36) - 86.1 13.9

TED survey (n=17) 0.0 100 0.0


TEIP Townships (n=89) 2.2 62.9 34.8


TED survey (n=17) 0.0 88.2 11.8




TED survey (n=16) 0.0 81.3 18.8




TED survey (n=16) 0.0 87.5 12.5

Could not afford to pay tuition fees TEIP Townships (n=74) 8.1 78.4 13.5


TED survey (n=16) 0.0 93.8 6.3



TED survey (n=17) 0.0 94.1 5.9


TEIP Townships (n=105) 7.6 71.4 21.0


TED survey (n=17) 0.0 70.6 29.4



TED survey (n=16) 0.0 87.5 12.5


TEIP Townships (n=107) 14.0 65.4 20.6


TED survey (n=17) 0.0 76.5 23.5




TED survey (n=16) 0.0 81.3 18.8





95

TED survey (n=17) 0.0 94.1 5.9

Middle graders could not follow schools’ lessons because they did not digest lessons in primary grades

TEIP Townships (n=109) 14.7 63.3 22.0


TED survey (n=17) 0.0 76.5 23.5


TEIP Townships (n=91) 16.5 64.8 18.7


TED survey (n=17) 17.6 82.4 0



TED survey (n=17) 0.0 88.2 11.8




TED survey (n=17) 0.0 100 0.0




TED survey (n=17) 0.0 88.2 11.8

Students are lured by human traffickers TEIP Townships (n=53) 3.8 79.2 17.0


TED survey (n=17) 0.0 100 0.0




TED survey (n=17) 0.0 100 0.0



TED survey (n=17) 0.0 87.5 12.5

High School Students



TED survey (n=16) 0.0 87.5 12.5

Could not afford to pay transportation cost TEIP Townships (n=20) - 80.0 20.0


TED survey (n=17) 0.0 76.5 23.5




TED survey (n=15) 0.0 93.3 6.7


TEIP Township (n=30) 3.3 63.3 33.3


TED survey (n=16) 0.0 93.8 6.3


TEIP Townships (n=30) - 73.3 26.7


TED survey (n=16) 0.0 87.5 12.5

Could not afford to pay tuition fees TEIP Townships (n=29) 6.9 79.3 13.8


TED survey (n=15) 0.0 93.3 6.7

Could not afford to pay costs collected by school



TED survey (n=17) 0.0 88.2 11.8



TED survey (n=17) 0.0 94.1 5.9


TEIP Townships (n=37) 16.2 59.5 24.3


TED survey (n=16) 0.0 100 0


96



TED survey (n=16) 0.0 75 25


TEIP Townships (n=37) 13.5 70.3 16.2


TED survey (n=16) 0.0 87.5 12.5




TED survey (n=17) 0.0 88.2 11.8




TED survey (n=17) 0.0 82.4 17.6

High graders could not follow school’s lessons because they did not digest lessons in primary and middle grades

TEIP Townships (n=37) 32.4 51.4 16.2


TED survey (n=17) 5.9 88.2 5.9


TEIP Townships (n=38) 18.4 60.5 21.1


TED survey (n=17) 0 88.2 11.8



TED survey (n=16) 0.0 87.5 12.5




TED survey (n=16) 0.0 100 0.0




TED survey (n=17) 0.0 100 0.0

Students are lured by human traffickers TEIP Townships (n=20) - 80.0 20.0


TED survey (n=17) 0.0 100 0.0


TEIP Townships (n=20) - 80.0 20.0


TED survey (n=17) 0.0 100 0.0



TED survey (n=17) 0.0 100 0.0

Teachers

Number of student outweigh number of teachers

TEIP Townships (n=162) 18.5 41.4 40.1


TED survey (n=17) 0.0 76.5 23.5

Some teachers have to teach more than one subject, and as a result, they do not have adequate skills

TEIP Townships (n=156) 12.8 53.2 34.0


TED survey (n=17) 0.0 82.4 17.6

Some teachers do not have adequate commitment to their work

TEIP Townships (n=137) 3.6 53.3 43.1


TED survey (n=16) 0.0 100 0.0

Some teachers use internet, computer game and cellular phone during teaching hours

TEIP Townships (n=110) 10.9 58.2 30.9


TED survey (n=17) 0.0 88.2 11.8

Some teachers do tuition that result in giving favours to students who take tuition from them or students fell like that

TEIP Townships (n=103) 5.8 65.0 29.1


TED survey (n=16) 0.0 100 0.0

TEIP Townships (n=122) 8.2 58.2 33.6


97

Some teachers only want to completion of students but do not care competency of students adequately


TED survey (n=17) 0.0 88.2 11.8

Some teachers do not have adequate accountability for maintenance of furniture, school building, etc.

TEIP Townships (n=150) 2.7 54.0 43.3


TED survey (n=17) 0.0 94.1 5.9

Students did not revisit the previous lessons after each chapter end test that consequently undermined quality of education

TEIP Townships (n=167) 19.8 51.5 28.7


TED survey (n=17) 5.9 94.1 0.0

Students had lower value on education because all students passed examinations of all grades except 5th, 9th and 11th grades regardless of their competency

TEIP Townships (n=170) 21.2 48.2 30.6


TED survey (n=17) 0.0 82.4 17.6

Your school have many projects that result in cumbersome work load and drain on time of teachers and head teachers

TEIP Townships (n=151) 21.9 57.0 21.2


TED survey (n=17) 5.9 88.2 5.9

Teachers have to take non-teaching tasks that undermine their teaching tasks

TEIP Townships (n=148) 22.3 59.5 18.2


TED survey (n=17) 0.0 94.1 5.9

You school do not have adequate non-teaching staff

TEIP Townships (n=146) 24.7 58.9 16.4


TED survey (n=17) 5.9 88.2 5.9

Assessment of quality of teaching is insufficient

TEIP Townships (n=147) 6.1 48.3 45.6


TED survey (n=16) 6.3 81.3 12.5

Teachers who receive training do not apply their knowledge into their teaching adequately

TEIP Townships (n=175) 4.0 49.7 46.3


TED survey (n=16) 5.9 94.1 0.0

Your school does not have sufficient building TEIP Townships (n=161) 16.1 56.5 27.3


TED survey (n=17) 0.0 82.4 17.6

Your school does not have sufficient furniture

TEIP Townships (n=168) 7.7 48.2 44.0


TED survey (n=17) 0.0 76.5 23.5

Your school does not have sufficient text books

TEIP Townships (n=143) 0.7 39.2 60.1


TED survey (n=17) 0.0 100 0.0

Your school does not have sufficient teaching aid

TEIP Townships (n=167) 3.6 52.1 44.3


TED survey (n=17) 0.0 88.2 11.8

Your school does not have sufficient latrine and dirking water or sufficient sanitation of them

TEIP Townships (n=155) 9.0 34.2 56.8


Your school does not have sufficient primary school teachers with accreditation

TEIP Townships (n=152) 13.2 49.3 37.5


Your school does not have sufficient middle school teachers with accreditation



Your school does not have sufficient high school teachers with accreditation



Your township does not have sufficient primary schools



Your township does not have sufficient middle schools





98

Your township does not have sufficient high schools



Key performance indicators compared between period of 2011-2012 and 2012-2013 and the period of 2013-2014 and 2014-2015

Worse Indifferent

Better

% % % Annual percentage increase in primary enrolment

TEIP Townships (n=233) 7.7 27.5 64.8

Non-TEIP Township (n=94)

6.4 33.0 60.6

TED survey (n=15) 20 0.0 80

Primary retention rate TEIP Townships (n=225) 10.7 29.8 59.6


9.7 34.4 55.9

TED survey (n=17) 29.4 11.8 58.8

Middle school retention rate TEIP Townships (n=90) 6.7 36.7 56.7


9.8 41.5 48.8

TED survey (n=16) 12.5 6.3 81.3

High school retention rate TEIP Townships (n=37) 8.1 35.1 56.8


5.6 44.4 50.0

TED survey (n=16) 37.5 0.0 62.5

Transition rate from primary to middle school TEIP Townships (n=198) 5.6 29.3 65.2


10.0 32.9 57.1

TED survey (n=17) 23.5 5.9 70.6

Transition rate from middle to high school TEIP Townships (n=59) 5.1 37.3 57.6


9.1 36.4 54.5

TED survey (n=17) 35.3 0.0 64.7

Percentage of grade 1 intake with ECCE experience

TEIP Townships (n=142) 6.3 29.6 64.1


5.6 40.7 53.7

TED survey (n=15) 13.3 0.0 86.7

Number of NFE primary education students TEIP Townships (n=25) - 44.0 56.0


TED survey (n=6) 50 16.7 33.3

Primary completion rate TEIP Townships (n=207) 8.2 31.9 59.9


4.7 39.5 55.8

TED survey (n=16) 56.3 0.0 43.8

Percentage of oversize classes in basic education schools (more than 40 students)

TEIP Townships (n=125) 13.6 44.8 41.6


16.3 46.9 36.7

TED survey (n=11) 36.4 0.0 63.6

Percentage of Schools with Student Teacher Ratio more than 40

TEIP Townships (n=126) 13.5 54.8 31.7


17.4 52.2 30.4

TED survey (n=13) 76.9 0.0 23.1

Percentage of certified primary school teachers TEIP Townships (n=198) 12.6 35.9 51.5


5.8 51.2 43.0

TED survey (n=14) 42.9 35.7 21.4


99

Percentage of certified secondary school teachers

TEIP Townships (n=135) 4.4 37.8 57.8


4.8 50.0 45.2

TED survey (n=15) 20 13.3 66.7

Percentage of certified high school teachers TEIP Townships (n=53) 1.9 32.1 66.0


5.0 35.0 60.0

TED survey (n=15) 26.7 20 53.3

Percentage of schools has one of the followings: (i) a boy scout/ a girl guide; (ii) student council; (iii) a school red cross association

TEIP Townships (n=168) 1.2 49.4 49.4


0.0 31.5 68.5

TED survey (n=12) 8.3 58.3 33.3

Percentage of private schools TED survey (n=3) 0.0 33.3 66.7

Percentage of schools submit routine reports 100% and on time

TEIP Townships (n=239) 0.0 27.2 72.8


1.1 21.1 77.8

TED survey (n=13) 0.0 46.2 53.8

Teaching capacity of teachers TEIP Townships (n=254) 1.2 27.2 71.7


1.0 20.8 78.1

TED survey (n=6) 50 33.3 16.7

General knowledge of student TEIP Townships (n=251) 1.2 32.3 66.5


0.0 27.1 72.9

TED survey (n=10) 30 20 50

Morality of students TEIP Townships (n=256) 2.0 27.7 70.3


0.0 16.5 83.5

TED survey (n=16) - - -

Obeying rules and regulations of students TEIP Townships (n=254) 2.4 24.4 73.2


0.0 14.4 85.6

TED survey (n=14) 0.0 79.6 21.4

Acquisition course of study of students TEIP Townships (n=254) 2.8 33.1 64.2


1.0 21.9 77.1

Life skills of students TEIP Townships (n=247) 1.6 33.6 64.8


0.0 31.2 68.8

TED survey (n=14) 0.0 42.9 57.1

Vocational skills of students TEIP Townships (n=192) 4.7 71.9 23.4


1.4 67.1 31.5

TED survey (n=13) 0.0 69.2 30.8



100

Annex 5.6 Other relevant findings

Factors related to development and implementation of TEIP Inadequate (per cent)

School survey (n = 262)

TED survey (n = 17)

Human Resources for development and implementing programme 76 (n=199) 64.7 (n=11)

Budget for development and implementing programme 95.8 (n=251) 70.6 (n=12)

Materials for development and implementing programme 92.7 (n=243) 58.8 (n=10)

Time for development and implementing programme 81.7 (n=214) 41.2 (n=7)

Need assessment skill for development and implementing programme

89.7 (n=235) 58.8 (10)

Planning skill for development and implementing programme 87 (n=228) 58.8 (n=10)

Resource mobilisation skill for development and implementing programme

87.8 (n=230) 47.1 (n=8)

M&E skills for development and implementing programme 84.7 (n=222) 35.3 (n=6)

Skills in utilisation of analysis of monitoring data 87 (n=228) 37.5 (n=6)

Ownership of stakeholders for development and implementing programme

71.8 (n=188) 23.5 (n=4)

Leadership of stakeholders for development and implementing programme

68.7 (n=180) 29.4 (n=5)

Accountability of stakeholders for development and implementing programme

71.4 (n=187) 23.5 (n=4)

Spare adequate time of stakeholders for development and implementing programme

82.1 (n=215) 64.7 (n=11)

Organizational management system, financial management system, M&E system for development and implementing programme

86.6 (n=227) 58.8 (n=10)

Source: Survey of School and TEDs

Involvement in TEIP programme Participate (per cent)

School Survey (n = 262)

TED Survey (n = 17)

Involved as facilitators in TEIP training/workshop 9.5 (n=23) 52.9 (n=9)

Involved as participants in TEIP training/workshop 42.1 (n=102)

100 (n=17)

Involved as TEIP working group members 11.4 (n=27) 88.2 (n=15)

Involved as TEIP working group members after attending TEIP training 12.8 (n=30) 94.1 (n=16)

Involved as TEIP working group members although did not attend TEIP training 8.7 (n=20) 60 (n=9)

Involved in TEIP planning process as TEIP working group members after joining TEIP training

12 (n=28) 87.5 (n=14)

Involved in TEIP planning process as TEIP working group members although did not participate in TEIP training

9.5 (n=22) 53.3 (n=8)

Involved in examining education needs of township level 5.6 (n=13) 68.8 (n=11)

Involve in utilizing needs identified to develop a TEIP 7 (n=16) 73.3 (n=11)

Source: Survey of schools and TEDs


101

TEIP addressed the following issues compared between period of 2011-2012 and 2012-2013 and the period of 2013-2014 and 2014-2015

Worse Indifferent Better

Discrepancy between male and female with respect to access to and the quality of education

School (n=106)

15.1 59.4 25.5

TED (n=17) 0.0 82.4 17.6

Discrepancy between children with four types of disability and without four types of disability with respect to access to and the quality of education

School (n=93) 17.2 58.1 24.7

TED (n=15) 0.0 80 20

Discrepancy between HIV infected and affected children and children not infected and affected by HIV with respect to access to and the quality of education

School (n=52) 9.6 65.4 25.0

TED (n=16) 12.5 81.3 6.3

Discrepancy between children from migrant family and children from non-migrant family with respect to access to and the quality of education

School Not ask Not ask Not ask

TED (n=16) 6.3 62.5 31.3

Discrepancy between children with extreme poverty and children who are better off with respect to access to and the quality of education

School (n=127)

10.2 44.1 45.7

TED (n=17) 5.9 64.7 29.4

Discrepancy between children from minority group and children from majority group with respect to access to and the quality of education

School Not ask Not ask Not ask

TED (n=16) 0.0 75 25

Source: Survey of schools and TEDs

Average service months and years of officials of township education department

Average service months

Average service years

TEO (n=11) 20 1.7

Deputy TEO (Admin and Finance) (n=16) 27 2.3

Deputy TEO (Education and Project M&E) (n=14) 6 -

Officer (Statistic) (n=2) 125 10.4

Assistant TEO 1 (n=15) 23 1.9




Source: Survey of TEDs