save the children international basic education … · in january 2013, save the children extended...

24
SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION PROGRAM TiANA PROJECT 2013 ENDLINE REPORT FOR ST ANTHONY AND NAMADIDI EDUCATION ZONES, TRADITIONAL AUTHORITY MLUMBE ZOMBA RURAL, MALAWI November, 2013

Upload: others

Post on 16-Mar-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

SAVE THE CHILDREN INTERNATIONAL

BASIC EDUCATION PROGRAM

TiANA PROJECT 2013 ENDLINE REPORT

FOR

ST ANTHONY AND NAMADIDI EDUCATION ZONES, TRADITIONAL AUTHORITY MLUMBE

ZOMBA RURAL, MALAWI

November, 2013

Page 2: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

Contents Abbreviations .................................................................................................................................. 1

List of figures ................................................................................................................................... 2

Acknowledgement .......................................................................................................................... 3

Methodology ................................................................................................................................... 4

Measurement ................................................................................................................................ 5

Who is still in school? ..................................................................................................................... 6

Baseline and Midline TiANA Project Profile .................................................................................. 7

Average Gains in Learning .............................................................................................................. 8

Equity in gains ............................................................................................................................... 9

Learning and use of reading skills in Zomba ................................................................................. 10

Change in home literacy environment ..................................................................................... 12

Change in community literacy environment ............................................................................ 12

Teachers and Schools ................................................................................................................... 13

Teaching strengths and challenges ............................................................................................... 13

Focus on Lesson Delivery ............................................................................................................. 16

Community Strengths and Challenges .......................................................................................... 16

Youth Literacy Movement (YLM) .................................................................................................. 17

Conclusions ................................................................................................................................... 18

Appendix A: Equity models........................................................................................................... 19

Appendix B: Community Literacy Models -- Chichewa ............................................................... 20

Appendix C: Community Literacy Models -- English ................................................................... 21

Appendix D: Change in Community Literacy Models .................................................................. 22

Page 3: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

1

Abbreviations AusAid Australia Aid DEM District Education Manager EGRA Early Grade Reading Assessment HLE Home Literacy Environment LB Literacy Boost MIE Malawi Institute of Education MSCE Malawi School Certificate of Education MTPDS Malawi Teacher Professional Development Service MUW Most Used Words ODL Open Distance Learning PEA Primary Education Advisor PTA Parents Teachers Association SCI Save the Children International SES Socio Economic Status SMC School Management Committee TA Traditional Authority TiANA Tiwerenge ndi Ana Athu USAID United States Agency for International Development YLM Youth Literacy Movement

Page 4: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

2

List of Annexes Appendix A: Equity models Appendix B: Community Literacy Models – Chichewa Appendix c: Community Literacy Models – English Appendix C: Change in Community Literacy Models

List of figures Figure 1. Percent of children present at end of year assessment by group

Figure 2. TiANA Chichewa reading skills profile: baseline and midline

Figure 3. TiANA English reading skills profile: baseline and midline

Figure 4. Reading English single words and text by HLE and group

Figure 5. Use of Reading Skills at baseline and midline

Figure 6. Community literacy environment (CLE) predicts skill gains

Figure 7. Change in Community Learning Environment predicts skill gains

Figure 8. Revisiting Baseline Areas of Teaching Challenges -- Chichewa

Page 5: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

3

Acknowledgement Save the Children would like to thank all partners for their support in providing insight into actions that help improve the quality of education. In particular Save the Children would like to thank the Ministry of Education, Science and Technology (MoEST), Malawi Institute of Education (MIE), District Education Manager (DEM) and Primary Education Advisors (PEAs) for Zomba Rural, Teacher Trainers from Blantyre and Machinga Teacher Training Colleges for their assistance in the data collection effort right from the development of instruments, pre testing to the actual data collection. Further, Save the Children would like to thank World Vision International (WVI), Australia AID (AusAID) and United States Agency for International Development (USAID)/Malawi Mission in a special way for their financial support and guidance throughout the implementation of the TiANA project.

Matthew Pickard Country Director

Page 6: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

4

Introduction

In January 2013, Save the Children International extended the Basic Education program interventions in Malawi through a project called “Tiwerenge ndi Ana Athu” (TiANA) with funding from the United States Agency for International Development (USAID). TiANA is an innovation under the All Children Reading Grand Challenge (USAID, World Vision International and AusAid). The project intends to underline the role of parents and the community in making reading a reality for all children in a bid to improve learner writing and reading in the lower primary through innovative community-based production and use of teaching and learning materials. The name of the project “Tiwerenge ndi Ana Athu” translates from Chichewa as "Let us read together with our children." This project is being implemented in 10 primary schools in St. Anthony and Namadidi Zones in the area of Traditional Authority Mlumbe in Zomba District and its value adding components include: youth engagement through the Youth Literacy Movement (YLM) to champion literacy activities, incentivising community action through establishment of village savings and loans and engagement of a media house to broadcast reading camp activities. The YLM is a grouping of primary/secondary school graduates which garner support for camp reading and add intellectual capacity in implementation of the project activities. TiANA project is set out to:

• Improve community capacity to support the sustained production of relevant reading, teaching and learning materials.

• Enhance use of appropriate teaching and learning materials by teachers. • Increase parental awareness and support for children’s reading outside school

perimeter.

TiANA project seeks to strengthen children’s reading ability through both community and teacher-focused elements. While the program originally intended to conduct a baseline at the beginning of the school year in the fall of 2012, all project activities were delayed and only started in February of 2013 during which a baseline was conducted to set bench marks for implementation of the project. At the end of the school year in July 2013, the project team revisited the schools to collect end of school year, project midline data. The assessment and survey were done to ensure that the project delivers initiatives that are informed by ongoing evidence of progress, or lack thereof. The results are also meant to assess the impact of the program at different stages, thereby aiding in the identification of key intervention areas that would assist in improving teachers’ professional skills and community assistance to support learning. This report therefore examines the results of a learner background survey and reading assessment conducted in July 2013.

Methodology The baseline (February 2013) and midline (July 2013) data collection was done in order to assist in rigorously identifying the effects of the TiANA project and includes treatment plus two levels of comparison schools:

1. Ten treatment schools (TiANA) from St Anthonys’ and Namadidi education zones in TA Mlumbe;

2. Ten first level comparison schools in the current SCI’s Literacy Boost (LB) intervention schools within Zomba in Chimwalira education zone in TA Chikowi;

Page 7: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

5

3. Ten ordinary public schools in St Michaels’ Zone in TA Mwambo within Zomba where there are no any SCI interventions.

These groups allow a comparison of TiANA versus Literacy Boost, and the normal approaches of teaching reading used in the public schools in Zomba. TiANA will use this assessment to identify gaps and measure improvements in reading skills.

The sample at baseline assessment encompassed 600 standard 3 learners, divided between 10 schools in the TiANA intervention area (n of learners = 200) and 20 comparison schools (n of learners = 400) split into Literacy Boost and comparison schools. At each school 20 (10 boys and 10 girls) learners in standard 3 were randomly selected. Curriculum-based assessments in English and Chichewa which were developed in partnership with the Malawi Institute of Education and Ministry of Education Science and Technology were then administered among the randomly selected learners.. In addition, children answered questions about their lives and homes.

The key research questions explored in this report include:

1. Who is in school at the time of the midline? 2. What has changed in the skill profile of TiANA learners across the study period? 3. How much learning progress has each group made? 4. What factors influenced greater progress in this school year and may be levers for

change in the coming year? 5. What progress have teachers made, and what challenges remain? 6. How do communities support these efforts, and what challenges remain?

By addressing these questions, this analysis and report aims to inform TiANA and Literacy Boost implementation improvements in 2013-2014.

Measurement School profile data including information on enrolment, class size and teacher background was collected through a survey of school head teachers at every school in the sample. . Teacher interviews and classroom observations of both English and Chichewa lessons were conducted. For the learners’ assessment, all learners in the sample were asked about their background characteristics (age, household possessions, household building materials, etc.) and literacy environment (availability of material, reading habits, etc.). After collecting this background data, all learners were given an emergent literacy test in both Chichewa and English composed of six components administered through five sub-tests: writing, letter awareness, single word recognition (reading of most used words), reading fluency & accuracy (words per minute read correctly and total percentage of passage words read correctly; within the same sub-test), and a set of comprehension questions linked to the fluency & accuracy passage. All instructions were given in Chichewa and children were assessed on word writing, letter identification, most used words, reading fluency, reading accuracy, and reading comprehension.

Page 8: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

6

Who is still in school? In February 2013, the TiANA team collected data on 200 children in each of the three groups: Comparison, Literacy Boost and TiANA impact area. During the July 2013 assessment, the project found all but 90 children in comparison schools, all but 65 children in Literacy Boost intervention area and all but 30 children in TiANA impact area. Figure 1 shows how different these samples were at end line.

Differences between the children who were present and those absent, reveal that for the Comparison schools children who are missing had significantly lower baseline Chichewa reading skill scores (in four of five skills) than those who were present. In addition, they had a significantly weaker home literacy environment. For the LB intervention area school children who are missing had significantly lower baseline English reading skill scores (in two of five skills) than those present. On the contrary, children from TiANA project area who are missing had significantly poorer and weaker home literacy environments than those present.

The most likely reasons for differences among the three school groups in percentage of children missing during the end line assessment could be that the number of teachers and learners in the areas is not the same, making teacher-learner ratios quite different. TiANA project area has a teacher to learner ratio of 1 to 74; Literacy Boost intervention area has a teacher to learner ratio of 1 to 110; while the comparison schools have a teacher to learner ratio of 1 to 104. Larger class sizes may also have contributed to lower attendance of learners. In addition, the area with highest absenteeism rate is along Lake Chirwa, Malawi’s second largest lake, where a lot of money-earning opportunities such as fishing and rice growing take place. This, coupled with the fact that data collection was done in July which is the peak of rice harvesting, could have contributed to the high absenteeism in this area. In 2014, data collection will be done in June to avoid absenteeism due to rice harvesting peak period. Further, the comparison schools do not have education projects while TiANA and Literacy Boost area have innovations that motivate learners to be interested in schooling. The fact that all ten of the comparison schools are in the lake region where there are competing options for children’s time but an absence of interventions for better quality schooling gives reasons as to why this group faced a double challenge and, therefore, the lowest sample presence is not surprising.

Average baseline scores of children present in school at midline, are only different between groups for letter knowledge: TiANA project area learners have significantly lower scores on average, knowing only 11 letters out of 26; while learners in LB and comparison areas know 14 out of 26 letters, on average. While this demonstrates that the learners present were

55%

68%

85%

0%10%20%30%40%50%60%70%80%90%

Comparison Literacy Boost TiANA

Figure 1. Percent of children present at end of year assessment by group

Page 9: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

7

comparable in most skills at baseline, it has been noted that the learners were not similar on all counts. All comparisons must be interpreted in light of the trends above: the lowest Chichewa scoring and lowest home literacy environment (HLE) comparison school learners, the lowest English scoring LB learners and the poorest and lowest HLE TiANA learners are missing. This, , therefore, makes it difficult for the project to know how they would have performed. Thus, the first recommendation of this report echoes that of Silver (2012) to look at attendance more carefully – especially among these missing groups – and work to keep children consistently in school. In addition, whether in treatment or comparison schools, every effort should be made at the end of the school year to find all children in the sample and assess their skills.

Baseline and Midline TiANA Project Profile Within TiANA schools, the children assessed in both February and July demonstrate progress across all reading skills – both in Chichewa (Figure 2) and in English (Figure 3).

While the learners are headed towards letter mastery, have doubled the number of words they can write, and read many single words common in their curriculum, they are not reading texts fluently to support full comprehension. In fact, 47 learners, or 27 percent of the remaining TiANA project sample did not read any single word of the Chichewa text correctly. While this improves upon the 47% of MTPDS Level 3, Standard 4 learners were still not able to read at endline, therefore, moving from letters and words to text should be an important focus for next year’s work in these schools.

As expected in a second language, English reading progress lags behind. Considering the Figures 2 and 3 together, greater gains were made in Chichewa than in English. Chichewa comprehension is hovering just above 50%, and very little progress was made in English comprehension. This, therefore, means that a lot of work has to be done to realize the goal of reading with comprehension. It has been observed that teachers are not always comfortable teaching in English, therefore, the basic education programme introduced content knowledge

42.3

6.1

29.2

10.1

33.0 36.0 33.3

69.2

12.1

64.2

21.6

60.4

42.0

53.9

0

10

20

30

40

50

60

70

80

letters (%) words written (#) most used words(%)

fluency (wpmc) accuracy (%) prediction (%) comprehension(%)

Figure 2. TiANA Project Chichewa reading skills profile: baseline and midline

baseline

endline

Page 10: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

8

support for teachers to be assisted academically in English through the long term sponsorship programme.

Average midline status among the remaining children in TiANA project, Literacy Boost and comparison schools is similar across all the skills in both languages. There are no differences in treatment versus comparison groups. Without the 45% of the comparison group that is missing, it is impossible to interpret this lack of difference. Progress in each skill is, therefore, compared among the learners who were present in school in the three comparison groups.

Average Gains in Learning To generate gain scores, baseline scores are subtracted from midline scores. The resulting score represents the learning progress of the learner during the study period, in this case five months. Table 1 presents average gains for boys and girls in each reading skill for both languages and by study group. The only significant gender differences are in LB schools where girls made significantly greater progress than boys in Chichewa word reading and fluency.

Table 1. Average reading skills gains by language, sex and group

Chichewa Boys Girls English Boys Girls

Gain in Number of Words Written

Comparison 4.18 5.07 Comparison 1.95 1.38

Literacy Boost 4.87 4.71 Literacy Boost 1.12 0.55

TiANA 7.04 4.94 TiANA 2.74 3.06

Percentage Point Gain in Words Read

Comparison 31.96 28.33 Comparison 17.62 15.82

Literacy Boost 25.4 43.28 Literacy Boost 13.61 13.07

1.9

10.6

3.6 5.9

26.0

4.1 4.8

31.9

9.9

26.0

31.4

9.0

0

5

10

15

20

25

30

35

words written(#)

most used words(%)

fluency (wpmc) accuracy (%) prediction (%) comprehension(%)

Figure 3. TiANA Project English reading skills profile: baseline and midline

baseline

endline

Page 11: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

9

TiANA 30.07 39.74 TiANA 21.23 21.36

Fluency Gain (words per minute correct)

Comparison 2.35 3.96 Comparison 6.56 5.38

Literacy Boost 6.62 12.4 Literacy Boost 4.13 5.84

TiANA 11.05 11.84 TiANA 7.71 5.05

Percentage Point Accuracy Gain

Comparison 24.65 23.4 Comparison 20.84 18.69

Literacy Boost 20.46 19.4 Literacy Boost 19.94 18.84

TiANA 22.5 32.1 TiANA 18.59 21.56

Percentage Point Comprehension Gain

Comparison 18.18 20.91 Comparison 8.18 3.64

Literacy Boost 16.04 20.59 Literacy Boost 1.12 0.74

TiANA 14.94 18.97 TiANA 3.05 4.89

The overall average gains among the learners are difficult to judge. The only national fluency data available comes from the 2010 National EGRA in Chichewa in which Standard 4 learners read 11 words per minute correctly on average and the more recent MTPDS program evaluation in which Level 3 Standard 2 learners read 9 words per minute correctly on average at endline. As TiANA project standard 3 learners still in school in this study gained 11.5 words per minute on average during the five month study period and read an average 21.6 words per minute correctly (see Figure 2), this seems a subjectively favorable outcome. Similarly, LB learners gained 9.5 words per minute on average and read 19.9 words per minute correctly. Comprehension scores too compare favorably to the 2010 and MTPDS scores in this manner. However, these comparisons, as those noted above are affected by the attrition from the samples and so no attribution to the TiANA project or LB initiatives can be made.

Equity in gains Using regression analysis, the project considered three dimensions of equity (gender, socioeconomic status, and home literacy environment) and how children at risk fared in the three comparison groups:. Home Literacy Environment (HLE) encompasses both the availability of reading materials in the home as well as how those materials are used to engage the child in reading and learning. Five dimensions of the home literacy environment relate to reading achievement in children: value placed on literacy, press for achievement, availability and use of reading materials, reading with children, and opportunities for verbal interaction. The LB Assessment operationalizes these dimensions into questions for children and synthesizes their responses into an HLE index.

Page 12: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

10

Fitting interactions with group membership, it was found that while there were no discernable differences in the effect of being in a group on girls vs. boys or poorer vs. better off children, there was a trend across English skills of impact for children in weaker vs. stronger home literacy environments. Figure 4 shows this finding and offers further insight into the average gains presented in Table 1.

Underlying the average gains in TiANA schools are similar gains for low HLE (red) and high HLE (yellow), while in the other schools, these differences are significantly different. The high HLE learners (blue) gained greatly – even above TiANA project average gains for both groups – while the low HLE gain significantly less. This pattern also exists in English reading fluency and Chichewa writing. Its presence across these several skills suggests that TiANA project is helping all children still present at endline equally well – a strength to build upon while exploring how to ensure that the missing poorest and lowest HLE learners are in school at the end of next session so that we can continue to learn more. The absence of this pattern in Chichewa reading skills is curious – indicating that TiANA project could be doing something differently in English than in Chichewa.

Learning and use of reading skills in Zomba As relevance and use of reading skills relates to the amount of practice and progress that learners make in mastering various component skills, this section reports on whether and how the data show changes in literacy practices, and how those changes relate to learning.

11.3% 11.3% 11.3% 11.3% 6.6% 6.6% 6.6% 6.6%

22.1%

20.4%

19.8%

19.8% 10.1% 13.0%

24.1% 30.2%

0%

5%

10%

15%

20%

25%

30%

35%

40%

TiANA lowHLE

TiANA highHLE

ComparisonLow HLE

Comparisonhigh HLE

TiANA lowHLE

TiANA highHLE

ComparisonLow HLE

Comparisonhigh HLE

English word reading English accuracy

perc

enta

ge p

oint

s

Figure 4. Reading English single words and text by HLE and group

baseline TiANA low HLE gain TiANA high HLE gain Comparison Low HLE gain Comparison high HLE gain

71%

35%

10%

12%

5%

22%

6%

4%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

baseline (n=382) midline (n=393)

Figure 5. Use of Reading Skills at baseline and midline

use in the kitchen

use in market

use at church/mosque

do not use

Page 13: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

11

When asked at baseline whether they use their reading skills, 71 percent of learners answered no. Among those who answered yes, many gave further detail about where they use them: in churches/mosques, kitchens and markets. Figure 5 shows that at midline, only 35 percent of these same learners answered no, and many more report using these skills at the market, while church/mosque and kitchen remain common locations of use of reading skills.

Whether this is a reflection of the awareness of relevance of skills or an actual use is not known, but either represents a shift in thinking among these learners. The midline data suggest that reading has increasing social relevance too. At baseline, 61.8 percent of learners reported having a friend they read with; at midline among the same group of learners, 75.6 percent reported having a friend with whom they read. Further, 26 percent reported helping others to or with reading at baseline, and by midline, 45.1 percent are doing so, including helping other children read or reading things for parents, etc. Finally, while participation in reading camps increased from 20.1 percent at baseline to 37.8 percent at midline, this seems low in a sample in which 67 percent of learners are in schools in which reading camps are promoted to support reading.

Our home literacy environment index denotes a measure of materials and interactions in the home to support reading. Their values at midline are highly related to Chichewa skills and basic English skills as well. But the notions of use and relevance reported above go beyond the home, so using these three indicators helps to broaden the notion to form a community literacy index. This index combines the data on the use of skills, assisting others with reading, and reading with friends. The community literacy index score at midline significantly predicts all skill gains controlling for baseline score, age, sex, SES, HLE, and workload with the exception of gains in letters (see appendix B for full models). Figure 6 shows this trend for reading single words, accuracy and fluency across both target languages.

32.0

11.3

35.5

12.2 3.4 6.6

34.4

7.3

20.7

12.7

4.1

16.2

5.5

3.9

8.9

7.0

2.4

6.4

5.5

3.9

8.9

7.0

2.4

6.4

0

10

20

30

40

50

60

70

80

single words(%)

fluency(wpmc)

accuracy(%)

single words(%)

fluency(wpmc)

accuracy(%)

Chichewa English

Figure 6. Community literacy environment (CLE) predicts skill gains

gain CLE=3

gain CLE=2

gain CLE=1

baseline

Page 14: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

12

Given that TiANA project and LB initiative aim to affect the home and community literacy environment, it is of interest to see whether change in these indicators is related to learning.

Change in home literacy environment Regression models revealed emerging patterns between change in HLE and learning, but nothing strong and consistent enough to present for interpretation. Whether this is due to the sample attrition, the short time of intervention or measurement error is not known, but efforts to collect data from all learners over the course of the full 2013-2014 school year of implementation will assist the project to learn more at the end of the program.

Change in community literacy environment Regression models considering change in community literacy environment, found that LB learners reporting such changes had significantly greater learning than learners in comparison groups, controlling for age, sex, initial score, SES and workload. Figure 7 shows this pattern for letters and Chichewa single word reading, while it also exists for Chichewa accuracy. If a learner in LB school began doing one (red) or two (blue) community literacy activities during the intervention period, then their predicted gain is significantly higher than the predicted gains of learners in either TiANA project or comparison schools who did so – though their gains were higher than those who did not begin such activities.

More generally across the sample, regression models link change in CLE with Chichewa and English accuracy (see Appendix C for full models). While there is hope to see this change in use and relevance of reading and its relationship to greater progress in learning in both LB and TiANA project schools, the challenges of attrition and a relatively short implementation period between baseline and midline might preclude this. The project hopes to learn more about CLE and its change associated with learning at the project endline.

52.1% 52.1% 52.1% 52.1%

32.0% 32.0% 32.0% 32.0%

22.3% 19.2%

39.2% 38.0%

26.1% 17.1%

48.4% 37.0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Literacy Boost Comparisons Literacy Boost Comparisons Literacy Boost Comparisons Literacy Boost Comparisons

letters single words

Figure 7. Change in Community Learning Environment predicts skill gains

baseline gain 1 CLE change gain 2 CLE change

Page 15: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

13

Teachers and Schools At midline, 30 teachers were interviewed and observed teaching English and Chichewa lessons. All but one spoke Chichewa at home and all but three had Malawi School Certificate of Education, or 12 years of education from primary to secondary final year; the other three had a Junior Certificate of Education that is Form 2 or 10 years of schooling and were working in TiANA project area (2 teachers) and LB schools (1 teacher).

The teachers had an average age of 31 years and had been teaching for seven years on average. Each group included teachers as young as 20 years old with 1 year of experience and those with a decade of experience as well as more than two decades of teaching. Although there are no differences among the school groups in these teacher characteristics, there are fewer female teachers in comparison schools (27%) than in LB schools (36%) or TiANA project schools (39%).

School-wise, an enrollment and attendance comparison shows that although the standard 3 enrollment is of similar size, there is a gap in attendance by school as seen in Table 2 that is mirrored in the assessment sample.

Table 2. Third grade enrolment versus attendance on day of assessment

Group Total on roll Present % Present

Comparison Schools 1,664 770 46

Literacy Boost Schools 1,417 876 62

TiANA Project Schools

1,429 1,219 85

Total 4,510 2,865 64

As noted previously, the LB and comparison school enrollment is much higher (14,102 and 13,622, respectively) than that of the TiANA project schools.

Teaching strengths and challenges The teacher observation instrument was split into two sections of Lesson Planning and Preparation and Lesson Delivery. Using this instrument, thirty standard 3 teachers from the sample were observed by trained observers who included Primary Education Advisers, Teacher Trainer from teacher Training Colleges and Save the Children Education staff. A variety of skills in English, and Chichewa reading instruction were observed and rated using the following scale: 0 = not done, 1 = Poor, 2 = Fair, 3 = Good and 4 = Outstanding. Items observed and rated using this scale included the following lesson preparation and lesson delivery areas:

• Suitability, relevance, liveliness of introduction, • Ability to cope with individual differences, • Degree of learners' participation and interaction, • Appropriate use of language, • Effectiveness of methods used,

Page 16: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

14

• Use of teaching and learning resources, • Logical development of the lesson, • Use of continuous assessment, • Class control and organization for learners activities, • Teacher – learner and Learner-learner relationship.

Ten teachers in each group of schools were observed. Overall, Figures 8 and 9 shows that the teaching in Chichewa and English classes improved during this short period. In each of the identified areas, where the average teaching observed was poor to fair at baseline, some progress was made. The greatest progress in Chichewa teaching in Figure 8 is seen in learners asking questions to the teachers, although this remains the greatest challenge and may not be surprising in a classroom with nearly 100 learners. The least progress was made in classroom displays, although locally available resource use is up between fair to good on average.

In teaching English, in Figure 9, there was progress in the short intervention period as well, and interestingly, the greatest progress was made in the use of locally available resources. The average teacher is headed towards doing this well – which should be of assistance as all the learners take up this second language.

0

0.5

1

1.5

2

2.5

Learners initiatedialogue among

themselves

Learners initiatedialogue withthe teacher

Learners askquestions to the

teacher

Use of locallyavailable

resources

Classroomdisplays

1.6

1.2

0.4

1.7 1.7

2.0

1.6

1.3

2.4

2.1

Figure 8. Revisiting Baseline Areas of Teaching Challenges -- Chichewa

Baseline Midline

Page 17: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

15

Again the weakest area observed was learners asking questions, an area worth following up with PEAs, mentors and SCI staff supporting teachers. Table 3 shows average group scores for eight challenging areas of teaching across both languages. These areas were identified by having overall average scores of lower than two (fair).

Table 3. Average Midline Observation Scores for Challenging Areas of Teaching by Language and Group

Chichewa

Schemes of Work Quality

Records of Work Quality

Attendance Registers

Availability

Attendance Registers Upkeep

Continuous Assessment

Records Availability

Upkeep of Continuous Assessment

Records

Learners initiate

dialogue w/teacher

Learners ask

teacher questions

TiANA 1.9 1.6 2.2 1.9 2.1 1.8 1.9 1.2

LB 2.7 2.4 2.6 2.1 2.2 1.9 1.7 2.0

Control 1.7 1.4 1.5 1.1 1.2 0.9 1.3 0.7

English

TiANA 2.1 1.6 2.2 2.0 2.0 2.0 1.7 1.5

LB 2.7 2.6 2.9 2.6 2.1 1.8 2.0 1.2

Control 1.7 1.4 1.8 1.4 1.7 1.5 0.9 0.4

It is interesting to note that of these eight, six relate to preparation, not delivery. The project should consider this an artifact of understaffing. With too few teachers to cover classes,

0

0.5

1

1.5

2

2.5

3

Learners initiatedialogue among

themselves

Learners initiatedialogue withthe teacher

Learners askquestions to the

teacher

Use of locallyavailable

resources

Classroomdisplays

1.2 1.1

0.7

1.5 1.7

2.0

1.5

1.0

2.7

2.1

Figure 9. Revisiting Baseline Areas of Teaching Challenges -- English

Baseline Midline

Page 18: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

16

teachers have to prepare for more than one class and this affects their preparation and these six areas in particular.

The observation data also make it clear that LB teachers outperform their counterparts on average. Given that LB has been supporting these schools since 2010 and an estimated 75 percent of teachers in these schools have participated in trainings prior to the January-July 2013 time period, this is not surprising. However, it is also important to note teacher mobility is high in this area and only half of the teachers observed at baseline were also observed at midline due to this mobility. Thus, while the data suggest areas for attention, the findings cannot be attributed to change in teaching due to LB or TiANA project.

Focus on Lesson Delivery In areas of lesson delivery, there were four patterns evident at midline. First, on average, teachers everywhere were observed to do three things well: praise and correct behavior in a friendly manner, present subject matter logically, and phrase questions clearly. Second, teachers in TiANA and LB schools did four things well, on average, while comparison teachers performed fairly on average by making the introduction brief, lively, and including activities for learners, providing individual help during lessons, using situations/learners’ experiences to motivate them to learn, and simplifying the content to the level of the learners. Third, comparison school teachers observed for English lessons were fair on introduction, relevant to new subject matter, use of varied methods (i.e. pair work, group work, role play, songs and demonstration), use of varied questions (who, where, how, why etc), and distribution of questions randomly. These elements were good on average in their Chichewa lessons and in the lessons of all TiANA and LB teachers. Finally, outstanding teaching was apparent in several places:

• LB English: rewarding learners’ efforts (answering, asking questions and participating) • TiANA project Chichewa: Displays competence in subject matter that is being taught • TiANA project English: Use of question - pause – name

While these findings offer insight into progress made, they also highlight the challenges ahead in 2014. Given the identification of strengths herein, program improvement might be ignited via a discussion of what this good practice looks like, even capturing it on video, and opening discussion of how to enable more teachers to use these methods.

Community Strengths and Challenges A total of 214 community leaders were interviewed, including School Management Committee members, PTA Executive Committee members, Village heads, and Selected Parents. Table 4 shows the gender breakdown of this leadership and reveals a bias in comparison schools towards male participation in the interviews.

Table 4. Community interview participation by group and sex

Group Men Women Total

Page 19: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

17

Literacy Boost 37 34 71

Comparison 44 19 63

TiANA 45 35 80

Total 126 88 214

The project has conducted many gender sensitization efforts in the schools it supports and this ensures that more and more women ladies are involved as is apparent in Table 4.

All Community Leaders in LB and 90% of community leaders in TiANA project schools reported talking to children and parents about dropping out from school, whereas only 60% do so in Comparison schools. They do this primarily through following up of learners through parents with the help of local leaders (43%) and encouraging the learners to go back to school (37%).

Among parents, all those interviewed in TiANA and LB schools reported assisting in teaching and learning of their children while only 40% of comparison parents do so. Parents reported supporting their children by: checking their work after school (30%), sending learners to reading camps (17%), borrowing books for their children (10%), asking/helping them to read (13%), and buying school materials for them (3%). Interestingly, there appears to be a comparison school with a reading camp – a development worth following up!

All SMCs monitor teaching and learning in the reading camps in TiANA and LB schools by looking into camp leaders’ attendance, learners’ attendance, checking availability of reading materials, and monitoring how lessons are conducted in the camps. As a result of these monitoring visits, it was noted that:

• Many learners were at least able to read better than before the camps were introduced • Many children were interested to go to the reading camps • School Management Committees requested parents to produce reading materials

Overall, changes were observed by all respondents in LB and 90% of TiANA project schools. These included: a change in number of children able to read, an increased number of children having interest in going to the reading camps, and an increased number of children reading fluently since the introduction of the camps.

Youth Literacy Movement (YLM) Youth Literacy Movement (YLM) is a group of 10 school leavers (boys and girls) per school. The YLM works alongside Bright Future Committee (BFCs) also a grouping of parents, SMC and PTA members, village heads and learners themselves who help m o n i t o r r e a d i n g c a m p a c t i v i t i e s a n d a l s o h e l p i n t h e p r o d u c t i o n o f low cost teaching and learning materials to be used by children. YLM presence is strongest in TiANA project and known in 8 of 10 LB schools as well. The youth are charged with the responsibility of encouraging children to go to reading camps, helping develop reading materials for the camps, helping supervise the reading camps, and mobilizing parents to support reading camps. While data was not collected on the extent and effectiveness of their work at midline, this could be an interesting topic for exploration at end line.

Page 20: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

18

Conclusions There is still a great deal of ground to cover in the 2013-2014 school year. The data collection at the beginning and end of the school year will span a longer period and offer more opportunity to see the impact of the intervention. The following recommendations come from the analysis above:

• Collect data in June 2014 to avoid peak rice harvest season in the comparison area of around Lake Chirwa.

• Make every effort to gather data on all learners, regardless of enrollment status. • Review attendance records carefully and work to keep children consistently in school,

paying particular attention to: o How TiANA project can better support lowest HLE and poorest children in

school. o How LB schools can better support low English achievers.

• Redouble efforts to take Standard 3 learners beyond words reading to tackle texts and reading with comprehension.

• Promote the use of reading skills, the assistance of others as well as reading with friends to see if the project can generate greater gains associated with these changes in practice so far.

• Support teachers in areas of challenges identified at baseline that are still in need of attention as well as identified weaknesses at midline.

• Build on and spread observed teaching strengths across the system of schools supported.

• Reinforce community inputs to learning as well as attendance, and investigate the comparison school with a reading camp.

• Consider how to monitor and adapt the work of the youth thus far and integrate them into the data collection at end line.

References

Silver R. (2012): “A Qualitative Analysis of Dropout, Absenteeism, and Learner Engagement in Malawian Primary Schools” Not published Malawi Teacher Professional Development Support (MTPDS) Program (2010): “2010 Early Grade Reading Assessment”

Page 21: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

19

Appendix A: Equity models

(2) (8) (9) (10)

VARIABLES Chichewa writing gain

English word reading gain

English fluency gain

English accuracy gain

TiANA * HLE quintiles (baseline) -0.737* -4.085** -2.231 -4.451*

(0.319) (1.147) (1.101) (1.713)

TIANA 3.076* 14.11* 5.906 11.86*

(1.339) (5.558) (3.348) (5.510)

LB -0.0167 -3.622 -0.842 0.0237

(1.281) (3.071) (2.913) (4.537)

HLE quintiles (baseline) 0.646* 3.496*** 2.206** 4.302***

(0.238) (0.701) (0.792) (1.171)

age -0.218 0.532 0.218 -0.360

(0.193) (0.893) (0.305) (0.809)

SES quintiles (baseline) -0.107 0.497 -0.196 0.303

(0.253) (1.004) (0.481) (0.881)

Sex -0.714 -0.109 -0.485 0.755

(0.815) (2.135) (1.150) (2.688)

Baseline Chichewa writing -0.359***

(0.0910)

Baseline English word reading

-0.163**

(0.0537)

Baseline English fluency

-0.105

(0.131)

Baseline English accuracy

0.466***

(0.106)

Constant 8.337** 3.432 -0.559 8.639

(2.536) (11.00) (3.937) (8.667)

Observations 407 407 407 407 R-squared 0.081 0.074 0.052 0.108 r2_a 0.0623 0.0551 0.0333 0.0899 Robust standard errors in parentheses *** p<0.001, ** p<0.01, * p<0.05

Page 22: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

20

Appendix B: Community Literacy Models -- Chichewa

VARIABLES

Chichewa writing gain

Chichewa single word gain

Chichewa fluency gain

Chichewa accuracy gain

Chichewa comprehension gain

community literacy environment (of 3)

1.126** 5.467* 3.872*** 8.928*** -8.571*** (0.319) (2.234) (0.895) (2.292) (2.067)

SES quintiles (baseline) -0.105 -1.490 0.448 -1.260 -0.287 (0.230) (1.021) (0.482) (1.681) (1.604)

HLE quintiles (baseline) 0.166 1.528 0.634 2.571 1.856 (0.191) (0.894) (0.599) (1.367) (0.946)

Sex -0.483 10.63* 3.444* 6.892 3.740 (0.838) (4.355) (1.591) (4.014) (4.100)

Number of chores -0.633 -7.100** -2.178 -6.416* -4.112 (0.415) (2.342) (1.166) (2.905) (2.924)

Pupil Age -0.192 0.606 -0.259 -0.849 -0.295 (0.216) (0.897) (0.500) (1.167) (1.242)

Baseline Chichewa writing

-0.411*** (0.0834) Baseline Chichewa word

reading -0.591***

(0.0456) Baseline Chichewa

fluency -0.476***

(0.0916) Baseline Chichewa

accuracy -0.521***

(0.0654)

Baseline Chichewa comprehension

-0.537***

(0.0669)

Constant

9.375** 46.73*** 10.13 43.81** 69.79*** (3.264) (11.29) (6.890) (15.10) (16.43)

Observations 407 407 407 406 588 R-squared 0.095 0.360 0.256 0.278 0.227 Robust standard errors in parentheses *** p<0.001, ** p<0.01, * p<0.05

Page 23: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

21

Appendix c: Community Literacy Models -- English

VARIABLES

English writing gain

English single word gain

English fluency gain

English accuracy gain

English comprehension gain

community literacy environment (of 3)

0.907*** 6.924*** 2.389** 6.326*** 3.050** (0.168) (1.179) (0.704) (1.464) (1.038)

SES quintiles (baseline)

0.118 0.485 -0.178 0.335 0.570 (0.173) (0.776) (0.449) (0.778) (0.634)

HLE quintiles (baseline)

0.437* 0.777 0.935 1.552 0.688 (0.168) (0.913) (0.599) (0.998) (0.622)

Sex -0.101 0.901 0.0193 1.923 -1.139 (0.371) (2.051) (1.103) (2.573) (1.856)

Number of chores 0.287 -2.737 -1.555* -3.252 -0.692 (0.193) (1.725) (0.705) (1.751) (1.080)

Pupil Age -0.00445 0.605 0.252 -0.275 -0.355 (0.159) (0.791) (0.288) (0.714) (0.484)

Baseline English writing

-0.375* (0.158) Baseline English word

reading -0.220***

(0.0571) Baseline English

fluency -0.145

(0.132) Baseline English

accuracy 0.381**

(0.111) Baseline English

comprehension -0.553***

(0.133)

Constant -1.007 4.016 1.195 10.90 2.976 (2.117) (11.16) (4.243) (9.738) (7.425)

Observations 406 407 407 407 407

R-squared 0.114 0.129 0.078 0.158 0.148 Robust standard errors in parentheses *** p<0.001, ** p<0.01, * p<0.05

Page 24: SAVE THE CHILDREN INTERNATIONAL BASIC EDUCATION … · In January 2013, Save the Children extended the Basic Education programInternational interventions in Malawi through a project

22

Appendix D: Change in Community Literacy Models (1) (3) (5) (16) (21)

VARIABLES Letter gain

Chichewa Single word gain

Chichewa accuracy gain

Chichewa accuracy gain

English accuracy gain

Literacy Boost school * change in CLE 1.521* 10.14* 9.956~

(0.716) (4.478) (5.704) Literacy Boost school -0.703 -8.936 -16.24

(1.005) (7.013) (10.19)

+ change in community literacy environment -0.526 -1.010 2.615 5.887* 3.802*

(0.367) (2.659) (2.723) (2.723) (1.776)

Pupil Age 0.0668 0.580 -1.086 -1.298 -0.559

(0.253) (0.879) (1.059) (1.055) (0.813)

SES quintiles (baseline) -0.0145 -0.845 -0.250 -0.479 0.867

(0.224) (1.044) (1.792) (1.780) (0.862)

Sex 0.992 10.07* 5.385 5.258 0.867

(0.820) (4.353) (4.052) (3.984) (2.826)

number of chores -1.174* -6.872** -6.255* -6.044* -3.120

(0.499) (2.361) (2.919) (2.859) (1.993)

Baseline letter score -0.459***

(0.106)

Baseline Chichewa singe word reading score

-0.528***

(0.0387) Baseline Chichewa accuracy score

-0.432*** -0.444***

(0.0576) (0.0609) Baseline English accuracy score

0.596***

(0.108)

Constant 12.61** 56.92*** 62.68*** 60.47*** 22.81

(3.481) (11.73) (14.35) (14.07) (11.29)

Observations 406 407 406 406 407 R-squared 0.372 0.351 0.252 0.234 0.098 Robust standard errors in parentheses *** p<0.001, ** p<0.01, * p<0.05