world bank document...the final draft report. the 2011 timor-leste egma was implemented as a direct...

MINISTÉRIO DA EDUCAÇÃO

Timor Leste 2011 EGMA:Baseline Report

THE WORLD BANK

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

i

Timor Leste 2011 EGMA:

Baseline Report

ABSTRACT

In 2011, the Timor-Leste Ministry of Education, with assistance from the World Bank and Ausaid, conducted the first Early Grade Mathematics Assessment (EGMA) survey in Timor-Leste. More than 1200 students in 65 schools were surveyed in Grades One, Two and Three.

The analysis of the EGMA 2011 Timor-Leste baseline survey calls for an immediate response to safeguard the educational future of young Timorese students. Although students perform well in basic, “Phase Zero” mathematics skills such as oral counting and correspondence counting, “Phase One” skills (number identification) shows signs of slow gains. Later manipulative and calculation skills, i.e. those in “Phase Two” and “Phase Three” (quantity discrimination, missing numbers, word problems and arithmetic) are poorly understood and used by Timorese students. After three years of schooling, Grade Three students can only answer 46% of simple subtraction problems correctly on average and only 72% of addition problems -all of which should have been understood by the end of Grade One. The low ability of Timorese students to handle basic numeracy puts in doubt their ability to cope with an increasingly stringent curriculum in later years.

Language use was one of the most concerning aspects of mathematics education revealed by the survey. Although the main language of the classroom is Tetum, students’ mathematics textbooks are in Portuguese. Some students did not speak enough Tetum or Portuguese to complete the survey without aid of a translator for their local languages. This linguistic diversity within the classroom indicates that further research into how Timorese students learn and in which languages is required.

In terms of factors that showed positive association with early mathematic abilities, the participation in daily mathematics lessons, working with others in those lessons and doing homework were all associated with statistically significant improvements in mathematics outcomes at the 95% level.

ii

Timor Leste 2011 EGMA : Baseline Report

ii

iii

The 2011 Timor-Leste Early Grade Mathematics Assessment (EGMA) survey is the result of the intention of the Ministry of Education of Timor-Leste to improve numeracy skills of the students in the education system, with recognition that the early years of education is the key period for foundational learning.

The report was prepared by Steph de Silva (Econometrician, World Bank), Bronwyn McNamee (Education Specialist within the ESSP project) and Luc Gacougnolle (Education Economist, World Bank). As of 2011, Kashif Saeed, Simao do Rosario and members of the Ministry of Education EMIS team provided assistance in school sampling, data cleaning and data entry. The report benefitted from significant comments, editing and additions by Barbara Thornton (International Development Consultant) and Myrna Machuca-Sierra (Education Specialist, World Bank). Wendi Ralaingita of Research Triangle Institute (RTI) generously directed the authors to research and reports of other countries where EGMA has been implemented and Yasmin Sitabkhan (RTI) provided helpful review comments on the final draft report.

The 2011 Timor-Leste EGMA was implemented as a direct result of commitment, encouragement and support, as of 2011, from Joao Cancio Freitas (Former Minister of Education), Paulo Assis Belo (Former Vice Minister of Education), Domingos de Deus Maia (Former Director General of School Administration, Innovation and Curriculum Development), Raimundo Neto (Former Director of Curriculum and Assessment), Alfredo de Araujo (Director of Basic Education).

As of 2011, Filomena Sequeira (Directorate of Curriculum and Assessment), José Antonio Cardoso (Chief of Department of Assessment in the Directorate of Curriculum and Assessment), Fernando Mouzinho Gama (Directorate of Curriculum and Assessment) and Pedro Laurentino da Silva (ESSP advisor) formed the team for the initial adaptation and trial of the survey.

The authors would especially like to thank and commend the EGMA core team of trainers, mentors and monitors from the Directorate of Curriculum and Assessment. Regional and District education personnel, school directors and teachers are commended and thanked for their generous and willing cooperation in terms of liaising and making preparation for the school visits by the enumerators and EGMA mentors.

The authors also wish to commend and express gratitude to the enumerators and supervisors who carried out the survey implementation professionally and efficiently and within the set time frame. The geography of Timor-Leste presents many challenges for travel, particularly in remote areas. These people performed with a determination and cheerfulness that made the 2011 Timor-Leste EGMA possible: Afriano Lopo Nono, Baltazar Pereira, Belita Lopes, Bendito José Casimiro, Benita Pereira, Brigildo Frans Xavier, Carlito Paixao Neno, Casimiro Sousa de Araujo, Deodata Gomes, Felisberto Pereira, Fidelia de Fatima Soares Correia, Januario Afonso Amaral, Januario da Costa, Januario Nunes Viegas, Julio Conceicao de Araujo, Julio Ximenes Ratuk, Leoneto de Araujo Freitas, Lourdes Maria, Lourenca Lalisuk, Luis Martins, Luzinha Gomes, Manuela Esposta, Maria de Fatima Soares, Natalina de Sousa Pinta, Paulino de Araujo, Supriyati da Costa Abi and Virgilio de Araujo.

Finally and significantly, acknowledgement must be given to the 1226 Timorese students who willingly participated in the survey. Obrigado barak.

Acknowledgements

iv

Contents

Abstract ....................................................................................................................................................................................................................................iAcknowledgements .........................................................................................................................................................................................................iiiContents ...................................................................................................................................................................................................................................ivList Of Figures .......................................................................................................................................................................................................................vList Of Tables ..........................................................................................................................................................................................................................vExecutive Summary ..........................................................................................................................................................................................................vi

Background .....................................................................................................................................................................................................................viFactors Associated With Better Numeracy Outcomes ..........................................................................................................................ixRecommendations .....................................................................................................................................................................................................ix

Chapter 1: Introduction ..................................................................................................................................................................................................1Chapter 2: Context, Survey Design And Implementation ........................................................................................................................3

Early Mathematics Learning And Egma ........................................................................................................................................................4Chapter 3: Instrument Design, Sample Design And Execution .............................................................................................................7

Instrument Design And Adaptation ................................................................................................................................................................7Enumerator Selection And Responsibilities, Training And Field Work ........................................................................................8The Unexpected Consequences Of Working In The Field .................................................................................................................8Reliability Of The Instrument ................................................................................................................................................................................9Sample Design ..............................................................................................................................................................................................................10

Chapter 4: Sub-Test Results ..........................................................................................................................................................................................11Sub-Test One: Oral Counting ................................................................................................................................................................................11Sub-Test Two: Correspondence Counting ...................................................................................................................................................12Sub-Test Three: Number Identification ..........................................................................................................................................................13Sub-Test Four: Quantity Discrimination .........................................................................................................................................................14Sub-Test Five: Missing Numbers .........................................................................................................................................................................15Sub-Test Six: Word Problems ................................................................................................................................................................................16Sub-Test Seven: Addition ........................................................................................................................................................................................17Sub-Test Eight: Subtraction ...................................................................................................................................................................................18Conclusion .......................................................................................................................................................................................................................19

Chapter 5: Language And Mathematics ..............................................................................................................................................................21Languages Used By Enumerators .....................................................................................................................................................................22Language And Performance ................................................................................................................................................................................23

Chapter 6: Factors Associated With Better Numeracy Outcomes ........................................................................................................27Chapter 7: Conclusions And Recommendations ...........................................................................................................................................31

References: ..............................................................................................................................................................................................................................35Annex One: School And Student Specific Factors .........................................................................................................................................37Annex Two: Egma Instrument ....................................................................................................................................................................................51Annex Three: Word Problems Per-Item Analysis .............................................................................................................................................53

v

LIST OF FIGURES

Figure 1: Oral Counting ...................................................................................................................................................................................................11Figure 2: Sub-test 2, Correspondence Counting ............................................................................................................................................12Figure 3: Sub-test 3, Number Identification .......................................................................................................................................................13Figure 4: Sub-test 4, Quantity Discrimination ..................................................................................................................................................14Figure 5: Sub-test 5, Missing Numbers..................................................................................................................................................................15Figure 6: Sub-test Six, Word Problems ...................................................................................................................................................................16Figure 7: Sub-test 7, Addition ......................................................................................................................................................................................17Figure 8: Sub-test Eight: Subtraction ......................................................................................................................................................................18

LIST OF TABLES

Table 1: Timor Leste EGMA Survey and Corresponding Curriculum Outcomes Part One ....................................................viiTable 2: Timor Leste EGMA Survey and Corresponding Curriculum Outcomes Part Two ....................................................viiTable 3: Timor Leste EGMA Survey and Corresponding Curriculum Outcomes Part Three .................................................viiiTable 4: Timor Leste EGMA Survey and Corresponding Curriculum Outcomes Part Four ...................................................viiiTable 5 Main Education Indicators ...........................................................................................................................................................................4Table 6: Phases of Early Mathematics Development ....................................................................................................................................6Table 7: EGMA Sub-tests and Phases of Early Mathematics Learning ...............................................................................................7Table 8: Cronbach’s Alpha, Complete Sample ..................................................................................................................................................9Table 9: Cronbach’s Alpha, Sample with Zero-Scoring Students Removed ...................................................................................9Table 10: Correlation Between Sub-tests .............................................................................................................................................................10Table 11 Gender and Grade Distribution .............................................................................................................................................................10Table 12: Sub-test One, Oral Counting ..................................................................................................................................................................12Table 13: Sub-test Two, Correspondence Counting .....................................................................................................................................13Table 14: Sub-test Three, Number Identification ............................................................................................................................................14Table 15: Sub-test Four, Quantity Discrimination ...........................................................................................................................................15Table 16: Sub-test Five, Missing Numbers ...........................................................................................................................................................16Table 17: Sub-test Six, Word Problems ..................................................................................................................................................................17Table 18: Sub-test Seven, Addition ..........................................................................................................................................................................18Table 19: Sub-test Eight, Subtraction .....................................................................................................................................................................19Table 20: Frequency of Enumerator and Child Languages Used; Oral Counting ......................................................................22Table 21: Frequency of Enumerator and Child Languages Used; Correspondence Counting ..........................................22Table 22: Frequency of Enumerator and Child Languages Used; Numbers Identified ...........................................................22Table 23: Frequency of Enumerator and Child Languages Used; Words Problems ..................................................................22Table 24: Oral Counting Results by Language ..................................................................................................................................................24Table 25: Correspondence Counting Results by Language .....................................................................................................................24Table 26: Number Identification Results by Language ...............................................................................................................................25Table 27: Missing Numbers Results by Language .........................................................................................................................................25Table 28: Word Problem Results by Language ................................................................................................................................................25Table 29: School and Student-Specific Variables ...........................................................................................................................................28Table 30: Effects of School and Student-Specific Variables on Average Scores Part A ............................................................29Table 31: Effects of School and Student-Specific Variables on Average Scores Part B ............................................................29Table 32: Per-item Analysis, Sub-test 6 ..................................................................................................................................................................67

vi

Executive Summary

Background

In 2011, the Ministry of Education of Timor-Leste carried out an Early Grade Mathematics Assessment (EGMA) under the World Bank funded Education Sector Support Project1 and with technical support from the World Bank and and Australia’s Program for International Development (AusAID). The purpose of the survey was to set up a baseline for early mathematics skills in Timorese schools.

Overall, survey results show cause for concern. Although the most basic mathematics skills, oral counting and correspondence counting, show healthy levels of attainment in the grades tested; other skills, such as quantity discrimination and simple calculations lag significantly behind curriculum expectations. Tables 1-4 give a comparison between curriculum expectations in Timor-Leste and the average performance of students who participated in the EGMA survey.

Despite EGMA tasks are arranged as a series of sub-tests in approximately the order of skill required; this does not imply a pre-determined order as students acquire different skills at different rates. Overall, Timorese students demonstrated relatively good ability in the first two sub-tests in EGMA involving number counting (table 1). In oral counting –i.e. the ability to recite numbers in order from memory-- the average Grade 1 student was able to count to 57 in a minute, while Grade 2 students were able to count, on average, to 66 and Grade 3 students to 72. Although average Grade 1 results are above curriculum expectations, Grade 2 and 3 results fall behind. Timorese students also performed strongly in correspondence counting (i.e. counting objects individually). The average Grade 1 student counted 75 objects, Grade 2 students counted 93 objects on average, and Grade 3 students 96 objects. These results are either in line with curriculum expectations or exceed them. Timorese students demonstrated worrisome progress when asked to identify numbers (i.e. students are asked to recognize the printed form of a number ranging from a single digit to four digits) and to discriminate between quantities (i.e. students were required to identify the bigger of two numbers). On average, Grade 1 students were able to recognize 25% of the numbers presented. There is a substantial increase on performance in Grade 2 when students are able to recognize, on average, 65% of the numbers presented; and Grade 3 students are able to recognize 80% of the numbers shown. Student performance on quantity discrimination showed a similar pattern: on average, Grade 1 students identified the bigger of two numbers correctly 54% of the time, Grade 2 students 73% of the time and Grade 3 students 85% of the time.

The next skill level is the basic manipulative skills, which allow a student to use numbers to solve both abstract and real-world problems. Table 3 gives the results for these sub-tests. Grade 1 students were able to answer just 19% of the questions correctly in the sub-test on missing numbers, while Grade 2 and 3 students were able to answer only 36% and 50% correctly on average, respectively. Students fared slightly better with word problems with Grade 1 students scoring an average of 28% while Grade 2 students scored 48% and Grade 3 students 62%.

The final sub-set of skills is the skills necessary to make a beginning to formal calculation: basic arithmetic skills of addition and subtraction. Although not all of the number combinations tested were expected of Grade 1 at the time of testing (see Table 4 for details), results indicate students start from a low average base and progress slowly in this vital set of skills. Grade 1 students were able to answer only 21% of the addition and 12% of the subtraction questions correctly. Grade 2 students were only able to answer 53% and 30% of addition and subtraction questions correctly, while Grade 3 students answered just 72% and 46% correctly.

The highest proportion of autostop cases was found in the numeracy skills requiring higher level reasoning: word problems, addition and subtraction.

1 IDA Grant No. H583-TP

vii

TABLE 1. Timor Leste EGMA Survey and Corresponding Curriculum Outcomes Part One2

Mathematics Skill EGMA Task

Curriculum expectations (based on half-way through school year)

in relation to the tasksEGMA Average Results

Notes

Grade 1 Grade 2 Grade 3 Grade 1 Grade 2 Grade 3

Oral counting: Count, from memory, in sequence, starting from 1.

Recite the number names in order, starting at 1, in preferred language for counting. Child counts until he cannot count further, makes an error, or time expires.

Up to 20 Up to 100 Beyond 100

57 66 72 Highest number students were on average able to count

The child is then asked how many circles he counted. Timed (to 60 seconds) as a measure of fluency.

Rational Counting: Count showing 1:1 correspondence and knowing the last number said is the number that represents the total of the collection (the quantity counted

Point and count 30 circles in a 5X6 array.

Up to 20 Beyond 30 Beyond 30 75 93 96 Average number of items students were able to count without mistake.

Timed (to 60 seconds) as a measure of fluency. Stop when child can’t count further or at point of error, including counting one circle twice or two or more circles as one. After counting, the child is asked how many circles he/she counted.

TABLE 2. Timor Leste EGMA Survey and Corresponding Curriculum Outcomes Part Two




Notes


Number identification: Naming of numbers

Identify numbers ranging from single digit to 4 digits. Move to next question after 5 second pause. Stop after 3 consecutive errors. Timed (to 60) as a measure of fluency seconds.

Up to 20 Up to 100 Greater than 1000

35% 65% 80% Overall score

Quantity Discrimination: Make comparisons about lower/higher position in counting order or lesser/greater in quantity represented by the number.

Point to the number of the pair that is greater. Pairs of numbers range from single digit to 4 digits.

Up to 20 Up to 100 Greater than 1000


2 It is worth noting that some curriculum expectations are measured in different units to those in the EGMA survey. Comparisons here are not straightforward

viii


TABLE 3. Timor Leste EGMA Survey and Corresponding Curriculum Outcomes Part Three




Notes


Missing Numbers: Discern the pattern of progression from one number to another, in a sequence of numbers and determine the number that is missing from the sequence.

Tell the number that is missing from the sequence. Move to next question after 10 second pause. Stop after 3 consecutive errors.

Up to 20 Up to 100 Beyond 100


Relevant Questions

in sub-test:

1,2,3,6,8

Relevant Questions

in sub-test: 4,5,9

Relevant Questions

in sub-test: 7,10

Word Problems: Apply mathematics concepts and skills in real-world problems, presented orally.

Solve one step word problems involving addition or subtraction with numbers to 10.

Questions 1,2

Questions 3,4

Questions 3,4


TABLE 4. Leste EGMA Survey and Corresponding Curriculum Outcomes Part Four




Notes


Addition: Understand the meaning of and complete addition algorithms.

4+5 * 21% 53% 72% Overall score

8+2 *

20+4

13+12

11+9 *

Subtraction: Understand the meaning of and complete subtraction algorithms.

9-5 * 12% 30% 46% Overall score

10-8 *

24-4

25-13

20-9 *

Key: Student expected to process calculation* First half of Grade 1 year, orally presented only

ix

Executive Summary

Factors Associated with Better Numeracy Outcomes

A short socio-economic and classroom practices survey accompanied the EGMA survey. In it students were asked to self-report on a number of student- and classroom-specific factors. Several of these were associated with positive, significant outcomes on the EGMA survey:

• Students participating in daily mathematics lessons is associated with increases in average scores of between 8 and 17% depending on the sub-test. These sub-tests included number identification, missing numbers, addition and subtraction.

• Students working with others during lessons is associated with increases in average scores of between 6 and 8% depending on the sub-test. These sub-tests included number identification, missing numbers, word problems and subtraction.

• Students completing mathematics homework is associated with sub-test scores of between 12 and 23% higher depending on the sub-test. These sub-tests included number identification, missing numbers, word problems, addition and subtraction.

• The household ownership of a television was a relevant socio-economic indicator and was associated with average scores of around 7% higher than those who did not live in a household with a television. This was the case for the sub-tests on missing numbers and subtraction.

Language was found to be a substantial factor associated with performance but the evidence does not support it being considered a causative factor in performance. Students answering in Portuguese were associated with higher scores in some sub-tests and students answering in Bahasa Indonesian were associated with higher scores in others. Research into language practice in the classroom is needed before any conclusions can be drawn from this, particularly in light of the results from the EGRA survey findings on this same subject of language.

Overall, the short socio-economic survey was not able to offer substantial guidance on future interventions in part due to the age of the respondents and in part due to its short length. However, classroom practices such as a daily mathematics lesson, working with others and homework had significant, positive associations with mathematics outcomes. A television at home had a positive, significant association with some sub-test results, but rather than a causal link it is likely that it is an instrument for socio-economic status.

Recommendations

Survey results call for immediate action to ensure that all students in east Timor have access to a numerate future. Specific recommendations to be considered are as follows:

Language:• Research is urgently required into language practices in the classroom. Timor Leste is a linguistically

rich country and students answered some sub-tests in the EGMA survey in up to three languages, indicating a substantial exposure to language in their community. In order to understand how and why students are learning, it must be known in what language and combination of languages they are actually learning in, rather than supposed to be learning according to curriculum mandates and workbooks provided.

Conceptual:• Further research is recommended into the confidence and skills of teachers and students to use

effectively the current mathematics student workbooks. These are currently used as textbooks and in Portuguese language.

x


• Further research into students’ ability to use the workbooks is required. Based on the findings of the 2011 EGRA (Early Grade Reading Assessment) Survey, and the poor facility of students in the Portuguese language there is substantial cause for concern that students may not be able to use these materials effectively. The ownership of mathematics texts was only self-reported by students at around 80%, despite a national distribution and the ownership of the text was not associated with a statistically significant increase in performance. These results suggest that research into the use, distribution and implementation of these resources is warranted.

• The establishment of mathematics benchmarks to monitor classroom improvements. These benchmarks can be used to assess students at-risk of falling further behind by a process of continuous assessment. It is important to consider that initial standards should be considered temporary reference standards since not enough is currently known about when and at what rate Timorese children should progress in learning mathematics. In this sense, benchmark standards should not be seen as high-stakes but an essential piece to monitor mathematics progression in the classroom. One way of doing this would be to use the percentage of zero-score students in selected sub-tests as a marker and track reductions in the shares at least biannually. Monitoring achievements over time will eventually provide more information on the rate and the way in which average mathematics skill develops in Timor. A modified version of the test could be used to screen students during the school year in need of additional support.

Teaching , Teachers and the Classroom:• Support for implementing and sourcing mathematics resources in a students’ main language of

instruction is suggested. In the light of evidence that supports early learning in the child’s first language, it would follow that priority should be given to the production of mathematics textbooks/workbooks and stimulus pictures in mother tongue.

• It is recommended that training of teachers, both pre-service and in-service, be designed to improve pedagogical skills in deepening conceptual understandings of mathematics. It is recommended that Escola Basica Deputy Directors and school inspectors be involved in parallel professional development.

• It is highly recommended teachers receive continued ongoing support in using the stimulus, display and manipulative materials that were distributed to all schools in 2010 and were further augmented in 2012. Pedagogy that supports deep understanding through participation, manipulation and interaction is most effective for mathematics learning and teachers need to be supported in achieving this.

• Ongoing support in mathematics program planning, teaching methodology and formative assessment is essential. The future of Timorese numeracy is in the hands of its teachers. Improvements in Timorese children’s mathematical skill depends not only on classroom resources but on effective understanding of how to implement curriculum, resources and assessment.

• It is recommended that the Ministry should leverage on the national distribution of materials and the EGMA results to identify grade-specific skills that teachers will need pedagogical methods to convey content knowledge to students. Without the pedagogical understanding of how to convey content to students, knowledge transmission cannot occur efficiently.

• Research into the confidence and skills of teachers to interpret and teach according to the current mathematics curriculum and teacher guide for mathematics is required. Modified versions of the Classroom Observational Snapshot tool used in other Pacific countries and tailored to the specific needs of the Timorese education system will provide information on the skills of teachers, while teacher surveys will provide elucidation on the thoughts, concerns, backgrounds and skills of these vital components of the Timorese education system.

xi

• Curriculum and teacher support to provide further opportunities for students to work in groups with each other is recommended. Students who engaged in group activities for mathematics learning had scores significantly higher than those who did not. Opportunities to engage in this process provide the foundation not only for improving numerical skills, but gaining language and social skills that will benefit Timorese students. However, teachers need to be supported in their efforts to engage their students in meaningful group activities and this should be a priority in future curriculum and lesson planning.

• Teacher encouragement and support to engage in mathematics learning on a daily basis is required. Students who were engaged in mathematics on a daily basis had a significantly higher score in the sub-tests examined than those who did not. Mathematics is a cumulative skill set, each concept building on those learnt previously. Especially in early grades, students are learning patterns of thinking that will be a valuable life-long asset. A healthy 91% of students self-reported that they engaged in mathematics learning daily, however, that leaves a large proportion of students who do not and this statistic may over-report the reality. Ensuring that each child has the opportunity to build on their foundational skills in a consistent daily manner should be a focus of lesson planning and teacher training.

Parents and the Community:• Promoting parent education and involvement is needed, not only to interest parents in their children’s

mathematics education and build accountability into the Timorese education system at the village level. It is also needed to build parents’ confidence, extending their knowledge and strategies that will enable them to engage with and assist their young children in mathematics. Only a single student out of the 1226 surveyed reported that they received assistance with their mathematics homework. Parental involvement and engagement in mathematics is a key method of improving outcomes for Timorese students.

• The evidence in the report suggests that providing opportunities for students to engage in meaningful learning at home will result in benefit to students. Students who reported doing mathematics homework had scores significantly higher than their counterparts. However, this should not be construed as an argument for large amounts of rote-learning to be sent home with children on a regular basis. The importance of parental engagement has been outlined above, however when only a single student in more than 1200 reported having assistance with their homework expectations of Timorese parents must be realistic. On a related note, 39% of students reported that their parents sell things in the market. This statistic indicates that a basic level of numeracy is prevalent in the community and that the community understands the need for numeracy skills. Providing children with the opportunity to engage in active learning at home through meaningful homework activities will have several effects: to encourage and cement concepts learnt in school, to engage parents and the community with the importance of numeracy as a concept and to increase children’s’ confidence in their own mathematical abilities. Examples of such activities may include assisting parents at market or in shopping, making change, counting and arranging items in groups in the home or market, correlating and observing sports scores or engaging in word problems with the assistance of every day objects (a skill many Timorese students struggled with). Encouraging parents to involve their children with every day numeracy opportunities may help demystify mathematical knowledge.

Executive Summary

xii


1

In 2011, an Early Grade Mathematics Assessment (EGMA) was carried out in Timor Leste. Results are concerning and call for immediate action. Although student fluency in both oral and rational counting meets curriculum expectations in Grade 1 (though not in Grade 2 or 3), the more formal skills required for more complex numeracy such as quantity discrimination, addition and subtraction show very poor levels of attainment, although there is slow average progression as students advance by grade.

The report takes the following format. Chapter 2 will discuss the EGMA survey and the context of Timor Leste. Chapter 3 will discuss the implementation, design and sampling of the survey. Chapter 4 will present the results of the sub-tests and Chapter 5 will discuss the effect of language on student performance. Chapter 6 will analyze the factors associated with student outcomes. Chapter 7 concludes with recommendations for the future of mathematical education in Timor.

Chapter 1 Introduction

2


3

Chapter 2 Context, Survey Design and Implementation

A decade after the restoration of independence, Timor-Leste is no longer a post-conflict least-developed country. It is now considered a country with a medium human development standing of 120 of the 169 countries in the 2010 Global Human Development Report (Timor-Leste Human Development Report, 2011).

The rapid development of Timor’s economy and democracy is matched by the expansion of its education system: particularly in the early grades. Population growth is high and near-universal enrollment has been achieved.

Net enrolment figures in primary education have increased from 68% to more than 90% from 2004/05 to 2011. There were rapid reductions in grade-to-grade dropout from 11-12% in 2008-09 to 3-4% in 2010. As identified in key sector diagnostics such as the Early Grade Reading Assessment (EGRA) and the National Education Strategic Plan itself, the sector continues to face major challenges in the quality and efficiency of education, including continued high levels of repetition (see Table 5).

4


Timor Leste faces challenges in ensuring access to quality education to all children. Gross enrolment rates are above 100%, indicating that the country has the physical capacity to provide educational services to all children of school age. Meanwhile, net enrolment rates remain below 100%, pointing at part of the school age population that remains excluded. Other challenges facing the system include institutional capacity, efficiency, with high repetition and drop-out rates, and quality of education and learning outcomes.

To tackle these challenges, the government of Timor-Leste has developed and launched in November 2010 its National Education Strategic Plan (NESP). Amongst its key Priority Programs is “Improving Teaching Quality”, with the specific short-term objective of “Improving the quality of education by substantially increasing the quality of teaching”. However, as the NESP recognizes, the country lacks regular, standardized, practical tools to establish baselines and targets, assess progress and target interventions in this endeavour of better quality of education. In response to this, the Ministry of Education, with financial support from the Education For All Fast Track Initiative (EFA-FTI, now Global Partnership for Education), provided stimulus and manipulative materials for mathematics. These were targeted for supporting the teaching of base competencies in the mathematics curriculum in the early primary grades, were supplied to all primary schools. A further scaling up of these materials is taking place at present with the delivery period likely to take place in mid 2012.

It is in this context of an increased focus on monitoring of education quality in early grades, that the Ministry of Education conducted a first Early Grade Mathematics Assessment (EGMA) in 2011.

Early Mathematics Learning and EGMA

It is clear that not all children learn mathematics using the same progression or at the same pace. For the purposes of this report we are arranging some of the basic skills of early mathematics learning into four phases, each of which requiring further skill or knowledge and each of which a child will progress through individually. Children may not necessarily learn in the same process or order. Table 6 gives these four phases of early mathematics development.

TABLE 5. Main Education Indicators

2004-05 2005-06 2006-07 2007-08 2008-09 2010 2011

Primary

Total enrollment 157,516 169,384 189,398 206,476 218,674 230,496 238,936

Gross intake rate (%) 94.4 92.7 117.0 133.2 126.4 144.7 121.1

Gross enrollment rate (%) 99.6 102.7 109.5 113.8 116.1 127.4 129.1

Net enrollment rate (%) 68.0 70.7 75.0 80.3 84.6 93.0 94.3

Primary Completion rate (%) - - 86.3 92.7 80.2 77.8 76.9

Repetition rate (%) 15.0 13.5 13.9 19.4 17.6 17.9 -

Pre-Secondary

Total enrollmentNo reliable Data

60,610 60,618 61,270

Gross enrollment rate (%) 80.1 77.1 75.3

Total Basic Education


279,284 291,114 300,206


Secondary


22,874 28,379 29,409


Enrolment rates in 2010 and 2011 use 2010 Census data of school age children as denominator while earlier indicators are based on 2004 population projection. Source: Timor-Leste Ministry of Education EMIS 2011

5

Phase Zero we can describe as a nascent Phase of developing numeracy. Children learn rote, oral counting (e.g. starting from one and continuing in order). Children recognize patterns in the ordering of numbers. They then learn to apply this ordering of numbers to objects in correspondence counting: the ability to number objects individually without repetition, missing objects or numbers and to know how many objects there are in a group. Repeated practice, through song, through number games, through other classroom activities results in a child’s automaticity in these early concepts of mathematics: an important step if students are to apply them to later Phases of mathematics development. In this developmental Phase, students begin to recognize numerals in print and to identify them with meaning.

Using the fundamental skills learnt in Phase Zero, most children proceed on to the next phase. This is a discernment phase. Children begin to discern the differences between groups in a mathematical way: more and less, many and few, bigger and smaller. They begin to discern numerical patterns and anticipate them.

Phase Two is a manipulative phase. Children learn to manipulate mathematical concepts. Understanding the concept of numerical patterns, they learn to manipulate them and to solve simple problems in the context of their every day lives. Phase Three is when children develop the ability to make formal calculations using mathematical formulae.

The Early Grade Mathematics Assessment (EGMA) instrument was originally designed by the Research Triangle Institute (RTI) in 2008. The EGMA is an individual survey administered as an interview and is suitable for students whose reading abilities are not yet well-developed enough for them to sit a formal test. It gives information on accuracy and fluency of skills and is based on these earliest Phases of mathematical development and the foundational conceptual understandings, which are a topic of instruction in schools. The EGMA interview takes about 20 minutes per child to administer.

EGMA has been adapted and implemented in a number of developing countries with the aim of providing baseline data for mathematics achievement in the early grades. It requires adaptation in each country it is used to reflect the standards and contextual setting for the particular country in which it is to be implemented.

In order to determine the fluency with which a student was able to complete a task, a measure of their automaticity, some of the sub-tests were timed. The purpose of this is to determine the level at which a student has internalized the skills: an important component for performing higher order skills at later Phases of mathematical development. The timed data was intended to be used to compare students by “per minute” measures. However, there were unexpected problems in the field and this will be discussed in the next chapter.

The Timor Leste EGMA presented students with 8 tasks. Table 7 presents the tasks and corresponding Phases. The three tasks in Phase Zero were oral counting, correspondence counting and number identification. Oral counting required students to count from one as high as they could in one minute, in order. Students were stopped when they had made a mistake and were asked to count in the language they were most familiar with.

The second sub-test, correspondence counting, presented students with a page of circles, evenly spaced and ordered as a grid. Students were asked to count as many circles as they could in one minute. Students were asked to count in the language they were most familiar with and they were stopped when they made a mistake. The number identification sub-test (Sub-test 3) presented students with a page of numbers and asked to identify as many as they could in 60 seconds. They were stopped if they had three consecutive incorrect answers or paused for more than five seconds.

TABLE 6. Phases of Early Mathematics Development

Phase Name Key Skills

Phase Zero Development Counting: reciting numbers from memory, object counting, numeral recognition

Phase One Discernment Discernment between actual quantities (groups) and patterns

Phase Two Manipulation Ability to use numbers in every day life to solve simple problems

Phase Three Calculation Ability to use numbers to perform simple abstract calculations

Context, Survey Design and Implementation

6


The sub-tests corresponding to Phase One were Sub-tests 4 and 5. Sub-test four was untimed. Students were presented with ten sets of two numbers and asked to identify the bigger of the two in each case. Children had access to counters as visual and tactile stimuli, if they wished. The enumerator provided two examples on how to use them in the preamble to the sub-test.The fifth sub-test presented students with a series of numbers, one of which was missing. They were asked to identify the missing number. The sub-test was untimed and there were ten items in total.

Phase Two aimed to identify children’s skills in manipulating numbers. The sixth sub-test asked students a series of simple word problems pertaining to familiar manipulations in numbers they might make in their everyday lives. One such question was, “Maria has four bananas. Her mother has five more. How many do they have now?” The questions were read to the child in the language most familiar to the child, although the default language was Tetum. There were four items in total.

Phase Three aimed to identify children’s skills in formal arithmetic. Sub-test 7 presented students with 5 addition problems presented to the child in print. Sub-test 8 presented students with 5 subtraction problems presented in print. These arithmetic problems were simple one and two-digit problems. Students had access to counters as visual and tactile stimuli if required and two examples before each sub-test to show them how to use them. The children had two minutes to complete each sub-test and were stopped if they made three consecutive errors.

TABLE 7. EGMA Sub-tests and Phases of Early Mathematics Learning

Phase Name EGMA Sub-test

Phase Zero Development Sub-tests 1, 2 and 3: Oral counting, correspondence counting, number identification

Phase One Discernment Sub-tests 4 and 5: Quantity Discrimination and Missing Numbers

Phase Two Manipulation Sub-test 6: Word Problems

Phase Three Calculation Sub-tests 7 and 8: Addition and Subtraction

7

Chapter 3 Instrument Design, Sample Design and Execution

Instrument Design and Adaptation

In April 2011, initial adaptation of the EGMA design commenced. Eight volunteer university students from Dili trialed the instrument and advised on the clarity of instructions, usage of symbols to assist in quick referral to interview instructions, cultural consistency and wording, layout of the interview protocol, ease of administration with students, timing for administration of the interview, usage and placement of the manipulative materials and minimizing chances of errors in recording.

Following agreement from the Ministry to proceed with the EGMA program, several workshops with representatives from curriculum and assessment directorate took place. The instrument was trialed with 24 school students in a Dili school.

Adapting the EGMA instrument gave consideration to both the design and individual tasks. The test items were adapted with a view to their consistency with the expectations of the Mathematics Curriculum of Timor-Leste. Word problems were adapted to ensure the content was suitable to the context and the language structure easily accessible to children of ages in Grades 1, 2 and 3.

The linguistic environment of Timor Leste is complex and required some careful design. The protocol was produced in Tetum, the national language and one required to be used for 75% of contact hours in Grades 1 and 2 by the syllabus. The NESP and Mother Tongue policy indicate that a children’s first language is to be the language of instruction in the early years of primary school. Enumerators (interviewers) were instructed to use local language or call an assistant for translation where necessary if the initial introductory conversation with the child revealed that the local language would be preferable to optimize the child’s full understanding of the tasks. However for the tasks where numbers and mathematical operations were read to the child, the language used would be Portuguese, consistent with language used in teacher in-servicing and current mathematics student work books.

Interpretation of these higher-order mathematical tasks should be considered in the light of this policy. Although students should be familiar with the mathematical concepts in Portuguese language, the EGRA 2011 Report indicated that many students struggle with the most basic understanding of oral Portuguese. This may explain some of the poor results exhibited by students and further suggests that research is required into the languages used for instruction.

8


A second instrument was developed to accompany the EGRA survey. The student context questionnaire gathered information about conditions at home and school. But also about classroom practices that may impact on learning, as well as the status and impact of national initiatives implemented in schools.

‘Home’ questions included those related to assistance at home with mathematics homework, nutrition (daily breakfast), money-handling experiences of the family and media inputs for the family via television and radio.

‘School’ questions related to the students’ participation in pre-school, self-images as mathematics learners, frequency of mathematics lessons, experience with text/workbooks, use of manipulative materials in learning mathematics, teacher pedagogical practices (group work for learning), homework and the school feeding program.

The initial Ministry of Education working group adapted the student questionnaire to reflect the current Ministry initiatives that the members considered should be positively impacting on development of student competencies. These included the Ministry of Education’s provision of text books and manipulative materials for mathematics, the promotion of collaborative group learning during teacher in-services and the school feeding (“merenda escolar”) program.

The protocol was further adapted with input from members of the Directorate of Curriculum and Assessment who were delegated as associates in the training, mentoring and monitoring of the enumerators. Enumerators also provided input during their training program, which resulted in the final instrument.

Due to this adaptation, the results are uniquely for Timor-Leste and not intended for international comparison. Both strengths and weaknesses indicated in the findings should be viewed primarily as opportunities for review of and planning for Ministry of Education policies and practices.

Enumerator selection and responsibilities, training and field work

Enumerators were required to have completed secondary school and to have commenced tertiary education in Dili. The enumerator teams contained at least one member who was fluent in the local language(s) in the respective allocated sub-districts for interviewing. There were 28 enumerators selected.

The enumerators participated in 6 days of training in Dili. The training period included practice interviews in Dili schools and was fundamental to the ensuring efficiency of the EGMA implementation. Practice was followed by input from trainers and the Ministry training and monitoring team. The training period concluded with certification of interviewers as suitable for the EGMA implementation.

Following training as enumerators, seven interview teams were formed, with three enumerators and a designated supervisor in each team. In every school visited, three enumerators each conducted an interview with one boy and one girl from each of Grades 1, 2 and 3. Each team was responsible for interviews in two or three Sub Districts over a two-week period.

The Unexpected Consequences of Working in the Field

Unfortunately, on analysis of the data, it was found that the timing data, which is used for determining student fluency, was unreliable. Out of range or unobservable responses were seen to be above 1%, which suggests that unidentifiable errors were also unacceptably prominent in the data. As a result and to ensure accuracy, the decision was taken to analyze the data as total scores rather than as fluency measures as intended by the EGMA survey.

9

Reliability of the Instrument

In order to determine the reliability of the instrument, Cronbach’s alpha was calculated. As a rule of thumb, the minimum acceptable coefficient is 0.7. Table 8 gives the results for the entire sample and the instrument appears reliable with an overall coefficient of 0.89.

However, large numbers of zero-scoring students can inflate measures of reliability such as Cronbach’s alpha and in some sub-tests, zero scoring students were in excess of 20%. Table 9 gives the reliability statistics for the sample where students scoring zero on one or more sub-tests were removed. The reliability of the instrument is still very good with a Cronbach’s alpha of 0.86.

TABLE 8. Cronbach’s Alpha, Complete Sample

Item N Sign Item-test Item-rest Inter-item correlation Alpha

Numbers Counted 1226 + 0.71 0.62 0.51 0.88

Circles Counted 1224 + 0.56 0.43 0.56 0.90

Number Identification (as a percentage) 1224 + 0.88 0.83 0.47 0.86

Quantity Identification (as a percentage) 1225 + 0.76 0.67 0.50 0.88

Missing Number (as a percentage) 1221 + 0.77 0.68 0.50 0.87

Word Problems (as a percentage) 1213 + 0.75 0.66 0.50 0.88

Additions (as a percentage) 1219 + 0.86 0.81 0.47 0.86

Subtractions (as a percentage) 1208 + 0.73 0.63 0.51 0.88

Test Scale 0.50 0.89

TABLE 9. Cronbach’s Alpha, Sample with Zero-Scoring Students Removed

Item N Sign Item-test Item-rest Inter-item correlation Alpha

Numbers Counted 1225 + 0.73 0.62 0.43 0.84

Circles Counted 1223 + 0.62 0.43 0.49 0.87

Number Identification (as a percentage) 1182 + 0.86 0.79 0.39 0.81

Quantity Identification (as a percentage) 1210 + 0.77 0.65 0.41 0.83

Missing Number (as a percentage) 1007 + 0.77 0.66 0.42 0.84

Word Problems (as a percentage) 965 + 0.63 0.48 0.46 0.86

Additions (as a percentage) 861 + 0.81 0.70 0.42 0.84

Subtractions (as a percentage) 632 + 0.72 0.60 0.44 0.85

Test Scale 0.43 0.86

Instrument Design, Sample Design and Execution

10


TABLE 10. Correlation Between Sub-tests

All Numbers Counted

Circles Counted

Number Identification

(as %)

Quantity Identification

(as %)

Missing Number

(as %)

Word Problems

(as %)

Additions (as %)

Subtractions (as %)

Numbers Counted 1

Circles Counted 0.43 1

Number Identification (as %) 0.64 0.45 1

Quantity Identification (as %) 0.46 0.29 0.68 1

Missing Number (as %) 0.46 0.29 0.65 0.53 1

Word Problems (as %) 0.41 0.29 0.58 0.49 0.52 1

Additions (as %) 0.50 0.35 0.74 0.64 0.62 0.66 1

Subtractions (as %) 0.37 0.23 0.53 0.46 0.54 0.56 0.68 1

Correlation between sub-tests is given in Table 10. Most sub-tests are well correlated with one another, however the second sub-test, correspondence counting or “circles counted” has a low degree of correlation with the other sub-tests. Possible reasons for this are given in Chapter Four.

Overall, despite some difficulties with the linguistic diversity of Timor and execution of the timed sub-tests, the instrument is reliable and offers an interesting insight into the learning of mathematics in the early grades of Timor Leste.

Sample Design

The sample was a stratified random sample using the EMIS schools list as a sample frame. The sample was stratified by district and school type. All 13 districts were represented in the sample. The target population was students in Grades 1-3 in schools implementing the national mathematics curriculum. Each grade sampled around 400 students and 1226 students in 65 schools were sampled, including 73 students from the piloting of the instrument. Gender parity was close to unity. Table 11 gives the sample frequencies by grade and gender.

TABLE 11. Gender and Grade Distribution

GenderGender

Total1 2 3

Male 206 203 200 609

Female 206 201 210 617

Total 412 404 410 1,226

11

Chapter 4 Sub-test Results

This chapter will present sub-test results by grade, by gender and overall. Special attention should be paid not just to average progression, but also median progression. Students who are particularly able can inflate average scores, but the median gives the score of the student on the 50th percentile (the middle student) and is a good counterbalance to this.

Sub-test One: Oral Counting

In this sub-test, students were asked to begin at one and count onwards as high as they could in one minute. As discussed in the previous chapter, timing data was dropped and the scores are analyzed in their raw form.Table 12 gives the results of this sub-test. Overall, results show average grade progression that is statistically significant. Only one student in Grade One was unable to count at all. On average, students in Grade One were able to count to 39 and the median student was also able to count to 39. Grade Two students were able to count to 56 on average and the median child counted to 57. Grade Three students were able to count to 66 on average and the median child in Grade Three counted to 69. Boys and girls performed very similarly and there were no significant differences between them. Figure 1 gives a graphical presentation of these results.

FIGURE 1: Oral Counting

Numbers counted in one minute

Grade 1

Grade 2

Grade 3

0 10 20 30 40 50 60 70 80

MedianMean

12


Sub-test Two: Correspondence Counting

In this sub-test students were asked to count a series of printed circles on a page: as many as they could in one minute. Table 13 gives the results of this sub-test. Like sub-test 1, students showed good results with Grade 1 students on average able to count 25 circles, Grade 2 students, 29 circles and Grade 3 students, 29 circles also. The median student in each grade counted 30 circles. Only one student was unable to count any circles.

Although there is not evidence of strong grade progression in this sub-test, this may be due to the nature of its presentation, rather than a particular lack in correspondence counting skills in older students. There were 30 circles on the page presented to students. Although some students counted up to 60 circles successfully (the page twice), it is possible some enumerators stopped students after they completed the page or that some students may have made a mistake when re-starting the page to count for the second group of 30 circles and been stopped. This may also account for this sub-test’s relatively weak correlation with the other sub-tests discussed in Chapter Three. Figure 2 gives the frequency distribution of the sub-test results overall. As can be seen, an unexpectedly disproportionate number of children were stopped, or stopped, at exactly 30 circles (over 78% of students). Less than 1% of students continued counting beyond 30 circles.

FIGURE 2: Sub-test 2, Correspondence Counting

TABLE 12. Sub-test 1, Oral Counting

Grade Mean SD Min Max Median N LCL UCL

Overall

1 39 20 0 100 39 412 37 40

2 56 20 1 100 57 404 54 57

3 66 20 1 100 69 410 65 68

Total 53 23 0 100 56 1226 52 55

Zero Scores Removed

1 39 20 1 100 39 411 37 41

2 56 20 1 100 57 404 54 57

3 66 20 1 100 69 410 65 68

Total 54 23 1 100 56 1225 52 55

Girls

1 39 20 0 100 39 206 37 42

2 54 19 9 100 56 201 51 57

3 67 19 7 100 70 210 65 70

Total 54 23 0 100 56 617 52 55

Boys

1 38 21 3 100 35 206 35 41

2 57 21 1 100 57 203 54 60

3 65 21 1 100 69 200 63 68

Total 53 24 1 100 56 609 51 55

Circles counted

Perc

ent

0 20 40 600

10

20

30

40

50

60

70

80

13

Sub-test Three: Number Identification

In this sub-test, students were given 12 numbers and asked to identify them. The numbers ranged from one figure up to four figures. The numbers were 2, 5, 9, 13, 10, 18, 65, 50, 97, 104, 468, 6430. Table 14 presents the results of this sub-test as a percentage of correct answers. Evidence of grade progression is strong. Grade One students recognized, on average, 35% and the median child 33%. Grade Two students recognized 65% on average and the median student recognized 75%. Grade Three students recognized 80% on average and the median child recognized 83%. Figure Three presents the results of this sub-test.

Only 3% of students were unable to answer this sub-test. Most of these students were in Grade One, where 9% of students did not recognize any numbers and only 1.2% in Grade Two. However, the fact that 9% of Grade One students unable to recognize any of 12 numbers is concerning.

Boys scored consistently higher than girls in all grades, however this difference is not significant.

FIGURE 3: Sub-test 3, Number Identification

TABLE 13. Sub-test 2, Correspondence Counting


Overall

1 25 8 0 60 30 410 24 26

2 29 4 5 46 30 404 28 29

3 29 3 2 31 30 410 29 30

Total 28 6 0 60 30 1224 27 28

Zero Scores Removed

1 25 8 1 60 30 409 24 26

2 29 4 5 46 30 404 28 29

3 29 3 2 31 30 410 29 30

Total 28 6 1 60 30 1223 27 28

Girls

1 26 8 1 60 30 206 25 27

2 29 5 5 46 30 201 28 29

3 29 3 2 30 30 210 29 30

Total 28 6 1 60 30 617 27 28

Boys

1 24 8 0 40 30 204 23 26

2 29 4 5 30 30 203 28 29

3 29 3 5 31 30 200 29 30

Total 28 6 0 40 30 607 27 28

% of numbers correctly identified

Grade 1

Grade 2

Grade 3

0% 20% 40% 60% 80% 100%

MedianMean

Sub-Test Results

14


Sub-test Four: Quantity Discrimination

In this sub-test, students were asked to identify the larger of two numbers. There were ten items in total and results are presented in Table 15 as a percentage. The complete sub-test is presented in Annex 2: EGMA Instrument. Results in this sub-test were showed reasonable grade progression. Grade One students scored an average of 54% and the median child 50%. Grade Two students scored an average of 73% and the median child 80%. Grade Three students scored an average of 85% and the median child in that grade scored 90%. The proportion of zero scoring students was very low at only 1.2%. Most of these were in Grade One where 3% of students were unable to answer any of the questions correctly. Boys performed consistently better than girls in this sub-test and there was a significant difference between the genders. The reasons for this are not clear and bear further investigation. Figure 4 presents the gender gap between students graphically.

FIGURE 4: Sub-test 4, Quantity Discrimination

TABLE 14. Sub-test 3, Number Identification


Overall

1 35% 23% 0% 100% 33% 410 33% 37%

2 65% 23% 0% 100% 75% 404 63% 67%

3 80% 17% 0% 100% 83% 410 79% 82%

Total 60% 28% 0% 100% 67% 1224 58% 62%

Zero Scores Removed

1 38% 21% 0% 100% 33% 373 36% 40%

2 66% 22% 0% 100% 75% 399 64% 68%

3 80% 17% 0% 100% 83% 410 79% 82%

Total 62% 26% 0% 100% 67% 1182 61% 64%

Girls

1 34% 22% 0% 100% 33% 206 31% 37%

2 63% 23% 0% 100% 67% 201 59% 66%

3 80% 18% 0% 100% 83% 210 77% 82%

Total 59% 28% 0% 100% 67% 617 57% 61%

Boys

1 36% 24% 0% 100% 33% 204 33% 39%

2 67% 22% 0% 100% 75% 203 64% 70%

3 81% 16% 0% 100% 83% 200 79% 83%

Total 61% 28% 0% 100% 67% 607 59% 63%

% of items correctly identified

Grade 1

Grade 2

Grade 3

0% 20% 40% 60% 80% 100%

BoysGirls

15

Sub-test Five: Missing Numbers

This sub-test asked students to complete a series of missing numbers. Table 16 gives the results of this sub-test in percentage form. There were ten items overall. Grade progression begins from a low base. Grade One students score only 19% on average with the median child scoring 20%. Grade Two students score 36% on average and the median child 30%. Grade Three students score just 50% on average and the median Grade 4 child 50%. There was a high incidence of zero-scoring students in this sub-test with 18% of students overall unable to answer any questions. The highest proportion was, as expected in Grade 1 with 30% of students unable to answer. However, Grade 2 and 3 had substantial proportions of students scoring zero: 14% and 9% respectively. Boys scored marginally higher than girls in this sub-test, however, these results were not significant. Figure 5 gives the proportion of zero-scoring students by grade in graphical format.

FIGURE 5: Sub-test 5, Missing Numbers

TABLE 15. Sub-test 4, Quantity Discrimination


Overall

1 54% 24% 0% 100% 50% 412 51% 56%

2 73% 23% 0% 100% 80% 403 70% 75%

3 85% 17% 0% 100% 90% 410 84% 87%

Total 70% 25% 0% 100% 80% 1225 69% 72%

Zero Scores Removed

1 55% 22% 0% 100% 50% 399 53% 57%

2 73% 23% 0% 100% 80% 402 71% 75%

3 86% 16% 0% 100% 90% 409 84% 87%

Total 71% 24% 0% 100% 80% 1210 70% 73%

Girls

1 51% 24% 0% 100% 50% 206 48% 54%

2 68% 24% 0% 100% 70% 200 65% 71%

3 84% 18% 0% 100% 90% 210 81% 86%

Total 68% 26% 0% 100% 70% 616 66% 70%

Boys

1 56% 23% 0% 100% 60% 206 53% 59%

2 77% 20% 0% 100% 80% 203 74% 80%

3 87% 15% 0% 100% 90% 200 85% 89%

Total 73% 24% 0% 100% 80% 609 71% 75%

% of correct responses

Grade 1

Grade 2

Grade 3

0% 10% 20% 30% 40% 50%

Proportion of zero scoring students Median Mean

Sub-Test Results

16


Sub-test Six: Word Problems

This sub-test presented students with 4 problems, administered orally. The problems are presented in the instrument, available in Annex 2. A complete item analysis is given in Annex 3. The problems were simple calculations such as the children may have been familiar with from their every day lives in household transactions. Table 17 presents the results of this sub-test in percentage form. Grade One students answered on average 28% of the problems correctly and the median student 25%; grade progression is very slow for such simple problems. Grade Two students answered just 47% of problems correctly on average and the median child 50%. Grade Three students answered just 61% on average correctly and the median child showed no progression from Grade Two, answering just 50% correctly. Although the average scores in each grade show significant differences, the lack of median progression is deeply concerning.

Students unable to answer any questions correctly make up a substantive proportion of this sample, 20% overall. The majority are in Grade One where 40% of students are unable to answer any of the questions, while in Grade Two 17% are unable to answer any questions. Grade Three indicates that the bottom cohort of students is making some progress with only 5% of students unable to answer any questions correctly. Figure 6 gives these results graphically.

Boys perform statistically significantly better than girls in this sub-test overall. By grade, the results are not significant in Grade One, but are significant in Grades Two and Three.

FIGURE 6: Sub-test 6, Word Problems

TABLE 16. Sub-test 5, Missing Numbers


Overall

1 19% 18% 0% 100% 20% 407 17% 21%

2 36% 25% 0% 100% 30% 404 34% 39%

3 50% 28% 0% 100% 50% 410 47% 52%

Total 35% 27% 0% 100% 30% 1221 33% 36%

Zero Scores Removed

1 27% 16% 0% 100% 20% 286 25% 29%

2 42% 22% 0% 100% 40% 349 40% 44%

3 55% 24% 0% 100% 60% 372 52% 57%

Total 42% 24% 0% 100% 40% 1007 41% 44%

Girls

1 19% 18% 0% 100% 20% 203 16% 21%

2 35% 25% 0% 100% 30% 201 32% 39%

3 49% 27% 0% 100% 50% 210 45% 53%

Total 35% 27% 0% 100% 30% 614 32% 37%

Boys

1 19% 18% 0% 100% 20% 204 16% 21%

2 37% 25% 0% 100% 30% 203 34% 41%

3 50% 28% 0% 100% 50% 200 46% 54%

Total 35% 27% 0% 100% 30% 607 33% 37%

Numbers counted in one minute

Grade 1

Grade 2

Grade 3

0% 10% 20% 30% 40% 50% 60% 70% 80%


17

Sub-test Seven: Addition

This sub-test presented children with five simple addition problems (4+5, 8+2, 20+4, 13+12, 11+9) to be completed within two minutes. Students were stopped if they answered three items incorrectly. Table 18 presents the results of this sub-test. Grade progression is significant overall, but begins from a very low base. In Grade One, the average student answers 21% of the questions correctly, but the median child cannot answer any. In Grade 2 substantial gains are made. The average student answers 53% of the questions correctly and the median child 60%. In Grade 3, the average child can answer 72% of the questions correctly and the median child 80%.

The proportion of zero-scoring students is substantial, 29% overall. Most of these are in Grade One, where 55% of students are unable to answer any questions. Despite the good average gains in Grade Two a concerning 23% of students, nearly one in four, are unable to answer any questions. In Grade 3, 10% of students are unable to answer any questions. Figure 7 gives these results graphically. Results on basic arithmetic problems raise substantial concerns about students’ ability to cope with the curriculum requirements in higher grades. Boys performed statistically significantly better than girls in this sub-test.

FIGURE 7: Sub-test 7, Addition

TABLE 17. Sub-test 6, Word Problems


Overall

1 28% 28% 0% 100% 30% 404 26% 31%

2 47% 29% 0% 100% 50% 400 44% 50%

3 62% 27% 0% 100% 50% 409 59% 64%

Total 46% 31% 0% 100% 50% 1213 44% 48%

Zero Scores Removed

1 47% 20% 0% 100% 50% 242 45% 50%

2 57% 22% 0% 100% 50% 333 54% 59%

3 65% 24% 0% 100% 80% 390 62% 67%

Total 58% 24% 0% 100% 50% 965 56% 59%

Girls

1 25% 26% 0% 100% 30% 201 21% 28%

2 43% 28% 0% 100% 50% 198 39% 47%

3 57% 27% 0% 100% 50% 209 53% 60%

Total 42% 30% 0% 100% 50% 608 39% 44%

Boys

1 32% 29% 0% 100% 30% 203 28% 36%

2 52% 30% 0% 100% 50% 202 48% 56%

3 67% 27% 0% 100% 80% 200 63% 70%

Total 50% 32% 0% 100% 50% 605 47% 53%

% of correct items

Grade 1

Grade 2

Grade 3


0% 20% 40% 60% 80% 100%

Sub-Test Results

18


TABLE 18. Sub-test 7, Addition


Overall

1 21% 29% 0% 100% 0% 409 18% 24%

2 53% 37% 0% 100% 60% 402 49% 56%

3 72% 32% 0% 100% 80% 408 69% 75%

Total 49% 39% 0% 100% 60% 1219 46% 51%

Zero Scores Removed

1 47% 25% 0% 100% 40% 183 44% 51%

2 68% 26% 0% 100% 80% 310 66% 71%

3 80% 23% 0% 100% 80% 368 78% 82%

Total 69% 28% 0% 100% 80% 861 67% 71%

Girls

1 18% 27% 0% 100% 0% 203 14% 22%

2 48% 37% 0% 100% 60% 200 43% 53%

3 66% 34% 0% 100% 80% 209 62% 71%

Total 44% 39% 0% 100% 40% 612 41% 47%

Boys

1 24% 30% 0% 100% 0% 206 20% 29%

2 58% 36% 0% 100% 60% 202 53% 63%

3 78% 29% 0% 100% 80% 199 74% 82%

Total 53% 39% 0% 100% 60% 607 50% 56%

Sub-test Eight: Subtraction

The final sub-test presented students with five simple subtraction problems (9-5, 10-8, 24-4, 25-12, 20-9) to be completed in two minutes. Table 19 gives the results of this sub-test. Grade progression is significant, but starts from a low base and is very slow. Grade One students are able to answer just 12% on average, while the median student can answer none. Grade Two students are able to answer 30% on average and the median student 20%. Grade Three students are able to answer 46% on average and the median student 40%. Boys performed significantly better than girls in this sub-test.

The proportion of zero-scoring students is very substantial, 48% of students overall. A majority of Grade One students, 67% were unable to answer a single question in this sub-test and 46% of Grade Two students were still unable to answer any questions. Grade Three students presented the most worrying statistic with almost 1 in 3, 30% unable to answer any of the questions. Figure 8 presents this graphically.

FIGURE 8: Sub-test 8: Subtraction

% of correct items

Grade 1

Grade 2

Grade 3


0% 20% 40% 60% 80%

19

TABLE 19. Sub-test 8, Subtraction


Overall

1 12% 21% 0% 100% 0% 405 10% 14%

2 30% 35% 0% 100% 20% 398 27% 34%

3 46% 38% 0% 100% 40% 405 42% 49%

Total 29% 35% 0% 100% 20% 1208 28% 31%

Zero Scores Removed

1 38% 21% 0% 100% 40% 132 34% 41%

2 56% 28% 0% 100% 60% 215 53% 60%

3 65% 28% 0% 100% 60% 285 62% 68%

Total 56% 28% 0% 100% 60% 632 54% 59%

Girls

1 10% 20% 0% 100% 0% 202 7% 13%

2 25% 32% 0% 100% 0% 197 20% 29%

3 39% 36% 0% 100% 40% 208 34% 44%

Total 25% 32% 0% 100% 0% 607 22% 27%

Boys

1 14% 22% 0% 100% 0% 203 11% 17%

2 36% 37% 0% 100% 20% 201 31% 41%

3 53% 38% 0% 100% 60% 197 48% 58%

Total 34% 36% 0% 100% 20% 601 31% 37%

Conclusion

Although Timor Leste students demonstrate reasonable abilities to recognize and discern numbers with solid results and steady, though slow grade progression in oral counting, correspondence counting, number identification and quantity discernment; higher order early mathematics skills are lacking, progress from a low base and very slowly. The proportion of students with no demonstrable skills in these manipulative and arithmetic skills (missing numbers, word problems, addition and subtraction) is worryingly high and low levels of attainment even amongst those students demonstrating some skill puts in doubt their ability to cope with a more complex mathematical curriculum in later years.

Sub-Test Results

20


21

Chapter 5 Language and Mathematics

In the linguistically diverse environment of Timor Leste, curriculum suggests that students are taught in both Tetum and Portuguese to a ratio of 3:1 contact hours in each language respectively. The workbooks that were distributed as part of the national intervention in mathematical materials are in Portuguese. The EGMA instrument used in Timor Leste was primarily delivered in Tetum with sub-tests 4 (quantity discrimination), 7 (addition) and 8 (subtraction) delivered in Portuguese so as to remain in line with the material presented to students in the classroom. Enumerators had language support in the form of a translator or member of the team who was proficient in local language. These were called upon when in the conversation with the student preliminary to the survey it became clear that students needed to be tested in their local language.

Information on languages used by enumerators and students was collected for sub-tests 1, 2, 3, 5 and 6. It is this information that is the subject of the analysis in this chapter.

Some children and enumerators used multiple combinations of languages in each sub-test. Some children and enumerators used up to three languages for an individual sub-test and these different combinations are difficult to analyze individually with any clarity.

To counteract this possible source of confusion, we have divided students and enumerators into groups: those who used Tetum in a sub-test and those who did not; those who used Portuguese in a sub-test and those who did not; those who used another language (Bahasa Indonesian or local languages) in a sub-test and those who did not. These form the basis of our comparisons. Given that some students and enumerators used multiple combinations of languages it is impossible to consider the language groups mutually exclusive and this should be borne in mind when interpreting the results.

22


Languages Used by Enumerators

The majority of the survey was in Tetum, however some parts were in Portuguese. The Portuguese-language sub-tests did not have accompanying language data collected. Enumerators were directed to use the language the student felt most comfortable with, information gathered during a preparatory conversation with the child to put him/her at their ease. Enumerator language information was collected for four sub-tests: oral counting, correspondence counting, number identification and the word problems. Comparing these to child language data collected for the same sub-tests we can see the relationships between child and enumerator conversations. These cross tabulatations are given in Tables 20-23. For example, in Table 20, we can see that in the oral counting sub-test, although the enumerator most often used Tetum (1052 surveys), the child did not respond in kind. Most students responded in Portuguese (1172 surveys). Only 73 students used another language than Tetum or Portuguese. In 50 of these cases, the enumerator did not respond in a third language.

TABLE 20. Frequency of Enumerator and Child Languages Used; Oral Counting

Child

Enumerator No Tetum Tetum Total

No Tetum 139 0 139

Tetum 1052 35 1087

Total 1191 35 1226

Child

Enumerator No Portuguese Portuguese Total

No Portuguese 50 1172 1222

Portuguese 1 3 4

Total 51 1175 1226

Child

EnumeratorNo other language

Other language

Total

No other language 998 50 1048

Other language 155 23 178

Total 1153 73 1226

TABLE 21. Frequency of Enumerator and Child Languages Used; Correspondence Counting

Child


No Tetum 145 2 147

Tetum 934 145 1079

Total 1079 147 1226

Child



Portuguese 2 5 7

Total 153 1073 1226

Child


Other language

Total



Total 1039 187 1226

TABLE 22. Frequency of Enumerator and Child Languages Used; Numbers Identified

Child


No Tetum 152 1 153

Tetum 1015 58 1073

Total 1167 59 1226

Child



Portuguese 1 7 8

Total 45 1181 1226

Child


Other language

Total



Total 974 252 1226

TABLE 23. Frequency of Enumerator and Child Languages Used; Word Problems

Child


No Tetum 149 13 162

Tetum 393 671 1064

Total 542 684 1226

Child

Enumerator No Portuguese Portuguese


Portuguese 0 7 7

Total 478 748 1226

Child


Other language

Total



Total 951 275 1226

23

Table 20 indicates that although the enumerator may ask the child a question in one language, the child will often choose to answer in a different language. This is especially true of sub-tests in which the child is asked a question in Tetum but may choose to respond in Portuguese. This is a direct contrast to the 2011 EGRA results, which indicated that children tested in Portuguese were unable to answer many, if any listening comprehension questions asked. It is likely a result of rote memorization of numbers and other numeracy concepts in Portuguese. This may be a direct result of the workbooks distributed and used extensively for mathematics teaching in Timorese classrooms. It may reflect the different strengths of the children’s vocabulary in the different languages at their command and their inherent facility at “switching” between languages in a multilingual environment.

Language and Performance

This section compares groups of students by the language(s) they answered the sub-tests for which there are language data available. Note that some students chose to use combinations of languages. These groups are not mutually exclusive and thus, the results should be interpreted with caution.

Table 24 gives a comparison by language for the first sub-test, oral counting. It indicates that whilst there are no significant differences between students speaking Tetum and not speaking Tetum, between speaking Portuguese and not speaking Portuguese; there is a significant difference between students answering in another language and those who do not. That is, there are statistical differences between those students speaking in a third language and those who do not. Students who answered this particular sub-test with at least one other language than Portuguese or Tetum scored significantly higher than those who did not. This unexpected result required further exploration.

Looking specifically at students who answered in Bahasa Indonesian (also in Table 24), it appears that these students have a substantially higher mean and median result than other students. This may be due to some unspecified socio-economic factor for which we do not have data.

Table 25 gives the results of the second sub-test, correspondence counting, by language. Due to the unexpected enumerator behavior described in the last chapter where most students were stopped at the end of the page (rather than repeating it), the vast majority of students stopped counting at 30 and there are no significant differences between the languages used.

Table 26 gives the results of the third sub-test, number identification, by language. It indicates that students who do not use Tetum have significantly better scores in this sub-test than those who do. Conversely (and not mutually exclusively) those who use Portuguese have significantly better results than those who do not. Other languages were not significant. This may indicate that those students able to understand Portuguese well enough to identify numbers are better able to absorb the mathematics curriculum which is, anecdotally at least, taught predominantly in Portuguese. This suggests that children unable to understand basic mathematics vocabulary in Portuguese are at risk of falling behind in the curriculum. Further research is required into how mathematics teaching is taking place in Timorese classrooms.

The missing number sub-test had a similar outcome. Table 27 gives the results of this sub-test by language. Table 27 indicates that whilst other languages are not significant, students who answer this sub-test in Portuguese are associated with higher scores than those who do not.

Table 28 gives the results for the word problems sub-test by language. Although none of the language variables are statistically significant, students answering in Portuguese have an average score higher than those who do not.

Overall, it appears that amongst those skills for which language data was collected, language is an important factor associated with student performance. However, it should not be seen as a causative factor in student performance, but rather one associated with performance. In an educational environment as linguistically diverse as Timor Leste, language is complex. Without data on the use and purpose of each language and its frequency, it is impossible to determine how concepts, pedagogy and language combine in the classroom to promote learning. Given that these results diverge substantially from what might be expected based on the EGRA 2011 assessment, more research into actual classroom language practices and possible socio-economic correlations is urgently required.

Language and Mathematics

24


TABLE 25. Correspondence Counting Results by Language (Non-overlapping confidence intervals given in bold)

Mean SD Min Max Median N LCL UCL

No Tetum 28 6 2 60 30 1077 28 28

Tetum 26 8 0 30 30 147 25 27

Total 28 6 0 60 30 1224 27 28

No Portuguese 25 8 0 31 30 151 24 27

Portuguese 28 6 1 60 30 1073 28 28

Total 28 6 0 60 30 1224 27 28

No other language (includes Indonesian) 28 6 0 60 30 1037 27 28

Other language (includes Indonesian) 27 6 3 31 30 187 27 28

Total 28 6 0 60 30 1224 27 28

TABLE 24. Oral Counting Results by Language (Non-overlapping confidence intervals given in bold)


No Tetum 54 23 0 100 56 1191 52 55

Tetum 51 27 2 96 51 35 42 60

Total 53 23 0 100 56 1226 52 55

No Portuguese 56 29 9 100 60 51 48 64

Portuguese 53 23 0 100 56 1175 52 55

Total 53 23 0 100 56 1226 52 55

No other language (includes Indonesian) 53 23 0 100 55 1153 52 54

Other language (includes Indonesian) 63 27 9 100 70 73 57 69

Total 53 23 0 100 56 1226 52 55

No Indonesian 53 23 0 100 55 1179 52 54

Indonesian 70 25 19 100 73 47 63 77

Total 53 23 0 100 56 1226 52 55

25

TABLE 28. Word Problems Results by Language (Non-overlapping confidence intervals given in bold)


No Tetum 46% 31% 0% 100% 50% 1066 44% 48%

Tetum 44% 32% 0% 100% 50% 147 39% 49%

Total 46% 31% 0% 100% 50% 1213 44% 48%

No Portuguese 41% 30% 0% 100% 50% 152 37% 46%

Portuguese 46% 31% 0% 100% 50% 1061 45% 48%

Total 46% 31% 0% 100% 50% 1213 44% 48%

No other language 46% 31% 0% 100% 50% 1027 44% 48%

Other language 45% 30% 0% 100% 50% 186 41% 49%

Total 46% 31% 0% 100% 50% 1213 44% 48%

TABLE 26. Number Identification Results by Language (Non-overlapping confidence intervals given in bold)


No Tetum 61% 28% 0% 100% 67% 1077 59% 62%

Tetum 54% 32% 0% 100% 67% 147 49% 59%

Total 60% 28% 0% 100% 67% 1224 58% 62%

No Portuguese 50% 32% 0% 100% 50% 153 45% 55%

Portuguese 61% 27% 0% 100% 67% 1071 60% 63%

Total 60% 28% 0% 100% 67% 1224 58% 62%


Other language 57% 29% 0% 100% 67% 186 52% 61%

Total 60% 28% 0% 100% 67% 1224 58% 62%

TABLE 27. Missing Numbers Results by Language (Non-overlapping confidence intervals given in bold)


No Tetum 36% 27% 0% 100% 30% 1075 34% 37%

Tetum 30% 27% 0% 100% 25% 146 26% 35%

Total 35% 27% 0% 100% 30% 1221 33% 36%

No Portuguese 25% 25% 0% 100% 20% 152 21% 29%

Portuguese 36% 27% 0% 100% 30% 1069 35% 38%

Total 35% 27% 0% 100% 30% 1221 33% 36%


Other language 30% 26% 0% 100% 30% 187 27% 34%

Total 35% 27% 0% 100% 30% 1221 33% 36%

Language and Mathematics

26


27

Chapter 6 Factors Associated with Better Numeracy Outcomes

As an accompaniment to the EGMA survey, a brief socio-economic survey was undertaken at the same time in order to determine which school- and student-specific factors have an association with early mathematics outcomes. The contents of the socio-economic survey were necessarily brief and decided on by the ministry in accordance with the research questions that were of interest. Specifically, during 2011 and 2012 an intervention in the form of mathematics workbooks (in Portuguese language) and physical resources such as manipulative objects was rolled out nationally. The ministry was interested to know how these affected mathematics outcomes.

When interpreting the results of this survey, it is important to note that the variables in this survey are self-reported by very young children. The results should be treated cautiously. Table 29 gives the basic statistics for the socio-economic indicators collected. Table 30 gives the relationships of the indicators to several key sub-tests of the EGMA survey. Significant variables are given in bold and the difference in average sub-test scores between groups with and without the indicators is also given. The significance level used in Table 30 is the 95% confidence level. Please note that whilst these indicators may have an effect on student performance, the results given in Table 30 should not be construed as causative but as associative. There are many factors that may be associated with student performance and these statistics given cannot account for them all. The figures in Table 30 should be used not as predictive measures but as measures of association.

Five sub-tests were chosen to compare the association of the socio-economic factors chosen for inclusion in the survey. These were chosen partly for reasons of brevity and partly to avoid misleading results by including the early sub-tests (oral counting and correspondence counting), which were discussed at length in previous chapters. The sub-tests chosen concentrated on the skills with which students had the most difficulty; as these are the skills most likely to require intervention in the near future. These were: number identification, missing numbers, word problems, addition and subtraction. Full statistics are given in Annex One.

28


Approximately 37% of students reported that they had attended preschool. Whilst this figure may seem quite high for a country of the development status such as Timor Leste, it should be noted that in other Pacific countries, the definition of “preschool” is an amorphous one. It can mean anything from an hour’s activities in the church hall on a weekly basis to a five-day per week specific early childhood development program. Its meaning to the young children answering the question is not defined. Attendance at preschool was not significantly associated with any of the sub-tests. Although students who self-reported as attending preschool had lower mean scores in each of the sub-tests than those who did not, these differences between estimated scores with and without attending preschool are very marginal and close to zero.

Just over 91% of students self-reported that they had mathematics lessons daily. The statistic is encouraging, but should be treated with caution since it is possible some local translators used were the children’s schoolteachers. Also encouraging is the fact that students who self-reported they had daily mathematics lessons had significantly higher sub-test scores on average in all sub-tests examined excepting word problems. The sub-test average scores were between 8% and 17% depending on sub-test higher for those children who had daily mathematics lessons.

Textbook ownership was high with 80% of students reporting they owned the text. However, given that the texts were distributed nationally, this figure may in fact be too low. Ownership of the text conferred no significant advantage to students over those who did not: although students with the textbook had on average higher sub-test scores than those who did not, these were not significantly different from zero at the 95% confidence level.

Students also reported high degrees of using objects in mathematics lessons (75%) and working with others (83%). Whilst both items had positive associations with better sub-test scores on average, only working with others was significant for the number identification, missing numbers and word problems sub-tests.

Over 90% of students reported doing homework, but only one student in the entire sample reported having help with homework. This suggests that parental involvement with mathematics in early grade education is minimal or non-existent. Doing homework had a significant, positive effect on all sub-test scores with increases ranging from between 12-23%,

TABLE 29. School and student-specific variables, descriptive statistics

Factor Proportion SD

Attended preschool 37% 48%

Maths Daily 91% 29%

Textbook 80% 40%

Uses Objects 75% 43%

Works with others 83% 37%

Does homework 90% 30%

Has help with Homework 0% 3%

Sells things 39% 49%

Breakfast that morning 89% 32%

Breakfast Daily 92% 28%

Merenda Escolar 95% 21%

TV at home 41% 49%

Radio at Home 56% 50%

29

TABLE 30. Effects of School and Student-Specific Variables on Average Scores Part A

Item

Subtest

Numbers Identified Missing Numbers Word Problems Addition Subtraction

Sign and Significance

Difference at Mean


Difference at Mean


Difference at Mean


Difference at Mean


Difference at Mean

Attended Preschool

- -0.027 - -0.018 - -0.035 - -0.026 - -0.008

Maths Daily + 0.159 + 0.079 + 0.066 + 0.167 + 0.080

Owns Text + 0.102 + 0.079 + 0.066 + 0.167 + 0.080

Uses Objects + 0.024 + 0.008 + 0.009 + 0.020 + 0.015

Works with Others

+ 0.078 + 0.056 + 0.071 + 0.020 + 0.057

Does Homework

+ 0.179 + 0.121 + 0.141 + 0.226 + 0.141

Child Sells in Marketplace

- -0.011 - -0.010 - -0.026 - -0.004 + 0.005

Child ate breakfast that morning

+ 0.051 + 0.018 + 0.018 + 0.010 + 0.052

TABLE 31. Effects of School and Student-Specific Variables on Average Scores Part B

Item

Subtest

Numbers Identified Missing Numbers Word Problems Addition Subtraction


Difference at Mean


Difference at Mean


Difference at Mean


Difference at Mean


Difference at Mean

Child Eats Breakfast Everyday

+ 0.048 + 0.015 + 0.010 + 0.026 + 0.024

Merenda Escolar

+ 0.079 + 0.065 + 0.057 + 0.094 + 0.096

Family Owns TV

+ 0.036 + 0.071 + 0.001 + 0.052 + 0.070

Family Owns Radio

+ 0.033 + 0.032 + 0.017 + 0.045 + 0.039

Factors Associated with Better Numeracy Outcomes

30


It was considered possible that the many Timorese children whose parents sell goods in the market place may have access to a means of numeracy education outside of school. Nearly 40% of students (38.5%) indicated that they engaged in this practice. However, there was no significant association with learning outcomes.

Childhood nutrition is known to be an important determinant of educational outcomes. Students were asked if they had eaten breakfast that day (89% had eaten breakfast), if they ate breakfast everyday (92%) and about school participation in the school feeding program merenda escolar (95%). The fact that fewer students had eaten that morning than eat breakfast every day can be ascribed to two things: either a random quirk from small children self-reporting, or that some children felt compelled to reply in the affirmative for reasons of pride. There is a statistically significant difference between the proportions of students who eat breakfast every day and those who had eaten that morning. That is, students self-reported incorrectly to a significant degree. This result urges caution in ascribing reliability to this section of the data. Furthermore, although all three nutritional variables had a positive association with sub-test average scores: none of them were statistically significant at the 95% level.

Lastly, students were asked about two consumer durables in their home: the presence of a TV (41%) and/or a radio (56%). Both of these had a positive association with student outcomes. Having a television in the home was significantly associated with positive outcomes in both the missing numbers and subtraction sub-tests at the 95% level and may reflect a positive socio-economic association with education. Another possibility is that the presence of a television in the home may offer exposure to languages used in the classroom whicb may not be in regular use in the home.

Overall, the short socio-economic survey was not able to offer substantial guidance on future interventions in part due to the age of the respondents and in part due to its short length. However, classroom practices such as a daily mathematics lesson, working with others and homework had significant, positive associations with mathematics outcomes. A television at home had a positive, significant association with some sub-test results, but rather than a causal link it is likely that it is an instrument for socio-economic status.

31

Chapter 7 Conclusions and Recommendations

The analysis of the EGMA 2011 Timor Leste baseline survey calls for an immediate response to safeguard the educational future of young Timorese students. Although students perform reasonably in basic, “Phase Zero” mathematics skills such as oral counting, correspondence counting and number identification; this may be in part due to a misinterpretation of the correspondence counting sub-test by many enumerators. Higher order discernment skills show signs of slow gains. Later manipulative and calculation skills, those in “Phase Two” and “Phase Three” (missing numbers, word problems and arithmetic) are poorly understood and reproduced by Timorese students. After three years of schooling, Grade Three students can only answer 46% of simple subtraction problems correctly on average and only 72% of addition problems: all of which should have been understood by the end of Grade One. The lack of facility for basic numeracy skills sets in doubt Timorese students’ ability to cope with an increasingly stringent curriculum in later years.

Language use was one of the most concerning aspects of mathematics education revealed by the survey. Although the main language of the classroom is Tetum, students’ mathematics textbooks are in Portuguese. Some students did not speak enough Tetum or Portuguese to complete the survey without aid of a translator for their local languages. This linguistic diversity within the classroom indicates that further research into how Timorese students learn and in which languages is required.

Survey results call for immediate action to ensure that all students in east Timor have access to a numerate future. Specific recommendations to be considered are as follows:

Language:• Research is urgently required into language practices in the classroom. Timor Leste is a linguistically

rich country and students answered some sub-tests in the EGMA survey in up to three languages, indicating a substantial exposure to language in their community. In order to understand how and why students are learning, it must be known in what language and combination of languages they are actually learning in, rather than supposed to be learning according to curriculum mandates and workbooks provided.

32


Conceptual:• Further research is recommended into the confidence and skills of teachers and students to use

effectively the current mathematics student workbooks. These are currently used as textbooks and in Portuguese language.

• Further research into students’ ability to use the workbooks is required. Based on the findings of the 2011 EGRA (Early Grade Reading Assessment) Survey, and the poor facility of students in the Portuguese language there is substantial cause for concern that students may not be able to use these materials effectively. The ownership of mathematics texts was only self-reported by students at around 80%, despite a national distribution and the ownership of the text was not associated with a statistically significant increase in performance. These results suggest that research into the use, distribution and implementation of these resources is warranted.

• The establishment of mathematics benchmarks to monitor classroom improvements. These benchmarks can be used to assess students at-risk of falling further behind by a process of continuous assessment. It is important to consider that initial standards should be considered temporary reference standards since not enough is currently known about when and at what rate Timorese children should progress in learning mathematics. In this sense, benchmark standards should not be seen as high-stakes but an essential piece to monitor mathematics progression in the classroom. One way of doing this would be to use the percentage of zero-score students in selected sub-tests as a marker and track reductions in the shares at least biannually. Monitoring achievements over time will eventually provide more information on the rate and the way in which average mathematics skill develops in Timor. A modified version of the test could be used to screen students during the school year in need of additional support.

Teaching , Teachers and the Classroom:• Support for implementing and sourcing mathematics resources in a students’ main language of

instruction is suggested. In the light of evidence that supports early learning in the child’s first language, it would follow that priority should be given to the production of mathematics textbooks/workbooks and stimulus pictures in mother tongue.

• It is recommended that training of teachers, both pre-service and in-service, be designed to improve pedagogical skills in deepening conceptual understandings of mathematics. It is recommended that Escola Basica Deputy Directors and school inspectors be involved in parallel professional development.

• It is highly recommended teachers receive continued ongoing support in using the stimulus, display and manipulative materials that were distributed to all schools in 2010 and were further augmented in 2012. Pedagogy that supports deep understanding through participation, manipulation and interaction is most effective for mathematics learning and teachers need to be supported in achieving this.

• It is recommended that the Ministry should leverage on the national distribution of materials and the EGMA results to identify grade-specific skills that teachers will need pedagogical methods to convey content knowledge to students. Without the pedagogical understanding of how to convey content to students, knowledge transmission cannot occur efficiently.

• Ongoing support in mathematics program planning, teaching methodology and formative assessment is essential. The future of Timorese numeracy is in the hands of its teachers. Improvements in Timorese children’s mathematical skill depends not only on classroom resources but on effective understanding of how to implement curriculum, resources and assessment.

• Research into the confidence and skills of teachers to interpret and teach according to the current mathematics curriculum and teacher guide for mathematics is required. Modified versions of the Classroom Observational Snapshot tool used in other Pacific countries and tailored to the specific needs of the Timorese education system will provide information on the skills of teachers, while teacher surveys will provide elucidation on the thoughts, concerns, backgrounds and skills of these vital components of the Timorese education system.

33

• Curriculum and teacher support to provide further opportunities for students to work in groups with each other is recommended. Students who engaged in group activities for mathematics learning had scores significantly higher than those who did not. Opportunities to engage in this process provide the foundation not only for improving numerical skills, but gaining language and social skills that will benefit Timorese students. However, teachers need to be supported in their efforts to engage their students in meaningful group activities and this should be a priority in future curriculum and lesson planning.

• Teacher encouragement and support to engage in mathematics learning on a daily basis is required. Students who were engaged in mathematics on a daily basis had a significantly higher score in the sub-tests examined than those who did not. Mathematics is a cumulative skill set, each concept building on those learnt previously. Especially in early grades, students are learning patterns of thinking that will be a valuable life-long asset. A healthy 91% of students self-reported that they engaged in mathematics learning daily, however, that leaves a large proportion of students who do not and this statistic may over-report the reality. Ensuring that each child has the opportunity to build on their foundational skills in a consistent daily manner should be a focus of lesson planning and teacher training.

Parents and the Community:• Promoting parent education and involvement is needed, not only to interest parents in their children’s

mathematics education and build accountability into the Timorese education system at the village level. It is also needed to build parents’ confidence, extending their knowledge and strategies that will enable them to engage with and assist their young children in mathematics. Only a single student out of the 1226 surveyed reported that they received assistance with their mathematics homework. Parental involvement and engagement in mathematics is a key method of improving outcomes for Timorese students.

• The evidence in the report suggests that providing opportunities for students to engage in meaningful learning at home will result in benefit to students. Students who reported doing mathematics homework had scores significantly higher than their counterparts. However, this should not be construed as an argument for large amounts of rote-learning to be sent home with children on a regular basis. The importance of parental engagement has been outlined above, however when only a single student in more than 1200 reported having assistance with their homework expectations of Timorese parents must be realistic. On a related note, 39% of students reported that their parents sell things in the market. This statistic indicates that a basic level of numeracy is prevalent in the community and that the community understands the need for numeracy skills. Providing children with the opportunity to engage in active learning at home through meaningful homework activities will have several effects: to encourage and cement concepts learnt in school, to engage parents and the community with the importance of numeracy as a concept and to increase children’s’ confidence in their own mathematical abilities. Examples of such activities may include assisting parents at market or in shopping, making change, counting and arranging items in groups in the home or market, correlating and observing sports scores or engaging in word problems with the assistance of every day objects (a skill many Timorese students struggled with).

Conclusions and Recommendations

34


3535

Berch, D. B. (2005). Making sense of number sense: Implications for children with mathematical disabilities. Journal of Learning Disabilities, 38(4), 333–339.

Clarke, B., Baker, S., Smolkowski, K., & Chard, D. J. (2008). An analysis of early numeracy curriculum-based measurement: Examining the role of growth in student outcomes.

De Silva, S., Guacgonolle, L. (2012). The Timor Leste 2011 EGRA: Tetum Pilot Results. The World Bank.

EGMA Malawi – Student stimulus sheets. Version 5.0, Malawi, October 2010. Retrieved from https://www.eddataglobal.org/documents/index.cfm?fuseaction=pubDetail&ID=307 (Retrieved April 2011)

EGMA Rwanda - Primary 4 – Enumerator Protocol – Pilot Instrument – January 2011. Retrieved from https://www.eddataglobal.org/documents/index.cfm?fuseaction=pubDetail&ID=307 (Retrieved April 2011)

EGMA Rwanda - Primary 4 – Student Sheets – Pilot Instrument – January 2011. Retrieved from https://www.eddataglobal.org/documents/index.cfm?fuseaction=pubDetail&ID=307 (Retrieved April 2011)

Eddata II. Early Grade Mathematics Assessment (EGMA): Administrator Instructions and Protocol. v.1 12/2008. Contract Number EHC-E-02-o4-00004-00 (Retrieved Oct. 2010)

Garnett, K. Developing fluency with basic number facts: Intervention for students with learning disabilities. Learning Disabilities Research and Practice (1992), 7:210-216.

Gervasoni, A. Extending Mathematical Understanding. Middle Years Specialist Teachers Manual. Australian Catholic University, Draft Edition, 2009.

Glewwe, P & Kremer, M. (2005). Schools, Teachers and Education Outcomes in Developing Countries. Centre for International Development at Harvard University Working Paper No. 122. September 2005

Griffin, S. (2004). Teaching Number Sense. Educational Leadership, 61(5), 39 – 42

Griffin, S. & Case, R. (1997). Re-thinking the primary school math curriculum. An approach based on cognitive science. Issues in Education, 3(1) 1-49.

Huayu Sun (2008). Chinese Young Children’s Strategies on Basic Addition Facts. Retrieved from http://www.merga.net.au/documents/RP602008.pdf (May 2011).

Kilpatrick, J., Swafford, J. & Findell, Bradford. Mathematics Learning Study Committee. Adding It Up (Helping Children Learn Mathematics). National Research Council [NRC], Washington, 2001),

Mansory, A. (2010). Do Children Learn in Afghan Schools? – Assessent of math and language achievements of students at the end of Grades 3 and 6 in SCA supported schools. Swedish Committee for Afghanistan (SCA), Education Technical Support Unit, Kabul, Afghanistan.

Ministry of Education National Education Strategic Plan, 2011 – 2030, 2nd Draft.

REFERENCES

36


36

The Timor-Leste 2011 EGRA: Tetum Pilot Results

Ministry of Education, Timor Leste, Deomocratic Republic of Timor-Leste, Dili.

Ministry of Education National Education Policy, 2007 – 2012. Ministry of Education, Timor Leste, Democratic Republic of Timor-Leste, Dili.

RTI International: Crouch, L & Reubens, A. (2009). Early Grade Mathematics Assessment (EGMA): A Conceptual Framework Based on Mathematics Skills Development in Children.

RTI International, 2011. Core EGMA, January 2011 Retrieved from https://www.eddataglobal.org/documents/index.cfm/Core%20EGMA%20%2D%20January%202011.pdf?fuseaction=throwpub&ID=305,https://www.eddataglobal.org/documents/index.cfm?fuseaction=pubDetail&ID=305 (April 2011)

RTI International. 2009. Early Grade Reading Assessment Toolkit. Prepared by RTI for The World Bank, Office of Human Development. Available at https://www.eddataglobal.org/documents/index.cfm?fuseaction=pubDetail&ID=149

(accessed September 03, 2010).

RTI International (Prepared by Andrea Rubens). 2009. Preliminary Snapshot of some of the tasks in the Early Grade Mathematics Assessment (EGMA) Data. EdData II Technical and Managerial Assistance, Task Number 2, Contract Number EHC-E-00-04-00004-00, Strategic Objective 3, December 2009.

RTI International (Prepared by Andrea Rubens and Tracy Kline). 2009. Pilot of the Early Grade Mathematics Assessment, Final Report, Dec 23, 2009. Prepared for review by the United States Agency for International Development under task order: EdData !! Technical and Managerial Assistance, Task Number 2, Contract Number EHC-É-02-04-00004-00, Strategic Objective 3, December 23, 2009.

UNDP Timor-Leste Human Development Report 2011.

World Bank (2010): An analysis of early grade reading acquisition in Timor Leste: Report of the Early Grade Reading Assessment (EGRA) 2009.

Wright, R., Martland, J. & Stafford, A. (2000). Early Numeracy: Assessment for teaching and intervention. London: Paul Chapman Publishing

3737

Annex 1: School and Student Specific Factors

Numbers Identified %


Did not attend Preschool 0.6103 0.2808 0 1 0.6667 765 0.5904 0.6303

Attended Preschool 0.5830 0.2850 0 1 0.6667 459 0.5569 0.6090

Total 0.6001 0.2826 0 1 0.6667 1224 0.5842 0.6159

Word Problems %




Total 0.4580 0.3128 0 1 0.5000 1213 0.4404 0.4756

Missing Numbers %

Mean SD Min Max Median N LCL

Did not attend Preschool 0.3559 0.2726 0 1 0.3000 764 0.3366

Attended Preschool 0.3379 0.2688 0 1 0.3000 457 0.3132

Total 0.3491 0.2712 0 1 0.3000 1221 0.3339

Additions %




Total 0.4861 0.3896 0 1 0.6000 1219 0.4643 0.5080

Subtractions %




Total 0.2949 0.3482 0 1 0.2000 1208 0.2752 0.3145

38




No Maths Daily 0.4553 0.2734 0 1 0.4167 110 0.4042 0.5064

Maths Daily 0.6144 0.2796 0 1 0.6667 1114 0.5980 0.6308

Total 0.6001 0.2826 0 1 0.6667 1224 0.5842 0.6159

Word Problems %


No Maths Daily 0.3977 0.2939 0 1 0.5000 110 0.3428 0.4526

Maths Daily 0.4640 0.3141 0 1 0.5000 1103 0.4454 0.4825

Total 0.4580 0.3128 0 1 0.5000 1213 0.4404 0.4756

Missing Numbers %


No Maths Daily 0.2773 0.2305 0 1 0.2000 110 0.2342 0.3204

Maths Daily 0.3563 0.2740 0 1 0.3000 1111 0.3401 0.3724

Total 0.3491 0.2712 0 1 0.3000 1221 0.3339 0.3644

Additions %


No Maths Daily 0.3339 0.3267 0 1 0.2000 109 0.2726 0.3953

Maths Daily 0.5011 0.3921 0 1 0.6000 1110 0.4780 0.5242

Total 0.4861 0.3896 0 1 0.6000 1219 0.4643 0.5080

Subtractions %


No Maths Daily 0.2224 0.2937 0 1 0.0000 107 0.1668 0.2781

Maths Daily 0.3019 0.3524 0 1 0.2000 1101 0.2811 0.3227

Total 0.2949 0.3482 0 1 0.2000 1208 0.2752 0.3145

39



Does not own text 0.5184 0.2879 0 1 0.5000 240 0.4820 0.5548

Owns text 0.6200 0.2778 0 1 0.7500 984 0.6026 0.6374

Total 0.6001 0.2826 0 1 0.6667 1224 0.5842 0.6159

Word Problems %


Does not own text 0.4078 0.3191 0 1 0.5000 236 0.3671 0.4486

Owns text 0.4701 0.3102 0 1 0.5000 977 0.4506 0.4895

Total 0.4580 0.3128 0 1 0.5000 1213 0.4404 0.4756

Missing Numbers %


Does not own text 0.3189 0.2696 0 1 0.3000 238 0.2847 0.3532

Owns text 0.3565 0.2713 0 1 0.3000 983 0.3395 0.3734

Total 0.3491 0.2712 0 1 0.3000 1221 0.3339 0.3644

Additions %


Does not own text 0.3908 0.3758 0 1 0.4000 239 0.3431 0.4384

Owns text 0.5094 0.3895 0 1 0.6000 980 0.4850 0.5338

Total 0.4861 0.3896 0 1 0.6000 1219 0.4643 0.5080

Subtractions %


Does not own text 0.2542 0.3149 0 1 0.0000 236 0.2141 0.2944

Owns text 0.3047 0.3553 0 1 0.2000 972 0.2824 0.3271

Total 0.2949 0.3482 0 1 0.2000 1208 0.2752 0.3145

Attachment

40




Does not use objects 0.5822 0.2904 0 1 0.6667 306 0.5497 0.6148

Uses objects 0.6060 0.2799 0 1 0.6667 918 0.5879 0.6241

Total 0.6001 0.2826 0 1 0.6667 1224 0.5842 0.6159

Word Problems %



Uses objects 0.4602 0.3153 0 1 0.5000 911 0.4397 0.4807

Total 0.4580 0.3128 0 1 0.5000 1213 0.4404 0.4756

Missing Numbers %



Uses objects 0.3510 0.2693 0 1 0.3000 915 0.3336 0.3685

Total 0.3491 0.2712 0 1 0.3000 1221 0.3339 0.3644

Additions %



Uses objects 0.4912 0.3925 0 1 0.6000 914 0.4658 0.5167

Total 0.4861 0.3896 0 1 0.6000 1219 0.4643 0.5080

Subtractions %



Uses objects 0.2985 0.3541 0 1 0.2000 910 0.2755 0.3215

Total 0.2949 0.3482 0 1 0.2000 1208 0.2752 0.3145

41



Does not work with others 0.5354 0.2950 0 1 0.5833 205 0.4950 0.5757

Works with others 0.6131 0.2784 0 1 0.6667 1019 0.5960 0.6302

Total 0.6001 0.2826 0 1 0.6667 1224 0.5842 0.6159

Word Problems %




Total 0.4580 0.3128 0 1 0.5000 1213 0.4404 0.4756

Missing Numbers %




Total 0.3491 0.2712 0 1 0.3000 1221 0.3339 0.3644

Additions %




Total 0.4861 0.3896 0 1 0.6000 1219 0.4643 0.5080

Subtractions %




Total 0.2949 0.3482 0 1 0.2000 1208 0.2752 0.3145

Attachment

42




Does not do homework 0.4387 0.2665 0 1 0.4167 121 0.3912 0.4862

Does homework 0.6178 0.2788 0 1 0.6667 1103 0.6013 0.6342

Total 0.6001 0.2826 0 1 0.6667 1224 0.5842 0.6159

Word Problems %



Does homework 0.4719 0.3104 0 1 0.5000 1093 0.4535 0.4903

Total 0.4580 0.3128 0 1 0.5000 1213 0.4404 0.4756

Missing Numbers %



Does homework 0.3611 0.2727 0 1 0.3000 1100 0.3450 0.3772

Total 0.3491 0.2712 0 1 0.3000 1221 0.3339 0.3644

Additions %



Does homework 0.5086 0.3894 0 1 0.6000 1098 0.4855 0.5316

Total 0.4861 0.3896 0 1 0.6000 1219 0.4643 0.5080

Subtractions %


Does not do homework 0.1678 0.2382 0 0.8 0.0000 118 0.1248 0.2108

Does homework 0.3086 0.3555 0 1 0.2000 1090 0.2875 0.3297

Total 0.2949 0.3482 0 1 0.2000 1208 0.2752 0.3145

43



Child does not sell 0.6042 0.2780 0 1 0.6667 752 0.5843 0.6240

Child sells items 0.5936 0.2901 0 1 0.6667 472 0.5674 0.6197

Total 0.6001 0.2826 0 1 0.6667 1224 0.5842 0.6159

Word Problems %




Total 0.4580 0.3128 0 1 0.5000 1213 0.4404 0.4756

Missing Numbers %




Total 0.3491 0.2712 0 1 0.3000 1221 0.3339 0.3644

Additions %




Total 0.4861 0.3896 0 1 0.6000 1219 0.4643 0.5080

Subtractions %




Total 0.2949 0.3482 0 1 0.2000 1208 0.2752 0.3145

Attachment

44






Total 0.6001 0.2826 0 1 0.6667 1224 0.5842 0.6159

Word Problems %




Total 0.4580 0.3128 0 1 0.5000 1213 0.4404 0.4756

Missing Numbers %




Total 0.3491 0.2712 0 1 0.3000 1221 0.3339 0.3644

Additions %




Total 0.4861 0.3896 0 1 0.6000 1219 0.4643 0.5080

Subtractions %




Total 0.2949 0.3482 0 1 0.2000 1208 0.2752 0.3145

45



Child did not eat breakfast 0.5546 0.2751 0 1 0.5833 139 0.5088 0.6003

Child ate breakfast 0.6059 0.2832 0 1 0.6667 1085 0.5891 0.6228

Total 0.6001 0.2826 0 1 0.6667 1224 0.5842 0.6159

Word Problems %




Total 0.4580 0.3128 0 1 0.5000 1213 0.4404 0.4756

Missing Numbers %




Total 0.3491 0.2712 0 1 0.3000 1221 0.3339 0.3644

Additions %




Total 0.4861 0.3896 0 1 0.6000 1219 0.4643 0.5080

Subtractions %




Total 0.2949 0.3482 0 1 0.2000 1208 0.2752 0.3145

Attachment

46




Child does not eat breakfast everyday 0.5561 0.2617 0 1 0.5833 101 0.5051 0.6071

Child eats breakfast everyday 0.6040 0.2842 0 1 0.6667 1123 0.5874 0.6207

Total 0.6001 0.2826 0 1 0.6667 1224 0.5842 0.6159

Word Problems %




Total 0.4580 0.3128 0 1 0.5000 1213 0.4404 0.4756

Missing Numbers %




Total 0.3491 0.2712 0 1 0.3000 1221 0.3339 0.3644

Additions %




Total 0.4861 0.3896 0 1 0.6000 1219 0.4643 0.5080

Subtractions %




Total 0.2949 0.3482 0 1 0.2000 1208 0.2752 0.3145

47



Child does not participate in merenda escolar

0.5249 0.2821 0 1 0.5000 57 0.4516 0.5981

Child participates in merenda escolar 0.6038 0.2822 0 1 0.6667 1167 0.5876 0.6200

Total 0.6001 0.2826 0 1 0.6667 1224 0.5842 0.6159

Word Problems %



0.4035 0.2825 0 1 0.5000 57 0.3302 0.4768


Total 0.4580 0.3128 0 1 0.5000 1213 0.4404 0.4756

Missing Numbers %



0.2875 0.2783 0 1 0.2500 56 0.2146 0.3604


Total 0.3491 0.2712 0 1 0.3000 1221 0.3339 0.3644

Additions %



0.3965 0.3955 0 1 0.4000 57 0.2938 0.4992


Total 0.4861 0.3896 0 1 0.6000 1219 0.4643 0.5080

Subtractions %



0.2036 0.3086 0 1 0.0000 56 0.1227 0.2844


Total 0.2949 0.3482 0 1 0.2000 1208 0.2752 0.3145

Attachment

48




Child's family does not own TV 0.5853 0.2802 0 1 0.6667 727 0.5649 0.6056

Child's family owns TV 0.6217 0.2851 0 1 0.7500 497 0.5967 0.6468

Total 0.6001 0.2826 0 1 0.6667 1224 0.5842 0.6159

Word Problems %




Total 0.4580 0.3128 0 1 0.5000 1213 0.4404 0.4756

Missing Numbers %




Total 0.3491 0.2712 0 1 0.3000 1221 0.3339 0.3644

Additions %




Total 0.4861 0.3896 0 1 0.6000 1219 0.4643 0.5080

Subtractions %




Total 0.2949 0.3482 0 1 0.2000 1208 0.2752 0.3145

49



Child's family does not own radio 0.5816 0.2788 0 1 0.6667 534 0.5580 0.6053

Child's family owns radio 0.6144 0.2849 0 1 0.7500 690 0.5931 0.6356

Total 0.6001 0.2826 0 1 0.6667 1224 0.5842 0.6159

Word Problems %




Total 0.4580 0.3128 0 1 0.5000 1213 0.4404 0.4756

Missing Numbers %




Total 0.3491 0.2712 0 1 0.3000 1221 0.3339 0.3644

Additions %




Total 0.4861 0.3896 0 1 0.6000 1219 0.4643 0.5080

Subtractions %




Total 0.2949 0.3482 0 1 0.2000 1208 0.2752 0.3145

Attachment

50


51

Annex Two: EGMA Instrument

52


53

Avaliasaun Matemátika ba Grau Sedu

(Early Grade Mathematics Assessment - EGMA)

Timor-LesteInstrusaun Sira ba Enumeradór

2° Esbosu

JUNE 2011

Attachment

MINISTÉRIO DA EDUCAÇÃO

54


Instrusaun JerálImportante maka estabelese uluk lai relasaun ne’ebé haksolok no hakmatek ho labarik sira ne’ebé ita atu avalia, liuhosi konversa dahuluk ne’ebé simples kona-ba asuntu sira ne’ebé labarik ne’e gosta. Labarik ne’e sei persebe avaliasaun ne’ebé tuirmai ne’e nu’udar jogu ida ne’ebé halo nia kontente duké hanesan situasaun ne’ebé ladi’ak bá nia.

Konsente Verbál Lee tekstu iha kaixa laran ne’e ho klaru ba labarik ne’e:

Molok atu hahu, ha’u hakarak dehan ha’u naran ba ó. Ha’u naran ...............Ha’u serbisu iha Ministériu Edukasaun.

Ami hakarak atu hatene kona-ba oinsá labarik sira aprende matemátika. Ami hili ó tuir sorte, hanesan liuhosi rifa ka sortéiu.

Ami presiza ó nia tulun ba ida ne’e. Maibé se ó lakohi atu hola parte mós laiha buat ida.

Ita sei halimar jogu balun kona-ba sura no kona-ba númeru.

Hodi uza tempu iha relójiu ne’e, ha’u sei haree ó sura to’o wainhira maka hotu.

Ida ne’e LA’OS ezame no LA’OS atu halo ó pasa ba klase iha eskola.

Ha’u mós sei husu ó kona-ba ó nia familia, hanesan buat hirak ne’ebé ó halo hamutuk ho ó nia familia iha uma.

Ha’u sei LA hakerek ó nia naran atu nune’e laiha ema ida maka atu hetene katak ó nia resposta sira maka ne’e.

Dala ida tan, se ó lakohi lalika hola parte iha hasoru malu ne’e.

Wainhira ita hahú, se ó lakohi atu hatán pergunta ruma, laiha buat ida.

Entaun, ó hakarak atu partisipa? LOOS (se hakarak) Ó pronto atu hahú?

(se la hetan konsente verbál, fó obrigadu ba labarik ne’e no muda fali ba labarik seluk, uza formuláriu ne’e nafatin)

55

ATIVIDADE 1: SURA (segundu 60)

A. Data ba avaliasaun: Tempu hahú avaliasaun:

B. Entrevistadór nia naran/ kodigu :

Tempu remata avaliasaun:

C. Eskola nia NARAN: F. Nivel Grau Estudante nian: o 1 = Grau 1o 2 = Grau 2o 3 = Grau 3D. Kódigu Eskola:

E. Sub Distritu: o feto o mane Idade: ___________

F. Distritu:

Attachment

56


Ha’u hakarak atu ó sura. Ha’u sei uza relójiu ne’e. Ha’u sei hatete wainhira maka hahú no wainhira maka para. Hahú sura husi um (1) iha lingua ne’ebé ó prefere atu uza hodi sura. Ó pronto ona? Loos, hahú.

•Seniasuralaloos•Setempu(segundu 60) hotu ona.

Haree ha’u sura. Um, dois, três. Entaun, ha’u hakarak ó sura hanesan ha’u iha lingua ne’ebé ó prefere. Entaun, hahú: um, …

Númeru loos ikus ne’ebé labarik temi: [ ]

Tempu iha relójiu: ____ segundu

Lingua saida maka ita boot uza ba atividade ne’e?

Tetun

[ ]Português

[ ]Indonesia

[ ]Seluk

[ ]

Se ita boot uza lingua seluk ba atividade ne’e, indika lingua saida maka uza ne’e:

Lingua saida maka labarik uza liuliu ba atividade ne’e?Hili ida deit ho marka [X]

Tetun

[ ]Português

[ ]Indonesia

[ ]Seluk

[ ]

Se labarik uza lingua seluk ba atividade ne’e, indika lingua saidamaka uza ne’e:

57

ATIVIDADE 2-2: SURA 1:1 KORESPONDÉNSIA SURAT TAHAN 2-2

Tau SURAT TAHAN 2-2 ho sírkulu 30 iha labarik nia oin. Hatudu ita boot nia liman husi karuk bá kuana iha sírkulu sira nia leten.

•Selabariksuralaloos•Seniasurasíkuluidadalarua,karuadalaida•Setempu(segundu 60) hotu ona.

Sírkulu sira seluk maka ne’e. Ha’u hakarak ó atu hatudu ba sírkulu sira ho ó nia liman-fuan no sura sírkulu sira.

Hatudu ita-boot nia liman-fuan ba sírkulu sira.Hahú iha ne’e no sura sírkulu sira.Loos, hahú.

Wainhira labarik remato ho sura husu nia lalais:Sírkulu hira maka ó sura ona?

Sírkuluikusne’ebésuraloloos: [ ]

Númerusírkulusirane’ebélabarikdehanniasura ona:

[ ]



Tetun

[ ]Português

[ ]Indonesia

[ ]Seluk

[ ]



Tetun

[ ]Português

[ ]Indonesia

[ ]Seluk

[ ]

ATIVIDADE 2-1: SURA. 1:1 KORESPONDÉNSIA EZEMPLU PRÁTIKA

SURAT TAHAN 2-1

Tau SURAT TAHAN 2-1 ho sírkulu haat iha labarik nia oin. Hatudu ita boot nia liman husi karuk bá kuana iha sírkulu sira.

Iha sírkulu balu iha ne’e. Ha’u hakarak ó atu hatudu ba sírkulu ho ó nia liman-fuan no sura sírkulu sira ne’e.Hatudu ba sírkulu dahuluk ho ita-boot nia liman-fuan. Hahú iha ne’e no sura sírkulu sira ne’e.Wainhira labarik remato ho sura husu nia lalais: Sírkulu hira maka ó sura ona?

Ne’e loos, 4.

Hatudu bá kada sírkulu ho ita boot nia liman-fuan no sura : Um, dois, três, quatro. Iha sírkulu quatro. Mai ita halo fali jogu seluk.

Attachment

58


ATIVIDADE 3: IDENTIFIKASAUN NÚMERU SIRA

SURAT TAHAN 3-1 SURAT TAHAN 3-2

(segundu 60)

Tau SURAT TAHAN 2-2 ho sírkulu 30 iha labarik nia oin. Hatudu ita boot nia liman husi karuk bá kuana iha sírkulu sira nia leten.

•Selabarikhalosala3tuituirmalu•Setempu(segundu 60) hotu ona.

•Selabarikparaihanúmeruidadurantesegundu5 nia laran.

Númeru balun maka ne’e. Ha’u hakarak ó atu hatudu ó nia liman-fuan ba númeru ida-idak no dehan mai ha’u númeru saida maka ne’e. Ha’u sei uza relójiu no sei fó hatene ó wainhira maka haú no wainhira maka remata.

Hatudu ho ó nia liman-fuan bá númeru primeiru.Hahú iha ne’e. Loos, hahú.

... Se labarik para iha númeru ida durante segundu 5 nia laran – hatudu ho ó nia liman-fuan bá númeru tuirmai. Númeru ne’e, númeru ne’e saida?

Kontinua bá SURAT TAHAN 3-2

Total resposta ne’ebé loos: [ ]



Tetun

[ ]Português

[ ]Indonesia

[ ]Seluk

[ ]



Tetun

[ ]Português

[ ]Indonesia

[ ]Seluk

[ ]

Se labarik uza lingua seluk ba atividade ne’e, indika lingua saidamaka uza ne’e:

LOOS LALOOS, LAIHA RESPOSTA

1 2

2 5

3 9

4 13

5 10

6 18

LOOS LALOOS, LAIHA RESPOSTA

7 65

8 50

9 97

10 104

11 468

12 6,430

59

ATIVIDADE 4A: KOMPARASAUN KUANTIDADE EZEMPLU PRÁTIKA P1, P2

objetu konta nian

EZEMPLU PRÁTIKA P1. Uza objetu balun hodi kria grupo rua. Grupu karuk iha objetu 5 no grupo kuana iha objetu 2. Wainhira ita-boot kria hotu ona grupu sira, hatudu grupu sira ho ita-boot nia liman-fuan.

Haree bá grupu rua ne’e. Dehan mai ha’u ida ne’ebé maka boot liu? Hatudu ho ó nia liman-fuan bá grupu ne’ebé boot liu.

Hatudu grupu 5. Grupu ne’é mak boot liu . 5 boot liu 2.

EZEMPLU PRÁTIKA P2. Kria grupo rua. Grupo karuk iha objetu 1 no grupu kuana iha objetu 3. Wainhira ita-boot kria hotu ona grupu sira, hatudu grupu sira ho ita-boot nia liman-fuan.

Haree bá grupu rua ne’e. Dehan mai ha’u ida ne’ebé maka boot liu? Hatudu ho ó nia liman-fuan bá grupu ne’ebé boot liu.

Hatudu grupu 3. Grupu ne’é mak boot liu . 3 boot liu 1.

ATIVIDADE 4: KOMPARASAUN KUANTIDADE

SURAT TAHAN 4-1 SURAT TAHAN 4-2

Tau objetu sira bá sorin no tau SURAT TAHAN 4-1 iha labarik nia oin.

•Selabarikhalosala3tuituirmalu.

•Selabariklarespondehafoinsegundu5liuona.

Agora, mai ita koko atu uza númeru balun. Haree bá númeru sira ne’e. Dehan mai ha’u ida ne’ebé maka boot liu.


RESPOSTA LOOS LOOS LALOOS, LAHATENE,

LAIHA RESPOSTA

1 4 2 (4)

2 7 8 (8)

3 14 17 (17)

4 19 18 (19)

5 40 96 (96)

6 79 70 (79)

7 32 36 (36)

8 65 56 (65)

9 145 163 (163)

10 1,400 1,235 (1,400)


Attachment

60


ATIVIDADE 5: NÚMERU FALTA

SURAT TAHAN 5-P SURAT TAHAN 5-1 SURAT TAHAN 5-2

EZEMPLU PRÁTIKA P2. Tau SURAT TAHAN 5-P iha labarik nia oin.Hatudu ita-boot nia liman husi karuk bá kuana. Hatudu ho ita-boot nia liman-fuan bá liña.

Númeru balun maka ne’e. Um, dois, três … Númeru saida maka mai iha ne’e?

Ne’e loos, QUATRO. Mai ita kontinua nafatin.

Ne’e, númeru QUATRO maka ne’e. Sura hamutuk ho ha’u. Hatudu ho ita-boot nia liman-fuan bá kada númeru. Um, dois, três, QUATRO . Ne’e, númeru QUATRO maka ne’e. Mai ita kontinua.

Tau SURAT TAHAN 5-1 iha labarik nia oin. Hatudu ita-boot nia liman husi karuk bá kuana bá pergunta.

•Selabarikhalosala3tuituirmalu.

•Selabariklarespondehafoinsegundu10liuona.

Númeru sira seluk maka ne’e.

Hatudu ho ita-boot nia liman bá liña.Dehan mai ha’u númeru saida maka mai iha ne’e.

Kontinua kada pergunta. Bá kada pontu pergunta iha liña nia okos no ho liña no hatete:Dehan mai ha’u númeru saida maka mai iha ne’e.


RESPOSTA LOOS LOOS LALOOS, LAHATENE,

LAIHA RESPOSTA

1 4 5 6 7 7

2 8 9 10 11 11

3 17 18 19 18

4 89 90 91 89

5 20 30 40 40

6 4 3 2 1 1

7 100 200 300 400 400

8 2 4 6 8 8

9 10 15 20 25 20

10 500 400 300 200 300


61

ATIVIDADE 6: PROBLEMA SIRA

Lapis, surat tahan, no objetu konta nian

EZEMPLU PRÁTIKA P2. Tau SURAT TAHAN 5-P iha labarik nia oin.Hatudu ita-boot nia liman husi karuk bá kuana. Hatudu ho ita-boot nia liman-fuan bá liña.

Númeru balun maka ne’e. Um, dois, três … Númeru saida maka mai iha ne’e?

Ne’e loos, QUATRO. Mai ita kontinua nafatin.

Ne’e, númeru QUATRO maka ne’e. Sura hamutuk ho ha’u. Hatudu ho ita-boot nia liman-fuan bá kada númeru. Um, dois, três, QUATRO . Ne’e, númeru QUATRO maka ne’e. Mai ita kontinua.

Muda dook husi livriñu pájina estudante nian. •Selabarikhalosaladala2tuituirmalu.

•Se labarik la responde ba pergunta hafoin segundu 10 liu ona, repete pergunta dala ida, hein to’o segundu 20, no muda bá pergunta tuirmai.

Ha’u iha problema balun ne’ebé ha’u hakarak ó atu rezolve. Iha ne’e sasán sira balun ne’ebé bele tulun ó atu sura.

Hatudu lapis, surat tahan, no objetu konta nian bá lababarik.

Ó bele uza sasán sira ne’e se ó hakarak, maibé la obriga ó. Rona ho atensaun didi’ak. Se ó hakarak, ha’u bele repete fali pergunta. Mai ita hahú.

RESPOSTE LABARIK

NIAN

RESPOSTA LOOS

LOOS LALOOS, LAHATENE,

LAIHA RESPOSTA

Labarik ne’e:

Liman fuan

Uza objetu konta nian

Halo kalkulasaun eskrita

1 Maria iha hudi-tasak fuan 4. Nia mamá iha fuan 5 tán. Hudi-tasak hamutuk hotu fuan hira maka agora sira na’in rua iha?

9 (4)

2 José iha hudi-tasak fuan 6. Nia hán tiha hudi-tasak fuan 3. Hudi-tasak fuan hira maka agora hela?

3 (17)

3 Diva iha lapis 8. Pedro iha lapis 3. Diva lori lapis hira mak liu Pedro?

5 (96)

4 Ha’u iha rebusada fuan 3. Ha’u hakarak fó rebusada ida-ida bá ha’u nia kolega na’in 9. Ha’u presiza hira tan?

6 (36)


Attachment

62


ATIVIDADE 7-P: ADISAUN NO SUBTRASAUNEZEMPLU PRÁTIKA P1 & P2

SURAT TAHAN 7-P


Tau objetu sira iha labarik nia sorin. Tau surat-tahan SURAT TAHAN 7-P iha labarik nia oin. Se labarik la responde loloos bá pergunta P1, repete pergunta dala ida, hein to’o segundu tolu, no pasa bá pergunta tuirmai.

EZEMPLU PRÁTIKA P1. Agora, ita atu rezolve problema balun kona-bá adisaun no subtrasaunHatudu bá ezersísiu P1. (1+2 = ___)Um mais dois iguala ... ?

LOOS [ ] LALOOS, LAHATENE, LAIHA RESPOSTA [ ]

Ne’e loos, um mais dois iguala três.

Um mais dois igual três. Uza kontadór sira no lee ezersísiu, dudu kontadór bá labarik. Ne’e, ne’e maka um.

Hafoin, dudu kontadór rua bá labarik. Ne’e, ne’e maka dois.Sura kontadór tolu ho lian moos. Um mais dois iguala três. Mai ita halo tán bá problema seluk.

EZEMPLU PRÁTIKA P2. Hatudu bá ezersísiu P2. (3–2 = ___)Três menos dois igual ... ?

LOOS [ ] LALOOS, LAHATENE, LAIHA RESPOSTA [ ]

Ne’e loos, três menos dois igual um ... ?

Nia resposta maka ida.Uza kontadór sira no lee ezersísiu, dud kontadór tolu bá labarik. Ne’e, ne’e maka três.

Hafoin, hasai tiha rua husi kontadór tolu ne’ebé ita-boot tau ona iha labarik nia oin. Menos dois.Sura kontadór ho lian moos. Um. Três menos dois igual um. Mai ita kontinua.

63

ATIVIDADE 7-1: ADISAUN

SURAT TAHAN 7-1 (minutu 2)


Tau SURAT TAHAN 7-1 iha labarik nia oin.

Hatudu ho ita boot nia liman-fuan bá ezersísiu adisaun nian dahuluk iha surat-tahan laran.

•Selabarikhalosaladala3tuituirmalu.•Setempu(minutu 2) hotu ona.

Hein to’o segundu 20, no muda bá pergunta tuirmai.

Iha koluna RESPOSTE LABARIK NIAN hakerek númeru ne’ebé labarik temi. Se labarik la fó reposta to ka nia la hatene, hakerek ( - ).

Lee ba labakik …

RESPOSTE LABARIK

NIAN

RESPOSTA LOOS


LAIHA RESPOSTA

Labarik ne’e:

Liman fuan


Halo kalkulasaun

eskrita

1

4+5 =Quatro mais dois igual… ?

9

2

8 +2=Oito mais dois igual… ?

10

3

20+4=Vinte mais quatro igual …?

24

4

13+12 =Treze mais doze igual … ?

25

5

11+9= Onze mais nove igual … ?

20


Tempu iha relójiu: ____ : _____

Se ita boot uza lingua seluk (ne’ebe laós Portugues) bá pergunta ne’e, indika lingua saida mak ita boot uza:

Attachment

64


ATIVIDADE 7-2: SUBTRASAUN

SURAT TAHAN 7-2 (minutu 2)


Tau SURAT TAHAN 7-2 iha labarik nia oin.

Hatudu ho ita boot nia liman-fuan bá ezersísiu adisaun nian dahuluk iha surat-tahan laran.

•Selabarikhalosaladala3tuituirmalu.•Setempu(minutu 2) hotu ona.

Hein to’o segundu 20, no muda bá pergunta tuirmai.

Iha koluna RESPOSTE LABARIK NIAN hakerek númeru ne’ebé labarik temi. Se labarik la fó reposta to ka nia la hatene, hakerek ( - ).

Lee ba labakik …

RESPOSTE LABARIK

NIAN

RESPOSTA LOOS


LAIHA RESPOSTA

Labarik ne’e:

Liman fuan


Halo kalkulasaun

eskrita

1

9-5 =Nove menos cinco igual… ? 4

2

10-8=Dez menos oito igual… ?

2

3

24-4=Vinte quatro menos quatro igual …?

20

4

25-12 =Vinte cinco menos doze igual ..?

13

5

20-9= Vinte menos nove igual … ?

11


Tempu iha relójiu: ____ : _____

Se ita boot uza lingua seluk (ne’ebe laós Portugues) bá pergunta ne’e, indika lingua saida mak ita boot uza:

65

REMATA ONA EZERSÍSIU EGMA – HAKEREK TEMPU REMATA NIAN IHA PÁJINA 1 MOLOK MUDA BÁ KESTIONÁRIU.

KESTIONÁRIU

Ita besik atu hotu ona! Agora iha pergunta oituan tán de’it kona-ba ó nia esperiénsia ho Matemátika.

1 Molok tama Primeiro Ano ó tama eskola TK ka lae?

LOOS [ ] LAE [ ] LAIHA RESPOSTA [ ] LAHATENE [ ]

2 Iha eskola, imi hetán disiplina Matemátika loron-loron?


3 Ó sente Matemátika difisil ka fasil? DIFISIL [ ] FASIL [ ] LAIHA RESPOSTA [ ]

4 Iha eskola, ó uza livru tekstu Matemátika?


5 Iha eskola dala ruma uza objetu sira hodi aprende no resove problema Matemátika?


6 Iha eskola, dala ruma servisu hamutuk iha grupu hodi aprende no resove problema Matemátika?


7 Ó nia profesór/a fó Traballu Para Kaza Matemátika bá imi?SE LAE BA PERGUNTA 9.


8 Ema ruma tulun ó ho nia Traballu Para Kaza?


9 Ó nia ama ka apa fa’an sasaan (kios/loja/merkadu)?


10 Iha ó nia uma iha televisaun ka lae? LOOS [ ] LAE [ ] LAIHA RESPOSTA [ ] LAHATENE [ ]

11 Iha ó nia uma iha radio ka lae? LOOS [ ] LAE [ ] LAIHA RESPOSTA [ ] LAHATENE [ ]

Remata ona! Ha’u kontente teb-tebes. Agora ó bele fila hirak bá loos klase. Halo favór labele ko’alia ho labara sira seluk ne’ebé atu mai.

SE ITA-BOOT HAKARAK, HAKEREK ITA-BOOT NIA KOMENTÁRIU IHA NE’E BÁ PREUKUPASAUN KONA-BÁ LINGUA NO PROBLEMA SIRA SELUK RUMA NE’EBÉ ITA-BOOT HETAN IHA IMPLEMENTASAUN EVALUASAUN IDA NE’E NIA LARAN

Attachment

66


67

Annex Three: Word Problems Per-Item Analysis

Sub-test 6 asked the student to answer four word problems. The problems were in Tetum and the text is given in Annex 2. An English translation of each of the problems is given as follows.

1. Maria has 4 bananas. Her mother has 5 bananas. How many do they have in total now?

2. Jose has 6 bananas. He eats 3 bananas. How many are left now?

3. Diva has 8 pencils. Pedro has 3 pencils. How many pencils does Diva have more than Pedro?

4. I have 3 sweets [bananas]. How many more bananas do I need if I want to give one sweet [banana] to my 9 friends.

Table 29 gives the percentage of students able to answer each item in this sub-test correctly. Problem 2 was answered correctly most often, which was surprising given that most students found the subtraction sub-test 8 more difficult than sub-test 7 (addition). Problem 1, an addition problem was answered correctly by 64% of students overall. Problem 3, was answered by only 20% of students while Problem 4 was answered by only 27% of students. These were also subtraction problems but required a more complex reasoning.

TABLE 29. Per-item Analysis, Sub-test 6

Grade Statistics Problem 1 Problem 2 Problem 3 Problem 4

1

Mean 0.4078 0.4854 0.1044 0.1117

SD 0.4920 0.5004 0.3061 0.3153

N 412 412 412 412

2

Mean 0.6683 0.7599 0.1881 0.2624

SD 0.4714 0.4277 0.3913 0.4405

N 404 404 404 404

3

Mean 0.8341 0.9000 0.3024 0.4220

SD 0.3724 0.3004 0.4599 0.4945

N 410 410 410 410

Total

Mean 0.6362 0.7145 0.1982 0.2651

SD 0.4813 0.4518 0.3988 0.4416

N 1226 1226 1226 1226

68


70


world bank document...the final draft report. the 2011 timor-leste egma was implemented as a direct...

Documents