sample and organization of work - global reading network · added to make up for differences in...

GUYANA

Early Grade Reading Assessment: October 2008 Results

Prepared by:

RTI International

JULY 2010 GUYANA

GUYANA

Early Grade Reading Assessment: October 2008 Results

Prepared for the World Bank under Contract No. 7146794

RTI International* 701 13th Street, N.W., Suite 750

Washington, DC 20005-3967 Phone: 202-728-2080

*RTI International is a trade name of Research Triangle Institute.

Table of Contents List of Tables ................................................................................................................................. iii

List of Figures ..................................................................................................................................v

Glossary of Statistical Terms ......................................................................................................... vi

Glossary of Reading Terms ......................................................................................................... viii

Executive Summary .........................................................................................................................1

Introduction......................................................................................................................................4

I. Importance of the Study.......................................................................................................5

II. Preparation of the Study and Implementation of Fieldwork................................................6

Capacity Building ............................................................................................................... 6 Sample and Organization of Work ..................................................................................... 6 Instrument ........................................................................................................................... 7

III. Analysis of Results ..............................................................................................................9

Letter Name Knowledge ................................................................................................... 10 Initial Sound Identification and Letter Sound Knowledge ............................................... 13 Familiar Word and Nonsense Word Reading ................................................................... 16 Oral Passage Reading and Comprehension ...................................................................... 20 Reading Comprehension................................................................................................... 23 Listening Comprehension and Dictation .......................................................................... 24 Dictation............................................................................................................................ 27

IV. Student and Teacher Surveys.............................................................................................29

Student Factors Affecting Student Achievement.............................................................. 29 Teacher Factors Affecting Student Achievement ............................................................. 35

V Conclusions........................................................................................................................39

EGRA Conclusions........................................................................................................... 39 Student Factors that Affect Achievement ......................................................................... 40 Teacher Factors Affecting Student Achievement ............................................................. 40 Implications....................................................................................................................... 41 Modified EGRA in the Classroom.................................................................................... 43

Annex A: EGRA Student Response Form.....................................................................................45

Annex B: Guyana EGRA—Student Stimuli Booklet ....................................................................56

i

Annex C: Teacher Questionnaire...................................................................................................63

Annex D: Psychometric Testing of the Instrument........................................................................69

ii

List of Tables Table 1. Distribution of students, by grade and gender .................................................................. 7

Table 2. Reliability of Guyana EGRA............................................................................................ 8

Table 3. Number of cases of students who discontinued................................................................ 9

Table 4. Letter name knowledge, by grade and gender ................................................................ 10

Table 5. Letter name knowledge, by region ................................................................................. 12

Table 6. Initial sound identification, by grade and gender ........................................................... 13

Table 7. Initial sound identification, by grade, gender, and region .............................................. 14

Table 8. Letter sound knowledge, by grade and gender ............................................................... 15

Table 9. Letter sound knowledge, by region................................................................................. 16

Table 10. Familiar word reading, by grade and gender ................................................................ 17

Table 11. Familiar word reading, by region ................................................................................. 17

Table 12. Invented word decoding, by grade and gender ............................................................. 19

Table 13. Invented word decoding, by region .............................................................................. 19

Table 14. Oral reading fluency, by grade and gender................................................................... 21

Table 15. Oral reading fluency, by region .................................................................................... 22

Table 16. Point values of the subsections ..................................................................................... 24

Table 17. Listening comprehension, by grade and gender ........................................................... 25

Table 18. Listening comprehension, by region............................................................................. 26

Table 19. Weighted dictation results, by gender and grade.......................................................... 27

Table 20. EGRA results by home language.................................................................................. 29

Table 21. EGRA results by textbook possession .......................................................................... 30

Table 22. EGRA results by home reading materials..................................................................... 31

Table 23. EGRA results by home reading habits.......................................................................... 32

iii

Table 24. EGRA results by nursery school attendance................................................................. 32

Table 25. EGRA results by homework habits .............................................................................. 33

Table 26. EGRA results by homework help ................................................................................. 34

Table 27. EGRA results by absences............................................................................................ 35

Table 28. Parent involvement in school........................................................................................ 36

Table 29. Teacher experience ....................................................................................................... 36

Table 30. Teacher use of given curriculum .................................................................................. 37

Table 31. Reading-related activities ............................................................................................. 37

Table 32. Teacher beliefs.............................................................................................................. 38

Table 33. Teachers’ use of evaluation .......................................................................................... 39

iv

List of Figures Figure 1. Letter naming by grade and benchmark ........................................................................ 11

Figure 2. Error rate, by letter ........................................................................................................ 12

Figure 3. Initial sound identification, by grade and benchmark ................................................... 14

Figure 4. Familiar word reading, by grade and benchmark.......................................................... 17

Figure 5. Invented word decoding, by grade and benchmark....................................................... 19

Figure 6. Oral reading fluency, by grade and benchmark............................................................. 21

Figure 7. Oral reading fluency, by benchmarks............................................................................ 22

Figure 8. Reading comprehension, percentage correct, by grade and gender .............................. 24

Figure 9. Listening comprehension, by grade and gender ............................................................ 26

Figure 10. Weighted dictation results, by grade ........................................................................... 28

Figure 11. Raw dictation results, by item and grade..................................................................... 28

v

Glossary of Statistical Terms

Analysis of variance (ANOVA): A statistical test that compares the differences between the means of two or more groups and determines if a significant difference exists between variables indicating a relationship between variables

Causal relationship: The inference that two variables are related in some way, meaning that a change in the independent variable leads to a change in the dependent variable(s). This can only happen if all other possible causes are eliminated.

Coefficient: The number or symbol placed before and multiplying another unit. It is constant and distinguished from a variable. It can serve as a measure of a property or characteristic.

Correlation: Strength and direction of a linear relationship between variables

Cronbach’s alpha: Often used as a measure of the internal consistency reliability of a psychometric instrument

Dependent variable: The variable observed as a result of the independent variable being manipulated in a study (or the effect of changing the independent variable)

Dispersion: Quantifiable variation of measurements of differing members of a population

Independent variable: The variable being changed, and its effect on the dependent variable being studied

Internal consistency: Measures whether or not several items that propose to measure the same general construct, or in this case, reading skill, produce similar scores

Interrater reliability: In this case, the extent to which enumerators/assessment administrators are able to produce the same results given the exact (or as close as possible) same task

Mean: The average score of a distribution or group, approximate statistical norm or expected value for a distribution or group

Performance: In this case, how well the student did on any particular subtest

Raw results: Results that are unprocessed or evaluated

Regression analysis: Tool for investigating relationship between dependent and independent variables. It helps explain how the typical value of the dependent variable changes as any independent variable is varied and others remain fixed.

Reliability: The extent to which a measuring procedure yields the same results on repeated trials

Sample: A group of individuals selected from a population, usually intended to represent the population in a study

Standard deviation: A measure of variation that indicates the distance between the scores of a distribution or group from the mean, the average distance between the mean score and other scores in the group. Smaller deviations indicate more scores closer to the mean while larger deviations indicate scores being more spread out and thus more variance among the group.

vi

Statistically significant: Denotes statistical evidence that a difference in distributions or groups does exist, meaning that the result is unlikely to occur by chance

t test: A statistical test that compares the differences in means between two groups, determining if significant differences exist between variables

Validity: The degree to which the study design supports the conclusion that the causal variable (independent variable) caused the effect (in the dependent variable)

Variability: How spread out or closely clustered a set of data is; or, the spread in the variable or probability distribution

Weight: A measure of the relative importance of an item in a distribution or group; weights are added to make up for differences in importance before statistical analysis

vii

viii

Glossary of Reading Terms Alphabetic code/knowledge/principle/process: Familiarity with the alphabet and with the

principle that written spellings systematically represent sounds that can be blended into meaningful words

Automaticity/fluency: The bridge between decoding and comprehension. Fluency in word recognition means that the reader is no longer aware of or needs to concentrate on the mental effort of translating letters to sounds and forming sounds into words. At that point, the reader is decoding quickly enough to be able to focus on comprehension.

Benchmark: A standard of achievement used as a reference and against which other performance can be measured

Connected text: Text that contains a coherent message

Decode/decoding skills: In the reading context, the ability to use the knowledge of spelling-sound relationships and the pronunciation of irregular words to derive a pronunciation of written words

Fluency: See automaticity/fluency definition, above

Letter sound correspondence: The principle that each letter represents a unit of sound; the importance is in knowing which sounds are represented by a particular letter

Phoneme: The smallest linguistically distinctive unit of sound allowing for differentiation of two words within a specific language (e.g., “top” and “mop” differ by only one phoneme, but the meaning changes).

Phonological awareness: A general appreciation of the sound structure of language, as demonstrated by the awareness of sounds at three levels of structure: syllables, onsets and rimes, and phonemes

Phonics: Instructional practices that emphasize how spellings are related to speech sounds in systematic ways

Rate (as in reading rate): Degree of speed

Sight words: Words that a child has memorized and can recognize at first sight, or that are so predictable from the context that it causes them to be quickly recognized

wpm: Words per minute, a standard measure of ability to read

Word recognition: Instant identification of words

Working memory: Theoretically, the structure in the mind that briefly stores and processes information

Executive Summary

RTI International was asked to assist the Guyanese Ministry of Education (MoE) in using an early grade reading assessment (EGRA) to assess student performance in English. The World Bank (WB) supports educational quality improvement globally, and has supported this effort. From 2008 to 2009, RTI worked with the MoE to adapt the EGRA instrument to the local context, train enumerators, support data collection, and analyze the results.

The study, conducted by the Guyanese MoE in conjunction with RTI, sought to answer the following questions: (1) “What basic skills for literacy are dominated by second, third, and fourth grade students in Guyanese schools?” and (2) “What factors influence students’ mastery of basic reading skills?” To answer these questions, an instrument was developed with two parts: a diagnostic section to measure reading skills, and a student questionnaire about student background information. Another questionnaire was given to the teachers of the assessed students to fill out on the day of testing. The report following this executive summary presents the results of the study.

The first steps of conducting the study were choosing a sample, adapting the EGRA instrument, and training of enumerators and supervisors to go into the field and correctly collect the data. The MoE, with guidance from RTI and the WB, decided that the sample should include the contrast groups according to region (hinterland and coastland) and grade level (grades 2, 3, and 4). Testing was undertaken at the beginning of the year, and so it is important to note that instruction was from the previous grades (grade 1 instruction if entering grade 2, and so on). Schools and students within these populations were chosen at random. The EGRA instrument was adapted and pretested for reliability and validity, then adjusted for the actual sample. Enumerators and supervisors were trained in how to administer the instrument in a friendly and consistent manner, and enumerator interrater reliability was tested, with results above 90% for all subsections except for initial sound identification, which was above 80%.

The last stages were field collection of the data, data entry, and analysis. Data were collected from a total of 2699 students (1439 female and 1260 male). Analysis was conducted using a variety of tests, including descriptive statistics (means and standard deviations in particular), t tests, and analysis of variance (ANOVA). A high degree of reliability was confirmed using Cronbach’s alpha. The Dynamic Indicators of Basic Early Literacy Skills (DIBELS; a similar assessment used in the United States) benchmarks for the first grade were used for a rough comparison of the level of achievement of Guyanese students to children in other countries. Some of the main results from each subsection (representing a particular foundational reading skill) are found below, followed by implications, a brief list of results from questionnaires, and conclusions. For the following sections, significant gender differences generally favored females, and significant regional differences generally favored the coastland.

Letter name knowledge: The means for this subsection of the instrument were highest, with a mean of 69.9 letters correctly named per minute in fourth grade, and 90% of students meeting the benchmark. There were significant gender differences in results from this subsection. Initial

1

sound identification: The results of this subsection, which tested phonemic awareness, showed low results across grades. Students seemed to have a general lack of understanding of this skill. Letter sound knowledge: The means for this subsection were well below benchmarks, regardless of region, grade, or gender. Familiar word reading: This subsection showed grade progression, but also gender and regional differences. Students seemed to improve from second grade, since 50% of second-grade students scored zero on this subsection (meaning they could not read a single word), but by fourth grade 40% of students met the benchmark. However, even in fourth grade the mean fell considerably below the benchmark. Invented word decoding: The mean for this subsection was quite low and most students scored a zero. Oral passage reading: Although this subsection showed grade progression, there was a gender gap that increased over grades, with girls reading about 8 words per minute faster in the beginning of second grade to over 20 words per minute faster in the beginning of fourth grade. There was also a regional difference favoring the coastland. Reading comprehension: These results varied by question. Listening comprehension: The mean in this subtest increased by grade, and showed improvement over reading comprehension results. Dictation: For this subtest, the mean increased by grade, and female students had higher means in all three grades.

Generally, EGRA results were both positive and negative. Positively, students were making progress from grade to grade. For the most part all scores went up from one grade to the next, which is what we would hope for.

Students were, unfortunately, not making the educational gains in reading that they should be in the three years of schooling assessed. There was a persistent gender gap, with females generally scoring better than males. Furthermore, there was generally a performance difference between the hinterland and coastland, favoring the coastland.

Passage reading and comprehension measures are commonly employed to gauge reading ability. In particular, words per minute (wpm) is a widely applied indicator that measures how well a child can read. This indicator combines all the other reading skills measured in EGRA, including knowledge and names of letters, word reading and decoding, and comprehension skills, as well as some not measured by this assessment, such as vocabulary. In the United States, the benchmark for students is 26–44 wpm in second grade, which climbs to 71–93 wpm in fourth grade. In Guyana, students in grade 2 are, on average, reading only 15 wpm. This number increases to 50 wpm in fourth grade, but is still well below the benchmark. More strikingly, of the total number of all students tested, 29.4% were unable to read a single word of text, corresponding to 46% in grade 2, 28% in grade 3 and 14% in grade 4.

At least one possible contribution to reading difficulty, as revealed in assessment results, was students’ lack of ability to distinguish sounds in a word, and associate letters (symbols representing sounds) with sounds. Some 63.8% of students tested were unable to read a single nonsense word, a measure of decoding ability. Although students were familiar with letter names, most students could not identify the first sounds in words (phonemic awareness). This skill, to hear and separate the sounds in a word, is an essential skill for understanding the relationship between the written and spoken word. Students also were not able to give the sounds of particular letters (phonics), demonstrating that they did not understand the relationship

2

between letters and sounds. Sounds are put together to make words, so students who don’t know these two skills have trouble reading new words, and read more slowly. These difficulties and a slow reading rate then become obstacles to understanding.

Gender differences were seen in all subtests, and across all three grades. A particularly important finding in this area was in the passage reading subtest. Although in second grade, the female and male scores were relatively close, the gap increased as grade level increased.

Factors related to student background and to teachers were also found to be significant. School factors that showed differences among students were student language, attendance both at their current school and at nursery school, and having the necessary materials. Home factors included if students had other reading materials in the home, if they did homework, and if they had help with homework. Teachers’ experiences, beliefs, and use of reading activities all showed a significant effect. Instructionally, students’ means on oral passage reading were higher when teachers reported using evaluations more frequently, used a variety of reading activities more frequently, expected students to master a variety of reading skills earlier, and used the recommended reading texts. A particularly strong result was for schools that identified themselves as having a functioning parent-teacher association (PTA), exhibiting student scores that were higher by almost 9 words per minute.

There are several conclusions that can be drawn from these findings. Although it is generally true that students were not meeting international benchmarks for reading, boys seemed to be most at risk for not acquiring basic reading skills. Boys’ scores were lower than girls’ on many of the skills measured by EGRA, and for some skills they were falling even further behind as they moved up in the grades. Second, lack of knowledge about the sounds in words and letter sounds is likely preventing all students from gaining reading speed and comprehension. Instruction in these aspects of reading could help raise reading levels for all students. Finally, factors outside of school, as well as teacher characteristics, affected reading results. Functional parent-teacher associations are one such example; schools that had them showed significantly higher reading scores.

3

Introduction 1. The World Bank (WB) is supporting the global expansion of educational quality and improvement. RTI International was contracted to assist the Guyana Ministry of Education (MoE) to assess Guyanese students in English, using the Early Grade Reading Assessment (EGRA) tool to inform teacher training and improve student learning outcomes. Specifically, RTI was asked to train field supervisors and MoE personnel to measure student achievement in a meaningful way to inform instruction.

2. Given the wide experience of RTI in this field conducting EGRA in more than a dozen countries and languages, the WB contracted RTI to support this study. Consequently, RTI committed to contribute to technical assistance to support the government in the adaptation of the EGRA instruments to fit the local context, train enumerators, plan and monitor the fieldwork, and analyze results. The technical assistance also built capacity that would allow the government to develop its own EGRA-based assessment program.

3. The study included the Guyana version of EGRA; a questionnaire about students’ lives and factors that may affect their achievement; and a teacher survey concerning teacher experience, knowledge, and beliefs regarding reading development. The Center for Educational Research and Social Action (CIASES), a research and education consulting firm based in Nicaragua, was in turn hired by RTI to develop a data entry system as well as to clean and prepare the data for analysis prior to RTI’s preparation of the report.

4. This report presents the main results of this study, summarizing the initial reading skills demonstrated by the Guyanese children who participated by taking the EGRA and completing the questionnaire.

5. This report is structured into five sections. The first section explains the importance of this type of study to improve the quality of learning in primary schools. The second briefly reviews the results of training, provides information on how the sample was organized, and details aspects of validity of the instrument and methodological issues in implementation. The third section presents the results of data analysis in all sections of the assessment: identification of letters, initial sound recognition, letter sound identification, familiar word reading, invented word reading, oral passage reading, reading comprehension, listening comprehension, and dictation. The fourth section presents the results of the student survey to determine what student factors might positively or negatively be associated with student achievement and the teacher interview results. The fifth section draws conclusions about the most important results in order to discuss some key issues and recommendations for the educational quality of Guyanese schools. Annexes contain copies of the instruments as well as more details on the psychometric testing that was carried out.

4

I. Importance of the Study 6. The ability to read and understand a simple text is one of the most fundamental skills a child can learn. Without basic literacy there is little chance that a child can escape the intergenerational cycle of poverty. Yet in many countries, students enrolled in school for as many as six years are unable to read and understand a simple text. Recent evidence indicates that learning to read both early and at a sufficient rate (with comprehension) is essential for learning to read well. Acquiring literacy becomes more difficult as students grow older; children who do not learn to read in the first few grades are more likely to repeat and eventually to drop out.

7. In general, average student learning in most developing nations is quite low, according to available evidence. A report by the World Bank’s Independent Evaluation Group in 20061 demonstrated through World Bank lending that while educational access is increasing, improvements to student learning are significantly delayed. At this moment, learning levels in the poorest countries, grade for grade, are around 30% of what they are in more developed countries. Results from such international assessments as the Program for International Student Assessment (PISA) or Trends in International Mathematics and Science Study (TIMSS) allow us to ascertain what students in low-income countries do not know, but they give us little to no information about what these students know. The scores tend to be so low that it is difficult to determine where students land on the knowledge continuum. Additionally, most of these assessments are paper-and-pencil tests given at fourth grade and above with the assumption that students can already read and write. If they do not know how to read and write, it is difficult to know if the results of the assessment are due to the lack of assessed knowledge or to the lack of basic reading and comprehension skills. By fourth grade, students may already be very far behind, and helping them to catch up to the appropriate levels of knowledge may take years. Assessments given early and orally can help identify students who are falling behind so they may have a better chance of gaining necessary knowledge.

8. In order to deal with these issues, the Early Grade Reading Assessment was adapted for use in Guyana to provide an overall snapshot of reading skills in second, third, and fourth grades (see Annexes A and B for copies of the instruments). Because EGRA is oral, is relatively fast to administer, and focuses on the skills of reading development, it was possible to gather large amounts of information that could inform policy dialogue and future instruction. Along with the administra-tion of the EGRA, a questionnaire was completed for each student who participated (Section 9 of Annex A). The survey data were to help identify factors that might affect student achievement.

9. The study was conducted by the Ministry of Education in conjunction with RTI and sought to answer the following questions:

• What basic skills for literacy are dominated by second, third, and fourth grade students in Guyanese schools?

• What factors influence students’ mastery of basic reading skills?

1 World Bank: Independent Evaluation Group. (2006). From schooling access to learning outcomes—An unfinished agenda: An evaluation of World Bank support to primary education. Washington, DC: World Bank.

5

II. Preparation of the Study and Implementation of Fieldwork

Capacity Building 10. Capacity building for the fieldwork took place in mid-September 2008 over six days: four days of workshop-based presentations, exercises, and data entry; and two days of practicum in schools. The training was conducted by Jennie Spratt and Emily Miksic of RTI. Enumerators’ interrater reliability or the ability of the enumerators to administer EGRA correctly was calculated for each subsection after the training. Results of the interrater reliability were above 90% on all but one subsection—initial sound identification, which was 80%. This subsection can be difficult to administer if enumerators do not know or are not sure of the correct pronunciation of the initial sound.

11. The objectives of the capacity building were the following: • Explain the importance of evaluation in education and theories of reading. • Develop skills of the participants to administer EGRA with the greatest possible

correctness and homogeneity.

Sample and Organization of Work 12. Fieldwork took place at the beginning of the school year over three weeks in October and November 2008. The final sample was made up of second-, third-, and fourth-grade students2 in 60 schools distributed over six (out of ten) regions of the country. A total of 2699 students were tested: 1439 male and 1260 female. The students attended school in different shifts: morning, afternoon, and full time.

13. The assessment was given to a random sample of students. The Ministry wanted a sample that would allow for analysis by grade level as well as by hinterland and coastland regions. In order to provide proportional sampling among the two zones, 15 and 10 or fewer schools were selected in each of 2 hinterland and 4 coastland regions (considered in this study) respectively.. All schools and regions were selected randomly for each zone.

14. The following procedures were used to select the final sample of schools and students (see also Table 1):

• Using a computerized list of all primary schools in the six selected regions, the list for each region was sorted by school size (from either largest to smallest or smallest to largest).

• The random sampling approach was applied to the regions, as was done to select children in classrooms: Once the number of schools needed for the sample in a given region was determined, the total number of primary schools in the region was divided by this desired subsample to determine the “skip” pattern (S), and then every “S”th school from the size-ordered list of schools in the region was selected. The initial step of sorting by size ensured that the resulting sample would have larger and smaller schools in roughly the same proportion that they exist in the population.

2 Given that it was the beginning of the school year, the target sample was shifted from grades 1–3 to 2–4.

6

• Then, to accommodate small enrollments wherever these existed in randomly selected schools, rather than attempting to hand-select “large” and “small” schools à priori, RTI used the following general rule:

o Twenty students per grade level, randomly selected, in all schools with at least 20 students per grade were tested.

o In schools with fewer than 20 students in a given grade level, all students in the grade level were tested.

Table 1. Distribution of students, by grade and gender Student gender

Grade Male Female Total 2 484 442 926

3 475 423 898

4 480 395 875

Total 1439 1260 2699

Instrument 15. Due to differences in language, culture, and educational expectations, the EGRA instrument is adapted and piloted anew for each country in which it is applied. In June 2008, RTI and the Ministry of Education met to adapt the instrument and carry out a small pilot study. Changes made to the instrument included adapting the language of the instructions to fit the Guyanese context, modifying various items throughout the subsections, making minor changes to the oral reading passage and comprehension questions, and rewriting the listening comprehension passage from expository to narrative. Once the instrument was adjusted, Ministry staff were trained to administer it, and a small pilot study was conducted to verify reliability and validity of the new instrument. One hundred and seventeen students in four schools were tested over two days.

16. To measure the internal consistency across the subsections, Cronbach’s alpha was calculated for the Guyana version of EGRA. Results of the pilot revealed that Cronbach’s alpha calculated on the full set of Guyana EGRA subsections (using standardized scores to adjust for scale range differences across sections) had a reasonably strong internal consistency, at 0.893 (N = 95 cases possessing all 10 subscores). Correlations between each section with the total of all other sections indicated that “orientation to print” (Section 1) was very weakly correlated with performance on other sections of the instrument, suggesting that this section measured skills that were unrelated to the other reading skills being measured. Other sections showed moderate (0.400–0.599) to strong (0.600 and higher) correlations. RTI made modifications to the instrument based on the pilot results, including deletion of the orientation-to-print task.

17. As noted above, the instrument that was implemented consisted of two parts. The first part was the diagnostic of reading skills; the second was the questionnaire concerning the students and factors that might affect student outcomes. The reading skills diagnostic consisted of eight subsections: letter naming knowledge, initial sound identification, letter sound knowledge,

7

familiar word reading, invented word decoding, oral passage reading plus reading comprehension, listening comprehension, and dictation. The questionnaire covered factors associated with student achievement, such as whether children had the school textbooks and if someone helped students with homework at home.

18. The EGRA subsections were analyzed to confirm reliability of the test. Table 2 shows the results of the analysis. The alphas for each subsection were all close to 1, indicating a high reliability among the subsections of the assessment. For further results of the reliability analysis, see Annex D.

Table 2. Reliability of Guyana EGRA

Item

Observa-tions

Sign

Item–test

correlation

Item–rest

correlation

Average inter-item

correlation

AlphaLetter Name Knowledge (correct letters per minute) 2699 + 0.763 0.691 0.554 0.909 Initial Sound Identification 2699 + 0.644 0.547 0.584 0.918 Letter Sound Knowledge 2698 + 0.671 0.579 0.577 0.916 Familiar Word Reading (correct words per minute) 2699 + 0.906 0.875 0.518 0.896 Invented Word Decoding (correct nonsense words per minute) 2699 + 0.814 0.756 0.541 0.904 Oral Passage Reading 2699 + 0.904 0.872 0.518 0.896 Reading Comprehension (correct answers to comprehension questions) 2699 + 0.840 0.788 0.534 0.902 Listening Comprehension (correct answers to comprehension questions) 2699 + 0.588 0.481 0.598 0.922 Dictation (correct answers to dictation questions; weighted score) 2576 + 0.852 0.804 0.533 0.901

Test scale 0.551 0.917

Managing the Instrument

19. Part of the administration of the EGRA was a “stop rule” to discontinue the administration of a subsection if students were unable to respond correctly to any of the items in the first line, or if they got the first five items incorrect, depending on the subsection. This rule was established to avoid frustrating students who did not have the skill or did not understand the task of the subsection. In this situation, the enumerator marked the box that read “Check this box if the exercise was discontinued because the child has no correct answers in the first line.”

20. Table 3 shows the number of students in each grade for whom the subsection was discontinued. The subsection with the most discontinued cases was the oral passage reading. The percentage of students over all three grades who were discontinued in this subsection was 29.4%.

8

This means that 45.7% of second graders, 28.8% of third graders, and 13.8% of fourth graders read no words correctly in the first line of the passage.

21. Other subsections where students had difficulty and were discontinued were the invented words (63.8%), familiar words (34.7%), and initial sound identification (34.6%). These subsections all involved the skills related to letter sound correspondence and it would be logical that students who had trouble with one subsection would have trouble with some or all of these subsections. The subsection that was discontinued the least was letter naming: only 4.2% of students tested could not complete the task.

Table 3. Number of cases of students who discontinued Grade % of cases discontinued Subsection

discontinued 2 3 4 Total 2 3 4 Total Letter Name Knowledge 66 34 12 112 7.1% 3.8% 1.4% 4.2%

Initial Sound Identification 368 301 264 933 39.7% 33.5% 30.2% 34.6%

Letter Sound Knowledge 352 261 259 872 38.2% 29.1% 29.6% 32.3%

Familiar Word Reading 489 294 153 936 52.8% 32.7% 17.5% 34.7%

Invented Word Decoding 740 555 427 1722 79.9% 61.8% 48.9% 63.8%

Oral Passage Reading 423 259 121 803 45.7% 28.8% 13.8% 29.8%

III. Analysis of Results 22. These results show that students’ scores on subsections improved at least somewhat from grade to grade. Overall, students seemed to have difficulty with subsections involving letter sound correspondence and word reading. Students had the highest means on the letter naming subsection and the lowest on the invented word reading subsection. There was also a clear distinction between the results of female students and male students in each subsection. Females outperformed males in all skills that were tested, on some more significantly than others.

23. The following analysis of the data includes a review of means, standard deviations, and t test results. Means or averages were calculated by gender for each group and also by the two main regions in the country: hinterland and coastland. Means for each subsection included all students who were tested, including those who scored zero and were discontinued, except for the reading comprehension subsection, where only students who attempted the subsection were included, or approximately 40% of the original sample. Standard deviations were also calculated to demonstrate the dispersion of scores around the mean. A low standard deviation indicates that students mostly obtained scores at or close to the mean, while a higher standard deviation indicates more variability or a wider range of scores. Seeing both would mean that there was a

9

range of knowledge or mastery of the skill measured by the subsection. Because t tests are a statistical test of whether the means of two groups are equal, they were calculated to determine if the differences in the means of groups were large enough to infer significant differences in knowledge between the groups. For example, for almost all subsections and grades, female students had higher means than male students; but not all of these differences were large enough to say that female students performed as a group significantly better than male students. T-tests were conducted to compare means of female and male students for each grade on each subtest. ANOVA was carried out to compare the means of students by grade and region (hinterland or coastland) for each subsection.

24. Students were compared to international benchmarks in order to gain a sense of how students scored in comparison to where they would be expected to score for their grade and the time of year. Because it was the first time EGRA was administered in Guyana, benchmarks on this assessment for Guyana did not exist. The international benchmarks were taken from a similar assessment used frequently in the United States, called Dynamic Indicators of Basic Early Literacy Skills (DIBELS). DIBELS contains the same or similar tasks although they are based on the U.S. context; that is, test items are different from EGRA but many of the skills measured are the same. These benchmarks are generally higher than what would be expected in low-income countries but they are also a helpful way of interpreting the results and understanding how students should be scoring compared to how they actually scored.

Letter Name Knowledge 25. This subsection consisted of a page of 100 randomly distributed upper- and lowercase letters. Students were asked to say the names of as many letters as they could in one minute. Scores for this subsection were the number of letters students correctly named in one minute (correct letters per minute); see Table 4. As stated above, means for this section were the highest of all the subsections. The means increased with grade progression. Second graders had a mean of 46.4. Third graders had a mean of 57.6, and fourth graders had a mean of 69.6. There was a significant difference in means between the genders in all three grades, indicating that males in each grade could name fewer letters than females. Differences between the males and females in all three grades were around 10 letters per minute. This implies that males were not learning letter names as quickly and might not be catching up with their female counterparts. Students who are behind at the beginning will find it more and more difficult to catch up as they progress through the grades.

Table 4. Letter name knowledge, by grade and gender Means Standard deviations

Grade Male Female Total Male Female Total p value 2 41.9 51.7 46.4 25.8 26.3 26.5 .016

3 53.8 62.2 57.6 28.6 26.6 28.0 .031

4 65.1 75.2 69.6 29.2 28.1 29.1 .027

Total 54.1 63.4 58.3 29.5 28.7 29.5 .006

10

26. A large percentage of students in all three grades were able to meet the international benchmark for letter naming (Figure 1). This indicates that students had a facility with letter naming even in second grade. This skill has been shown to be a predictor of reading achievement in the United States, although the effect internationally is not as clear. This knowledge is one of the first steps in understanding the alphabetic code, which is a skill that is necessary but not sufficient for learning to read.

Figure 1. Letter naming by grade and benchmark

Letters per minute by Grade

010

2030

4050

6070

80

90

100

Grade 2 Grade 3 Grade 4

Grade

Perc

enta

ge Scored zero

Did not meet benchmarkMet benchmark

27. Figure 2 is an item analysis that shows the percentage of incorrect answers for each letter. This analysis identifies which letters students had the most difficulty with. The letters that proved the most difficult across all three grades were: q, d, h, and y (all lowercase). The letter q gave students significantly more trouble than any other letter, with nearly 35% of students misnaming this letter (among those students who read that far in the list of letters). Often q is confused with other letters such as p and it is also a letter used less frequently in English; thus, students see and use it less often. The next highest letters missed were Q at 15% and d, h, and y at 12%. These letters also resemble other letters and/or are used less often. The results indicate that students may need added instruction or practice with these particular letters. Overall, students had less than 10% difficulty naming the majority of the letters. The least difficult letters to name were j, o, and x.3

3 Note that scores were calculated based on the number of students who accurately named the letter among those who had sufficient time to read the letter (“j,” for example, appeared once and was letter 98 in the list of 100). This may have biased the scoring somewhat in favor of higher accuracy on the later-appearing letters, as students who did not run out of time before reading the 98th letter in the list were likely very accurate in their reading. For letters that appeared multiple times, the accuracy scores were averaged for performance across all of the appearances (e.g., “O” appeared seven times in the list of letters).

11

Figure 2. Error rate, by letter

28. When results are divided between the hinterland and the coastland (Table 5), means for letter naming follow patterns similar to those for the whole country. Scores increased as grades increased and female students had higher means than male students in each grade. Students in the coastland had slightly higher means than students in the hinterland but the differences are too small to draw conclusions concerning differences in performance between zones.

Table 5. Letter name knowledge, by region Hinterland

Means Standard deviations Grade Male Female Total Male Female Total

2 34.9 47.5 41.3 24.8 25.4 25.7

3 50.8 61.9 56.3 24.8 26.4 26.5

4 64.6 78.2 70.8 24.2 23.8 25.0

Total 50.1 61.6 55.7 25.9 26.2 25.0

Coastland Means Standard deviations

Grade Male Female Total Male Female Total 2 42.6 55.2 47.0 25.8 26.4 26.5

3 54.1 62.2 57.7 28.9 26.6 28.2

4 65.1 75.9 69.5 29.5 28.4 29.4

Total 54.5 63.6 56.6 29.7 28.7 29.6

29. Students’ knowledge of letter names appeared to be very good. Students performed better on this subsection than any of the others. Also more students met the international benchmark on letter naming than the other subsections. Over 90% of fourth graders met the benchmark, which

12

demonstrated almost 100% knowledge of letter names with automaticity. This implies that students were being taught letter names sufficiently and may not be in need of additional instruction in this particular skill of reading development.

Initial Sound Identification and Letter Sound Knowledge 30. In order to read, each of us must turn the letters we see into sounds, sounds into words, and words into meaning. Successfully managing this process requires the ability to work in reverse; that is, in order to understand the process of moving from letters to sounds to words, students should also grasp that words are composed of individual sounds and understand the process of separating (and manipulating) words into sounds. This ability to identify sounds in words, to separate words into sounds, and to manipulate those sounds is termed phonemic awareness. This skill is one of the building blocks of reading. Students need to understand that the words they hear are made up of individual sounds and those sounds will be represented in writing with letters.

31. The subsection called initial sound identification assessed students’ ability to hear a word and separate and identify the first sound in the word. This is one way to measure students’ phonemic awareness. Results for the initial sound identification showed that across grades, students had about the same amount of facility with this skill (Table 6). Second-grade students had a mean of 4.1. Third-grade students had a mean of 4.9, and fourth-grade students had a mean of 5.0. These means were based on students identifying the initial sounds of 10 words. There was virtually no increase in performance between the grades, indicating that students were not learning this skill at any point in the three grades. Significant differences were found once again among all grades between female and male students. Female students generally answered one more item correctly than male students but there seemed to be a general lack of understanding of this particular skill. The possible unfamiliarity with the skill or task could have led all students to struggle with the subsection. Obviously, if students have not been taught this skill, they are unlikely to perform well. Students seemed to have the most difficulty with initial sounds U, G, N, and L.

Table 6. Initial sound identification, by grade and gender Means Standard deviations

Grade Male Female Total Male Female Total

t test 2 3.6 4.6 4.1 3.8 3.9 3.9 .042

3 4.5 5.4 4.9 3.8 3.8 3.8 .047

4 4.5 5.7 5.0 3.7 3.5 3.7 .033

Total 4.2 5.2 4.7 3.8 3.8 3.8 .007

32. Figure 3 below presents how students performed compared to international benchmarks of more than seven out of 10 items. Students in second grade had the highest percentage of zero scores and the lowest scores for students who did and did not meet the benchmark. There was little increase over the grades. Approximately one third of students met the benchmark in each grade, one third did not, and one third scored zero. These results indicate a lack of knowledge of

13

phonemic awareness, which may affect students’ ability to hear words and identify the sounds in them so that they can be written by the letters that represent the sounds. As expected, low performance on this subtask correlated with low performance on spelling accuracy in the dictation task.

Figure 3. Initial sound identification, by grade and benchmark

33. In Table 7, the results for the hinterland and coastland can be compared. The results show that students in the coastland had significantly higher means. Females had higher means than males, as they did above. Means were generally between three and five for all groups, implying that almost no students who were tested had the phonological awareness required by this task or the understanding of the task itself.

Table 7. Initial sound identification, by grade, gender, and region Hinterland


2 3.1 3.8 3.4 3.7 4.1 3.9

3 3.5 4.2 3.8 3.8 4.0 3.9

4 4.1 4.4 4.2 3.9 3.9 3.9

Total 3.6 4.1 3.8 3.8 4.0 3.9

Coastland

Grade Means Standard deviations Male Female Total Male Female Total 2 3.7 4.7 4.1 3.8 3.9 3.9

3 4.6 5.5 5.0 3.8 3.8 3.8

0 5

10 15 20 25 30 35 40 45 50


Grade

Perc

enta

ge

Scored zero Did not meet benchmark Met benchmark

14

4 4.5 5.8 5.1 3.7 3.8 3.7

Total 4.3 5.4 4.8 3.8 3.7 3.8

34. Initial sound recognition requires the ability to hear and separate the sounds in words. This task can be difficult if students have not been taught or are not practiced in it. Students who had not been taught to separate words into sounds would be unfamiliar with this task and would likely not perform well. Yet this is an essential foundation skill for students to understand the relationship that exists between the written and spoken word (and is highly predictive of later reading skill). Additional instruction in this skill should be considered so that students will be able to master this important piece of reading development.

35. The letter sound identification subsection asked students to produce the sound of as many letters as they could in one minute (correct letter sounds per minute). This task measured students’ knowledge of the sounds that were represented by the written symbols we call letters. Means for this subsection (Table 8) demonstrated that students across all grades had limited competence with this skill. In this subsection, girls seemed to have an advantage over the boys but were still low in their knowledge of letter sounds. The gender differences for this subsection were all significant when the overall means were compared, but only fourth-grade students showed significant differences by gender. The means also showed small differences among the grades, which indicates that students were not gaining knowledge of letter sounds throughout the early grades.

36. The overall mean was 13.5, which is well below the international benchmark of greater than 27 letter sounds per minute. Knowledge of letter sounds is one of the first steps in learning to read words. Students must know which letter makes which sounds in order to be able to decode new words. Students in the sample were still lacking this knowledge, which will affect their ability to read words and eventually connected text. Additionally, the low scores likely resulted from unfamiliarity with the task, demonstrating the need for additional practice with this skill.

Table 8. Letter sound knowledge, by grade and gender Means Standard deviations

Grade Male Female Total Male Female Total t test 2 9.9 11.9 10.8 14.0 13.7 13.9 .311

3 12.8 16.8 14.6 14.3 15.2 14.8 .053

4 12.5 17.4 14.7 13.2 17.4 15.5 .023

Total 11.8 15.5 13.5 13.9 15.8 14.9 .015

37. Student means in the coastland were slightly higher (differences were not significant) than means of students in the hinterland (Table 9). This signifies that all students in Guyana—regardless of location, grade, or gender—knew few letter sounds with automaticity.

38.

15

Table 9. Letter sound knowledge, by region Hinterland


2 8.3 12.1 10.2 11.4 15.1 13.5

3 10.5 13.5 12.0 11.9 14.1 13.1

4 10.2 12.6 11.3 12.0 14.6 13.3

Total 9.7 12.7 11.1 11.8 14.6 13.3

Coastland


2 10.1 11.9 10.9 14.2 13.5 13.9

3 12.9 17.1 14.8 14.4 15.3 15.0

4 12.7 17.8 15.0 13.3 17.6 15.6

Total 12.0 15.8 13.7 14.0 15.8 15.0

39. In contrast to letter names, students demonstrated sufficiently less knowledge of letter sounds. The means for this subsection were less than half of what they were for letter naming. This is significant because it is the knowledge of the letter sounds that will most help students to read new words. If students do not know which letter makes which sound, they will not be able to sound out new or unfamiliar words. The effects of this lack of knowledge can be seen throughout the other subsections such as word reading, invented word reading, and oral passage reading. In this subsection gender differences also existed, although they were significant only in grade 4. Female students of all three grades had higher means than the males by three or four letter sounds.

Familiar Word and Nonsense Word Reading 40. Children’s decoding skills are often assessed using reading lists of unrelated words. This allows for a purer measure of word recognition and decoding skills than does reading comprehension paragraphs, as children are unable to guess the next word from the context. In the familiar word reading task, students were given a list of common simple words with instructions to read as many as they could in one minute. Familiar word reading means (in correct words per minute) revealed grade-level progression in reading rates: 8.0 in second grade, 18.5 in third grade, and 28.6 in fourth grade, with word reading ability improving with each grade (Table 10). Results from this subsection also demonstrated differences between male and female students. Female students in all three grades had higher means than the male students. The mean differences were statistically significant when t tests were run, demonstrating differences between genders and grades. Interestingly, the mean differences by grade and gender were less significant in third grade than they were in second and fourth grade.

16

Table 10. Familiar word reading, by grade and gender Means Standard deviations


3 15.5 22.2 18.5 21.0 20.9 21.2 .022

4 22.5 36.1 28.6 24.2 28.3 26.9 .007

Total 15.1 23.3 18.8 20.9 24.2 22.8 .002

41. Figure 4 below demonstrates the increase in word reading. Forty percent of second grade students were reading below the benchmark and over 50% scored zero in this subsection. Third graders who were tested had fewer zeros but more students were unable to read at the benchmark level, indicating more students read some words but not enough to meet the benchmark. Finally, 40% of fourth grade students met the benchmark and 40% did not, but they were reading some words, which was an improvement over the other grades.

Figure 4. Familiar word reading, by grade and benchmark

Familiar Words per Grade

0

10

20

30

40

50

60


Grade

Perc

enta

ge Scored zeroDid not meet benchmark

Met benchmark

42. The means by gender followed patterns similar to those in previous sections; however, there was a significant difference in performance between the hinterland and coastland (Table 11). Students in the coastland had considerably higher means than the hinterland on this task.

Table 11. Familiar word reading, by region Hinterland


2 2.7 7.4 5.1 6.4 12.0 9.9

3 9.2 16.2 12.7 16.2 19.3 18.1

17

4 18.1 30.2 23.6 18.9 24.9 22.7

Total 10.0 17.3 13.5 16.1 21.3 19.1

Coastland Means Standard deviations

Grade Male Female Total Male Female Total 2 6.5 10.5 8.4 12.0 12.3 12.3

3 16.0 22.8 19.0 21.3 21.0 21.4

4 22.9 36.7 29.0 24.3 28.5 27.2

Total 15.5 23.9 19.3 21.2 24.4 23.1

43. The means for familiar word reading were considerably below the international benchmark. This is important because the ability for a student to read words with automaticity is connected to students’ ability to comprehend text that they read. This subsection demonstrates that students were learning words as they progressed through the grades but that they were learning very few. It is likely that they were not learning to read new words but had memorized what words look like. Therefore, any word students did not already know would be difficult if not impossible for them to read. The increase in word reading rate could be due to exposure to more words and text over time.

44. Invented word reading is a measure of decoding ability and is designed to avoid the problem of sight recognition of words. Many children in the early grades learn to memorize or recognize by sight a broad range of words. To be successful readers, children must combine both decoding and sight-recognition skills. Tests that do not include a decoding exercise can overestimate children’s ability to read unfamiliar words, as the words tested may be part of the sight-recognition vocabulary. Similar to the word reading measure, students were asked to read as many invented words in one minute as they could.

45. Means for invented word reading were also increasing by grade level but were low in comparison to the international benchmark, which is more than 25 words read correctly per minute (Table 12). Students read an average of 6.5 words per minute out of the 50 invented words introduced. Again, gender differences appeared, with female students reading more invented words than male students. Significant differences between genders were found in the means overall and in fourth grade, similar to the mean differences for the letter sound task. Letter sound knowledge is generally a prerequisite for performance on this task, so it makes sense to see similar patterns in this section. The means were also significantly lower from those of familiar word reading for all grades, indicating a lack of letter sound knowledge because students can memorize what words look like and “read,” but invented word reading requires decoding the words, which requires knowing the sounds letters make.

18

Table 12. Invented word decoding, by grade and gender Means Standard deviations


3 6.4 7.5 6.9 11.9 12.4 12.1 .462

4 7.4 12.8 9.8 12.5 15.8 14.3 .016

Total 5.3 7.9 6.5 10.9 13.0 12.0 .021

46. Similar to the pattern of low means on this subsection, almost no students met the international benchmark and most students scored zero in each grade (Figure 5). There were students who read some invented words but they were in the minority. It appeared that students’ overall lack of letter sound knowledge made decoding invented words very difficult.

Figure 5. Invented word decoding, by grade and benchmark

47. Means for the coastland and hinterland revealed the same pattern as letter sound knowledge and letter naming (Table 13). Students from the coastland had slightly higher means but the differences were small and insignificant. It is reasonable that if students did not have great differences in means in letter sound recognition, a similar lack of differences would be seen in this decoding task. This further demonstrates that all students were lacking in knowledge of letter sound correspondence, which kept them from being able to read new words. On the opposite end of the spectrum, letter naming may not have shown significant regional differences because students were all achieving fairly high means on that task.

Table 13. Invented word decoding, by region Hinterland


2 0.8 3.0 1.9 3.1 8.0 6.2

0 10

20

30

40

50

60

70

80

90


Grade

Perc

enta

ge Scored zero

Did not meet benchmark Met benchmark

19

3 3.6 5.7 4.6 9.0 9.6 9.4

4 6.4 11.3 8.6 9.3 14.0 12.0

Total 3.6 6.4 4.9 8.0 11.2 9.8

Coastland


2 1.8 2.9 2.3 6.0 6.7 6.3

3 6.6 7.7 7.1 12.1 12.6 12.3

4 7.5 12.9 9.9 12.7 15.9 14.5

Total 5.5 8.0 6.6 11.1 13.2 12.2

48. The results of this subsection seem to confirm that without good knowledge of letter sounds, students were not learning or did not understand the letter sound relationship that is needed to read new words. Means on this subsection were lower overall than the means on the familiar word reading. Students either did not know enough letter sounds to decode new words or did not understand the relationship between the letters that make up a word and the sounds in the word. In this subsection, almost no students met the international benchmark, but the number of students who got zero on the subsection did decrease over the three grades. This shows that students were learning something about letter sounds and their correspondence to the grapheme but perhaps not enough to help them become efficient readers of new words.

49. Results of the two familiar and invented word reading subsections may indicate that as students were exposed to common words, they were able to memorize them, and in each succeeding grade level students had learned more words but had not yet learned to decode new words.

Oral Passage Reading and Comprehension 50. Oral reading fluency is a measure of overall reading competence: the ability to translate letters into sounds, unify sounds into words, process connections, relate text to meaning, and make inferences to fill in missing information.4 As skilled readers translate text into spoken language, they combine these tasks in a seemingly effortless manner; because oral reading fluency captures this complex process, it can be used to characterize overall reading skill.

51. The oral reading passage was a fictional narrative, 61 words in length. Students read as much of the passage as they could in one minute. Means of the oral passage reading subsection were similar in trend as the familiar word reading (Table 14). There were significant increases between grades: second grade – 14.9, third grade – 33.9, and fourth grade – 49.9. This demonstrates approximately 19 words per minute difference between second and third grade and 10 words per minute between third and fourth grade. Female students also continued to score higher than the male students in all three grades. T-tests revealed significant differences between 4 Hasbrouck, J., & Tindal, G. A. (2006). Oral reading fluency norms: A valuable assessment tool for reading teachers. The Reading Teacher, 59(7), 636–644.

20

the means of female and male students in all three grades. As grades increased, so did the gap between female and male students, suggesting that female students may have been progressing in their reading development at a faster and better rate than male students.

Table 14. Oral reading fluency, by grade and gender Means Standard deviation


3 27.5 41.7 33.9 34.1 37.9 36.6 .017

4 39.7 62.3 49.9 39.8 43.2 42.9 .005

Total 26.8 41.8 33.6 34.5 40.0 37.8 .001

52. Figure 6 illustrates the number of students who were not yet meeting the benchmark in oral passage reading. Percentages of students meeting the benchmark increased throughout the grades but the number of students below the benchmark also increased with grade progression given that the number of students that scored zero decreased considerably between second and fourth grade. . This indicates that many students could read some words but were still much slower than needed for comprehension. Yet the increase in words correct per minute is obvious from grade to grade. This implies improvement in the ability to read connected text.

Figure 6. Oral reading fluency, by grade and benchmark

Oral Reading Fluency by Grade

0

10

20

30

40

50

60

70

80


Grade

Perc

enta

ge Scored zeroDid not meet benchmarkMet benchmark

53. Figure 7 compares the students’ means to the benchmark, but this figure also clearly illustrates the differences in gender in each grade compared to the benchmark. Second grade male and female students had relatively close means but male students in fourth grade had a mean of over 20 words per minute lower than the female students.

21

Figure 7. Oral reading fluency, by benchmarks

Oral Reading Fluency in Guyana Compared to International Benchmarks

26

52

70

53

67

8071

83

96

44

68

90

77

92

110

93

105

118

11.2

27.5

39.7

19.4

41.7

62.3

2ndBeginning

2nd Middle 2nd End 3rd Beginning 3rd Middle 3rd End 4th Beginning 4th Middle 4th End

Grade and Time of Year

Oral Reading

Fluen

cy

Some risk Low risk Guyana boys Guyana girls

54. Oral reading fluency was significantly different between coastland and hinterland regions (Table 15). Differences were, however, not as pronounced as gender differences. By fourth grade, coastland students in fourth grade had a means of about 8 words per minute higher than hinterland students.

Table 15. Oral reading fluency, by region Hinterland


2 4.9 12.3 8.6 10.6 19.3 16.0

3 16.8 39.5 23.1 25.5 33.4 30.3

4 32.2 54.2 42.3 33.1 42.8 39.4

Total 18.0 30.8 24.2 27.3 36.8 32.9

Coastland


2 11.8 20.3 15.6 18.6 22.9 21.1

3 28.4 42.7 34.8 36.5 38.1 36.9

4 40.4 63.7 50.5 40.2 43.2 43.1

Total 27.2 42.1 34.5 35.3 40.3 38.1

55. Because students were scoring lower on the previous subsections that involved letter sound correspondence and word reading, it is not surprising that students also did not meet international benchmarks on the oral reading passage. Without firm knowledge of letter sounds and word reading skills, fluency in connected text reading is difficult to attain. In this subsection the means

22

were lower than what would generally be seen as sufficient rate of word reading for comprehension. This implies that students were having to use all of their attention to read each individual word and had little working memory left for recognizing the meaning of the words and sentences in order to comprehend the passage. The results of this subsection demonstrate that students were learning to read some words over time, but the rate of reading was slow. It is also important to note that nearly 30% of students given the assessment had to stop this section early because they could not read any words correctly in the first line.

Reading Comprehension 56. This subsection was based on how much of the oral reading passage students were able to read and understand. Students were only asked to attempt questions for the comprehension section that pertained to the section of text they read in the passage. Thus, if they only read half the text, students were only asked the questions that could be answered by half the passage. Also if students did not read any of the text, they did not get asked any questions. Thus, for this subsection, the sample size is different from the original sample size. It is made up of only students who read some part of the oral reading passage, or about 70% of the total sample.

57. Results for the reading comprehension subsection varied by question. There were six questions of varying levels of comprehension; i.e., some were intentionally easier to answer than others. For example, the first question, What is the name of Pat’s brother?, had the highest means in all groups. This question could be answered at the beginning of the passage, making it the most frequently asked question. It is also an explicit question with the answer in the text, making it one of the easiest questions to answer.

58. In contrast, the second question asked, How many children were in Pat’s family? This question received correct responses from the least number of students. It required students to find the answer in multiple parts of the story in order to come up with the correct response. As a result, students who did not read the entire story may not have had enough information to answer the question correctly.

59. The tendency was for the questions to become more difficult as students progressed through the passage. One generally would expect correct answers less frequently for the more difficult, higher-level questions. However, the last two questions had a higher percentage of correct answers than other questions. The pattern can be seen in Figure 8 below, except for one question that required students to make an inference. Question 6, Where does Pat’s mother work?, was inferential because it was not in the text; yet more students who attempted the question answered correctly in comparison to other questions that were more explicit. The higher percentage of correct responses on this question may be explained because only students who had read most of the passage would have attempted the question. These students would generally be the better readers and would be more likely to have developed better comprehension skills.

23

Figure 8. Reading comprehension, percentage correct, by grade and gender

Reading comprehension percentage correct by Grade and Gender

0 20 40 60 80

Boys

Girls

Boys

Girls

Boys

Girls

Gra

de 2

Gra

de 3

Gra

de 4

Perc

enta

ge

Where does Pat's mother work?

How often do the children go to thefarm?What do Pat, Sam and Ann like todo?Where does Pat live?

How many children are in Pat'sfamily?What is the name of Pat's brother?

60. In summary, the results for this subsection were somewhat different from those for the previous subsections because the number of questions asked differed by child, based on how much of the passage the child read. The questions with the highest percentage of correct answers were the first and most explicit question, and the last two inferential questions. These results could be revealing in terms of the types of questions generally asked of students during reading comprehension instruction. As noted earlier, the higher percentage of correct answers on the two inferential questions could have occurred because of the particular students who attempted them. If these students had to read the entire passage in order to attempt these questions, they were probably the better readers and therefore may have had more experience with reading for comprehension.

Listening Comprehension and Dictation 61. The listening comprehension section consisted of a short (48-word) passage that was read to the students. Students were then asked five comprehension questions of varying difficulty. This subsection differed from the reading comprehension subsection in that it measured all students’ ability to comprehend. Because students did not have to read in order to answer the questions, all students attempted to answer all questions in the subsection. Thus, students were only measured on their comprehension ability.

62. There were five items in the oral comprehension section. Each was given a value of 20 points, for a total of 100 (see Table 16).

Table 16. Point values of the subsections Section Subsection Points Total

Listening Comprehension 7.1 – 7.5 20 points/item 100 points

24

63. As with the previous comprehension section, the results of the listening comprehension results varied by grade, gender, and question, with the means of each grade increasing (Table 17). Second-grade students obtained a mean of 27.3, third-grade students had a mean of 35.8, and fourth-grade students’ mean was 41.7 in listening comprehension. The means were slightly higher for female students than male students except for the third-grade sample, where male students for the first time performed only slightly better than female students. However, these gaps were not significant like those of subsections where students were required to read words. This suggests that more students were able to comprehend when they did not have to focus intently on reading the individual words, especially unfamiliar words.

Table 17. Listening comprehension, by grade and gender Means Standard deviations


3 37.0 34.4 35.8 28.4 29.0 28.7 .298

4 40.8 42.9 41.7 27.6 30.9 29.2 .444

Total 35.3 35.2 35.3 27.4 29.2 28.3 .951

64. The results in listening comprehension varied by question. For example, the question answered correctly most often was Question 2: Who fell in the mud last? This question was an inferential question requiring students to use more than the words in the story to give the correct response.

65. The question with the least correct responses was Question 3: What did the duckling see that was important? This question was also asked students to think beyond the words in the story to answer the question.

66. In Figure 9 and Table 18 below, the percentages of correct responses to each question by grade, gender, and region can be seen. None of the questions was explicit with an obvious response. Interestingly, students answered the question What is a duckling? correctly more often than How did the frog and duck get out of the mud? The story did not define “duckling”; the main problem in the story was how the two animals got out of the mud. Students’ correct answers did not seem to follow the usual pattern of giving more accurate answers to questions that were more explicit or obvious in the story over ones that required more complex comprehension and thinking.

25

Figure 9. Listening comprehension, by grade and gender

Listening comprehension by gender and grade

0 20 40 60 80

Boys

Girls

Boys

Girls

Boys

GirlsG

rade

2G

rade

3G

rade

4

Perc

enta

ge

Why do you think the frog and theduck are friends?How did the frog and the duck get outof the mud?What did the duckling see that wasimportant?Who fell in the mud last?

What is a duckling?

Table 18. Listening comprehension, by region Hinterland


2 21.9 22.1 22.0 21.5 23.2 22.4

3 26.4 26.4 26.4 26.0 24.3 25.2

4 32.9 33.2 33.0 27.7 27.5 27.6

Total 27.1 27.0 26.2 25.6 25.3 25.5

Coastland


2 27.7 28.2 27.9 24.2 25.3 24.7

3 37.9 35.2 36.7 28.4 29.3 28.8

4 41.4 43.8 42.5 27.5 31.0 29.2

Total 36.0 36.0 36.0 27.5 29.5 28.4

67. The results of this subsection reveal an interesting pattern of comprehension. Based on the percentage of correct answers, students seem to have answered questions that were more inferential better than they did the explicit questions. The significance of this is different from reading comprehension, because all students were asked all questions in this subsection. The pattern could be a function of the types of questions they were used to answering in other instructional situations, or in everyday life; or it may have had something to do with the story or the questions themselves. Student comprehension in this subsection, however, improved over that of the reading comprehension, implying that students did in fact have comprehension skills

26

and that it might have been their inability to read the passage fluently that kept them from comprehending.

Dictation 68. Teachers often use dictation assessment to test both oral comprehension and writing skills. The reading process can also be tested in reverse: students’ ability to hear sounds and correctly write the letters and words corresponding to the sounds they hear demonstrates their success with the alphabetic principle. This subsection of the instrument consisted of students being read a sentence of no more than 10 words, three different times, while they attempted to write what they heard. Students were scored on a simple scale that captured accuracy for vowel and consonant sounds, spelling, spacing and direction of text, capitalization, and punctuation. Each category had a total of two possible points for total accuracy, with one for some accuracy and zero for no accuracy. During analysis, these variables were added up and weighted for a single score variable.

69. Correct answers to the dictation variable were created using the following weights (to obtain a variable with a maximum score of 100). Spelling of the first four words (shop, buy, rice, sugar) was scored at 10 points each (total of 80% of the variable weight), with the remaining 20% of the weighted variable as follows: spacing and capitalization were given 3 points each; direction and period at end of sentence (full stop) were given 2 points each. Each of these weights was then multiplied by the score (0, 1, or 2) and added to obtain a final point total. Spelling of the critical words was given greater weight due to our interest in understanding student knowledge of the alphabetic principle, with less weight given to issues of grammar.

70. Results of the dictation subsection (Table 19) showed patterns similar to those in previous subsections. Means increased with grade, and female students had higher means than male students in all three grades. Similar to other subsections involving letter sound correspondence, students generally received low scores on this subsection. The maximum score was 100 while the group with the highest mean was fourth-grade female students, with a mean of 54.5.

Table 19. Weighted dictation results, by gender and grade Total Male Female t test

Grade N Mean SD N Mean SD N Mean SD (male vs. female) 2 876 12.0 20.6 462 8.5 17.3 414 15.9 23.2 -5.36

3 862 28.1 30.3 455 23.3 28.8 407 33.4 31.1 -4.96

4 838 45.5 33.7 459 38.2 32.6 379 54.5 32.9 -7.18

Total 2576 28.3 31.8 1376 23.3 29.6 1200 34.0 33.2 -8.67 N = sample size SD = standard deviation

71. The scores by grade on the dictation are shown below. Figure 10 demonstrates the range of scores for each grade. Students in second grade had the smallest range, while students in fourth grade had a higher mean and a wider range of scores. Improvements can be seen as grade level increases, but mean scores still remained well short of the maximum score of 100.

27

Figure 10. Weighted dictation results, by grade

020

4060

8010

0C

orre

ct A

nsw

ers

to D

icta

tion

2 3 4

72. Figure 11 shows the correct answers per item, by grade. Students in all three grades had the highest percentage correct on the direction of the sentence, demonstrating that students were aware of text direction. In terms of spelling, there were students in each grade who were able to spell each of the words correctly. The words with the highest percentage of correctness were “rice” and “shop.” The word with the lowest percentage correct was “sugar,” as the sounds in this word do not necessarily match to the letters and may require more knowledge of possible ways to use letters to form these sounds. The use of capitalization and punctuation had the lowest percentages, implying that students even in fourth grade were not using or understanding these basic features of a sentence.

Figure 11. Raw dictation results, by item and grade

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Shop Buy

RiceSug

ar

Spacin

g

Directi

on

Capita

lizatio

n

Full s

top

Dictation Item

Perc

ent C

orre

ct

Grade 2

Grade 3

Grade 4

28

73. Similar to the other subsections that measured letter sound knowledge, student scores on this subsection revealed lack of this knowledge. Student scores overall were well below the maximum score of 100. The scores revealed a knowledge of text direction and an increase in scores in the higher grades, but even in fourth grade, students were still having trouble spelling common words and using punctuation and capitalization. This subsection emphasized the relationship between reading and writing and the importance of students’ need to have knowledge of letter sounds. Without this knowledge, writing as well as reading is difficult at best.

IV. Student and Teacher Surveys

Student Factors Affecting Student Achievement 74. After completing the assessment, students were asked several questions about their background and factors that might affect their achievement in school (see Section 9 of Annex A). The means, standard deviations, and t tests for each group of interest were calculated for the subsections of EGRA measuring students’ knowledge and facility of letter sound correspondence and reading fluency. The subsections used were letter sound recognition, familiar word reading, invented word reading, and oral passage reading. While causal relationships cannot be proved, some possible factors affecting reading skills can be seen in the significant differences of the means between groups. All tables below report scores in terms of correct words per minute.

Do you speak the same language at home and at school?

75. Students were asked if they spoke the same language at school and at home (85% of students reported “yes”; see Table 20). Having knowledge of the school’s language is clearly important to student learning in general as well as learning to read. If students do not speak the language used at school, they will not understand instruction, will not know the sounds of the language required to read words, and will not have the vocabulary to comprehend what words they do learn to read. Students who did speak the same language at home and school had overall higher means on all four subsections than students who did not. However, the differences in the means were generally not significant. Significant differences between language groups (see “*” in table) were found in third-grade letter sounds, third- and fourth-grade familiar word reading, and fourth-grade oral passage reading. This indicates that for these students, language may have made a difference in their ability to perform the task. The lack of significant differences between other grades and subsections may be explained by the lack of students’ overall knowledge of letter sounds. Because of this lack, students’ prior knowledge of the school language (by speaking the same language at home) did not seem to help them in performing these reading skills.

Table 20. EGRA results by home language Same language at school and home Different language at school and home EGRA

subsection Grade 2 Grade 3 Grade 4 Total Grade 2 Grade 3 Grade 4 Total

Letter Sound 12.0 16.0 15.7 14.7 8.5 10.1 11.4 10.0

29

Knowledge

Familiar Word Reading

8.8 20.1 31.3 20.8 5.7 14.2 18.5 13.1

Invented Word Decoding

2.6 7.7 10.6 7.25 1.3 4.5 7.0 4.4

Oral Passage Reading

16.4 36.2 54.0 36.77 10.9 28.3 35.0 25.3


EGRA subsection t test t test t test Letter Sound Knowledge .104 .017* .153

Familiar Word Reading .133 .040* .016*

Invented Word Reading .100 .060 .208

Oral Passage Reading .141 .093 .021*

* Significance level p < 0.05.

Do you have the school reading textbook?

76. Students who had the reading textbook had higher means than students who did not (overall, 66% of students reported having the school reading textbook). Most of the differences between grades and subsections were significant (Table 21). This was especially true of familiar word reading and oral passage reading, where significant differences were found between all grades. Students with a textbook would be exposed to more words and have more opportunity to learn words in the book, which could be the source of the differences. This implies that having the reading textbook could improve students’ achievement. While it seems that simply having the textbook would be important for all students’ achievement, it is important to consider the quality of the textbook and the way in which it was used, which could also have influenced the results.

Table 21. EGRA results by textbook possession Have a textbook Does not have a textbook EGRA

subsection Grade 2 Grade 3 Grade 4 Total Grade 2 Grade 3 Grade 4 Total Letter Sound Knowledge 12.4 16.2 16.2 15.1 8.9 11.3 11.9 10.6

Familiar Word Reading 10.4 21.9 31.7 22.3 4.9 11.4 23.8 12.9

Invented Word Decoding 3.1 8.3 10.7 7.7 1.8 4.0 8.6 4.4

Oral Passage Reading 19.1 39.5 55.4 39.6 9.4 22.2 41.2 23.4

30

Grade 2 Grade 3 Grade 4 EGRA subsection t test t test t test

Letter Sound Knowledge .065 .019* .087

Familiar Word Reading .005* .006* .023*

Invented Word Decoding .010* .008* .030*

Oral Passage Reading .005* .008* .021*


Apart from your schoolwork, are there other books, newspapers or other things to read at your house?

77. This question seemed to reveal one of the biggest differences between groups. Students who said that there were other reading materials in their homes all had higher means on all four subtests than students who did not (75% of students reported having other materials, with hinterland students favored over coastland by 7 percentage points; see Table 22). All differences in means were significant, excluding means of third graders on the letter sound subsection. These differences indicate that having reading material in the home could be very important to students’ reading development. Students who were living in homes with reading materials likely were exposed to literacy at home and earlier than students who did not. Students with reading materials in their homes might also have had more support for learning to read, since reading materials were important to other members of the family.

Table 22. EGRA results by home reading materials Reading material in home No reading material in home EGRA

subsection Grade 2 Grade 3 Grade 4 Grade 2 Grade 3 Grade 4 Letter Sound Knowledge 12.0 14.8 14.6 7.7 11.7 8.8

Familiar Word Reading 9.4 18.4 29.2 4.3 11.2 16.8

Invented Word Decoding 2.7 7.15 10.5 0.78 3.5 5.3

Oral Passage Reading 17.2 33.4 51.6 8.6 21 29.4


EGRA subsection t test t test t test Letter Sound Knowledge .036* .069 .004*

Familiar Word Reading .011* .022* .003*

Invented Word Decoding .037* .015* .012*

Oral Passage Reading .0007* .017* .002*


31

Does anyone else read in your house or family?

78. Means for the two groups of students on this question were higher for students who said “yes,” someone else read at home or in their family; but no significant differences were found among the groups (Table 23). It is likely that students with other readers in their homes or families had more exposure to text and literacy and may even have received more support with their schoolwork. However, without a clearer understanding of what a positive response to the question meant, it would be difficult to draw conclusions. Also students’ lack of knowledge of the skills being measured may make it difficult to see effects.

Table 23. EGRA results by home reading habits Someone else reads at home No one else reads at home EGRA






EGRA subsection t test t test t test Letter Sound Knowledge .082 .117 .074

Familiar Word Reading .152 .085 .064

Invented Word Decoding .126 .070 .087

Oral Passage Reading .151 .118 .092

Did you go to nursery school before first grade?

79. This was another question that seemed to reveal important differences in students’ reading development (Table 24). Students who went to nursery school had higher means on all subsections in all three grades (89% of students reported attending nursery school). The differences in means were mostly significant for third and fourth graders, indicating that students who went to nursery school may have had an advantage over students who did not, although the advantage did not show up until later grades. Student who went to nursery school likely had more and earlier exposure to words, text, reading skills, and literacy in general. These students could have had more instruction and/or exposure to letters, letter sounds, words, and the general concepts of reading before first grade. These results support the advantage of early instruction in reading skills.

Table 24. EGRA results by nursery school attendance EGRA subsection Went to nursery school Did not go to nursery school

32

Grade 2 Grade 3 Grade 4 Total Grade 2 Grade 3 Grade 4 Total Letter Sound Knowledge 11.1 15.4 15.0 13.9 7.6 6.3 10.5 7.9









Do you do homework studies after school?

80. Doing homework (Table 25) also seemed to make a difference in student performance (90% of students reported that they did their homework). While a causal relationship cannot be drawn, students who did homework generally had higher means than those who did not, indicating that homework might have been helping students to develop reading skills to a greater degree than among those who did not. This seemed to make the most difference for second and fourth graders in familiar word reading and oral passage reading. Students had significantly different means on these subsections. The effect on word reading tasks could have been because of the types of homework students were being given. Regardless, there seemed to be better results when students were doing homework—or at least understood that it was important enough to answer “yes” to this question.

Table 25. EGRA results by homework habits Do homework Do not do homework EGRA






EGRA Subsection t test t test t test

33

Letter Sound Knowledge .075 .130 .095




Does someone help you with your homework?

81. Students who had help with their homework (Table 26) generally had higher means than students who did not (70% of students reported getting help on the homework). These means were only significantly different for third graders, on all subsections. Other students who said “yes” to having homework help showed no significant differences but still had higher means than students who did not have help. Students who are learning to read generally need a large amount of practice and support, and these differences could be an indication of this fact. Students’ lack of significant differences cannot be explained by these results alone, but it is possible that students’ definitions of “help” may have varied, thereby affecting their self-reporting for this question. More investigation is needed to fully understand this result.

Table 26. EGRA results by homework help Someone helps with homework No one helps with homework EGRA








Invented Word Decoding 0.282 .024 .555

Oral Passage Reading 0.570 .038 .855

Last year, were you absent from school for more than a week?

82. Results for this final question demonstrated significant differences among fourth-grade students who were and were not absent for more than a week (20% of students responded in the affirmative; see Table 27). The students with higher attendance had higher means than those who were absent more often, but only fourth-grade differences were significant. The higher means seem to imply that absenteeism made a difference in student performance. However, students’

34

lack of knowledge of the reading skills being measured might have kept the majority of these differences small and insignificant. Logic tells us that students who miss school will miss opportunities to learn and will therefore have trouble keeping up with schoolwork and need more reinforcement, but if students are not receiving instruction in these reading skills anyway, high rates of absenteeism may make less of an impact.

Table 27. EGRA results by absences Absent more than a week Not absent more than a week EGRA






EGRA Subsection t test t test t test Letter Sound Knowledge .583 .924 .226




Teacher Factors Affecting Student Achievement 83. Teachers were also given a survey concerning the school as well as their experience, beliefs, and instructional habits (see Annex C). Questions were asked in a variety of forms: yes/no, open ended, and multiple choice. Regression analysis was conducted comparing means of students’ oral passage reading scores and teachers’ answers to the survey questions. Below are the results of the analysis for questions that showed significant differences in means of students’ oral passage reading scores. While causal relationships cannot be drawn through this analysis, the analysis does indicate differences among students that are important when considering factors affecting students’ achievement.

84. The coefficients below can be interpreted by considering the means of the oral passage reading for each grade (see Table 14). With each grade, students gained approximately 18 words per minute in oral reading fluency. If we divide this over the nine months of the school year, we can estimate that students gained approximately two words per minute each month. Thus, the coefficient for the parent involvement question shows a positive difference in oral passage reading of almost 9 words per minute between groups of students, corresponding to 4.5 months of instruction (or half of a school year). Students whose teachers said their school had a

35

functioning parent-teacher association were about half a school year ahead of students whose teachers did not report that their school had a functioning PTA (Table 28).

Table 28. Parent involvement in school

Item Coefficient Significance

level Does school have functional PTA 8.83 <.01

85. Teachers’ background was also an important factor in student achievement (Table 29). Teachers who held a teaching certificate indicating that they had attended a program to learn how to be a teacher had students who read 4.4 words per minute more than teachers who did not. This indicates that teacher knowledge and preparation made a significant difference in student learning, most likely due to preparation in managing a classroom and knowledge of reading development that other teachers without a certificate would not have.

Table 29. Teacher experience


level Do you hold a teacher’s

certificate 4.38 <.01

Distance from home to school (1=60 min plus, 2=46-60 min, 3=31-45 min, 4=45-60 min)

1.68 0.03

86. Teachers also were asked how far they lived from the school (Table 29). Significant differences were found between students of teachers who lived further away and those who lived closer to the school. Students with teachers who lived closer to school had oral passage reading means that were higher than for those with teachers who lived farther away from the school. Teachers closer to the school would have better access and possibly more investment in the students if they were of the same community. One can imagine that this might affect teacher motivation and dedication to students.

87. Teachers’ use of curriculum materials was associated with varying differences in means of oral passage reading (Table 30). Use of recommended reading texts was associated with a negative effect on students’ oral passage reading means, indicating that students with teachers who used the texts read 2.5 words per minute less than students with teachers who did not. The question about use of the text did not indicate whether the texts were being used appropriately. It is possible that teachers were using the texts even though students were not yet able to read them. If the texts—especially those of third and fourth graders—are above the students’ reading level, then using the texts will not be helpful.

36

Table 30. Teacher use of given curriculum


level Do you use the recommended

texts for reading -2.46 <.001

How often do you use the recommended texts for reading 4.46 <.001

How useful is the teacher’s guide -5.1 0.02

88. Likewise, teachers who found the teacher’s guide useful also saw a negative difference in their students’ reading performance. Again, this teacher’s guide may have been written for students who were reading on grade level and thus it would not apply to students with lower reading abilities. Teachers who said they used the recommended texts for reading often had students who read almost 5.5 words per minute more than students with teachers who did not use the text often. This suggests that these teachers may have been giving students more opportunities to read than others. Having many opportunities to practice reading is essential for developing readers and could explain this positive difference in means between the two groups.

89. These differences in means, while significant, suggest that perhaps there are more factors to be investigated, because a teacher who used the recommended text and one who used the text often would likely have students with mean differences similar to those who did not. These results suggest the opposite.

90. The three reading-related activities Table 31 were the activities that demonstrated significant differences in students’ oral passage reading means. Teachers were asked how often in the past five days the activities took place. The positive coefficients suggest that students who participated in these activities read between 2 and 4 words per minute more than students who participated in these activities less often. Retelling a story and learning new words are both related to comprehension of text and are important for helping students with the overall goal of reading. The third activity, reading on their own, gives students opportunities to practice reading, which is necessary if students are to become successful readers. Again, having multiple opportunities to participate in reading-related activities is important for reading development, and teachers who were providing these opportunities to students would likely have students who were reading and comprehending at higher rates.

Table 31. Reading-related activities


level Students retold a story they read 2.01 0.04

Students learned meanings of new words 2.41 <.01

Students were assigned to read on their own time 3.8 <.001

37

91. Teachers were also asked at what grade they believed students should first be able to demonstrate different reading skills. These questions were used to understand teachers’ beliefs and expectations of students’ reading development. The reading skills listed in Table 32 are the activities that demonstrated significant differences between students’ oral passage reading means. All of the coefficients were negative for these reading skills. This indicates that the earlier teachers believed students should be able to demonstrate the skill, the higher the difference in means were. Therefore, teachers who believed students should be able to accomplish the skill in first grade had students with higher means than did teachers who thought the task did not need to be demonstrated until third or fourth grade. While this does not prove a causal relationship, it speaks to possible effects of teachers’ beliefs about reading development on students’ achievement. Teachers’ beliefs that students should master reading skills earlier denotes that they are probably teaching these skills earlier and expecting more of their students—unlike teachers who believe in the later mastery of the same skills. Teachers who expect more of their students generally have higher-performing students.

Table 32. Teacher beliefs


level Read aloud a passage with a few mistakes -3.16 <.001

When a child should write name (1=before grade 1, 2=Grade 1, 3=Grade 3, 4=not

important) -2.34 <.001

When a child should understand stories they read (1=before grade 1, 2=Grade 1, 3=Grade 3,

4=not important) -1.33 0.02

When a child should sound out unfamiliar words (1=before grade 1, 2=Grade 1, 3=Grade

3, 4=Grade 4) -0.79 0.1

When a child should understand stories they hear (1=before grade 1, 2=Grade 1, 3=Grade 3,

4=not important) -2.49 <.001

When a child should recite alphabet (1=before grade 1, 2=Grade 1, 4=not important) -1.94 <.001

92. The final group of questions had to do with teachers’ use of evaluation in the classroom. The teachers who used different types of evaluation with frequency had students with higher oral passage reading means, by up to 5 words per minute (Table 33). All mean differences for teachers’ use of any kind of evaluation were positive, indicating that student achievement may have been positively affected. Using evaluations to inform instruction and make decisions about students has been shown to be important to student achievement, because it allows teachers to know what students have mastered and what information teachers should reinforce. Therefore, if teachers are frequently evaluating reading skills development, they have more information about what students are capable of and where they need more work. They can adjust instruction to meet those needs. This may explain the higher means for students of these teachers.

38

Table 33. Teachers’ use of evaluation


level Frequency of written

evaluations 2.99 <.01

Frequency of oral evaluations 5.2 <.001

Frequency of review of portfolios and projects 2.76 <.001

Frequency of reviewing copy books 2.36 0.01

Frequency of reviewing homework 2.58 0.02

V Conclusions

EGRA Conclusions 93. In general, students in the sample were skilled at letter naming but were lacking in knowledge of the relationship that exists between letters and sounds. Students seemed to be able to name letters and read some words but did not have knowledge of the letter sounds and therefore lacked the ability to read new words they did not know—as is obvious by their invented word decoding scores. It may be that students were not taught to decode in school as much as they were taught to read whole words.

94. Gender differences could be seen throughout the test. In all subsections of EGRA, female students had higher means than male students, and this pattern could be seen across the three grades. Particularly interesting was the gap between genders on oral passage reading. Scores of students in second grade were relatively close, but as the grade levels became higher, means moved further apart between the male and female students. Gender differences are not uncommon in school performance and are generally slanted toward female students having higher performance than males in areas related to reading and literacy. Females are more likely to read more while males are more likely to be involved in more physical activities, giving female students more practice and engagement with literacy than males. However, this is just a generalization of gender differences and cannot necessarily explain these gender differences without further investigation.

95. Analysis of students in the two different zones showed differences between students on some tests and not on others. Students in the coastland generally had higher means than students in the hinterland. Oral reading fluency was significantly different between coastland and hinterland regions, benefiting coastland students. Particular tests that did not find significant differences included letter naming, letter sound knowledge, and invented word decoding. Letter naming likely showed little difference because students from all over the country were fairly good at this very basic skill. Minor differences between zones in naming letter sounds and

39

reading invented words probably show an overall lack of knowledge of letter sounds and the ability to apply that knowledge in reading, using decoding skills.

Student Factors that Affect Achievement 96. Results of the student survey showed that factors such as doing homework, being supported and exposed to literacy early on, and having consistent attendance may have affected student reading development. Conclusions about the student survey can be categorized into two groups: school factors and home factors.

School Factors

97. School factors that showed differences among students were student language, attendance both at their current school and at nursery school, and having the necessary materials. Students who spoke the same language at home and at school had higher means on subsections of EGRA dealing with letter sound correspondence, possibly because they could understand the instruction and had already learned some of the sounds of the language. Attending nursery school also appeared to make a difference in student reading development. Students being exposed to text and reading skills early would have had more time and more opportunities to learn than other students who did not go to nursery school. Opportunities to learn could also explain differences between students who were absent more and students who were absent less. Finally, findings regarding having the reading textbook suggested a need for each student to have the proper materials, at least in second and third grades, where differences were most significant.

Home Factors

98. Student factors that appeared to have an impact on student means for the four subsections were having other reading materials in the home, doing homework, and receiving help with homework. Each of these factors revealed higher means for students and at least some significant differences. Students with reading materials were likely to be exposed to more text and literacy and also to be better supported in learning to read if someone could help them outside of school. Exposure to and support with literacy have a considerable effect on students’ potential achievement. Students who had someone close to them to help with school work, model literacy, and practice reading and comprehending typically had better reading skills. This idea of support and exposure to literacy carried over to the students who had help with their homework. If there were other readers in the home, they might be able to help with homework, and having that help seems to have made a difference for student reading development.

Teacher Factors Affecting Student Achievement 99. The teacher survey also revealed factors that could be important to student reading development, when teacher responses were compared to student scores on oral passage reading. Results suggest teachers who were teaching, evaluating, and expecting students to master skills had students whose reading development improved over teachers who did not.

40

Parent Involvement

100. A functioning parent-teacher association showed the greatest results. Students in schools with functional PTAs had higher scores by almost 9 words per minute. These differences suggest the great importance of having a school where parents can be actively involved. While a causal relationship cannot be drawn on these results alone, it is generally understood that students whose parents are able to be present and active in their education have higher achievement levels than those who do not.

Teacher Experiences

101. Teachers’ experiences, beliefs, and use of reading activities all demonstrated students with significant mean differences. Teachers with teaching certificates and those who lived closer to school had students with higher oral passage reading scores. Teachers who had been prepared in some way to teach may have had more knowledge not only in managing a classroom but also about how reading is developed. Students’ mean differences suggest that teacher preparation could be essential to students’ achievement. Also teachers who lived closer to the school in which they worked had students with higher means. Teachers who had easier access to the school may have been spending more time there and investing more in the students who lived in the same community as they did. Students may have benefitted from the extra time and dedication the teacher could have with easier access to the school.

Instruction

102. Instructionally, students’ means on oral passage reading were higher when teachers (1) reported using evaluations more frequently, (2) used a variety of reading activities more frequently, (3) expected students to master a variety of reading skills earlier, and (4) used the recommended reading texts. Teachers’ use of evaluations seems to demonstrate differences in students’ oral passage reading means, suggesting that teachers who evaluated students more often may have known more about what students knew and needed help with. Also, having students participate frequently in reading activities gives students more opportunities to practice reading skills and reading in general, which are important to student development. The understanding that students could master reading skills early in primary school also raised expectations of students and suggested that teachers were teaching these skills in the earlier grades, which may explain the higher means in students of these teachers.

Implications

Exposure to Literacy Outside of School

103. The role of other readers in the home or families of students appeared to be noteworthy to students’ performance on EGRA. This exposure to text or literacy-related activities could support and extend students’ learning and ensure a more successful school experience. Considering that there may be possible advantages of having more readers in students’ homes and families, it would be useful to extend education programs to family members of the students and to also to make clear to the families the importance of being able to read and support

41

students’ learning. In some instances, this idea is not immediately apparent to parents and other relatives. Along the same lines, having parents more involved in the school and students’ education in general would be important, where possible, to support students’ literacy development. This could be done through PTA activities, especially since PTAs seem to have had a large impact on student achievement.

Teachers

104. Considering the results of the teacher survey, teacher preparation and training in literacy development seem to be an important next step. Teachers who know how to develop reading skills and know that reading skills can and should be developed early will likely help students to gain literacy skills more efficiently. Also the knowledge and use of evaluations to inform instruction are key to student progress. Evaluations such as EGRA can inform teachers regarding how to adjust instruction to meet students needs. Training teachers to understand reading skills and their development, and to use evaluation to inform instruction, could be a significant place to start in improving reading scores.

Early Exposure to Literacy and Reading Skills

105. Based on the differences in means between students who attended nursery school and those who did not, it is likely that students who start early in their education will have a better opportunity to gain literacy skills needed for later schooling. These students’ means in all grades on all subtests were higher than for those who did not have the opportunity to attend school before first grade. This head start in education gave students more opportunities to learn and practice skills than other students. They also likely developed more knowledge about language and text in general. It would be useful to consider making nursery school more accessible and expected for all students. Also, teachers who believed students should master reading skills earlier seemed to have an impact on student means. Thus, training teachers to understand when students are capable of developing reading skills, and how to teach them, would support earlier reading skills development and achievement.

Skills as Strategies for Reading Text

106. Recommendations for the instruction of students based on the outcomes of the EGRA sample would be to focus on teaching letter sound correspondence and automaticity. Students, in general, were lacking knowledge of letter sounds. Without knowing the sounds of letters and understanding their importance for reading, students will only be able to read the words they memorize.

107. It appears that students were either not understanding or not being taught the alphabetic code as a tool to read words. This is evident in the means of the invented word subsection. If students had knowledge of letter sounds and understood the relationship between the visual symbol of the letter or grapheme and the sounds they heard in words, they could decode words and thus would be able to read more words. Also practice improving automaticity of letter and word reading would be useful so that students would be able to read more words per minute. This rate is important because students who read one word at a time, decoding each letter or

42

word, will not have enough working memory to hold onto the meaning of the words and make sense of what they are reading. Students need to be able to read letters and words automatically so that they can concentrate on the meaning of what they are reading. This automaticity will lead to improved reading comprehension. Also to improve comprehension students, should have strategies to use before, during, and after reading so that they are able to follow the meaning of the text. Teaching students how to use different levels of questions to check their understanding would be central for developing comprehension skills.

108. A comprehensive system for teaching all of five components of reading—phonemic awareness, alphabetic principle, vocabulary, fluency, and comprehension—would help students to understand how to read and understand text more efficiently.

Modified EGRA in the Classroom 109. The original purpose of EGRA was to obtain national or large-group data of early grade reading development using a sample of students to represent the groups. However, if modified properly, EGRA can also be used in the classroom to evaluate individual student progress toward reading development. Using a modified EGRA can give teachers information on the specific needs of students, allowing teachers to use the data to inform decisions about instruction and grouping of students within the classroom. In order for EGRA to be used in this way, three important changes need to be made to the instrument and how it is implemented.

110. First, instead of only testing a small representative group of students, all students in the class would need to be tested. Teachers need to know what each student knows and does not know in order to make decisions and modify instruction.

111. Second, the subsections of EGRA would need to be considered as individual measures rather than one combined measure. Teachers would use only subsections that were appropriate and made sense for assessing each student. Because teachers know their students, they would know if it made sense to assess a given student on letter naming or to begin with familiar word reading. Then if the student was successful, they would move on to oral passage reading; and if not, they would move to the letter sound and invented words subsections. It should be noted that adaptation should still include lower-level skills, because often students in higher grades are still lacking these skills and teachers need a way to know this and follow student progress.

112. Finally, using EGRA at the classroom level would require training teachers not only to administer but also to interpret the data and to know what the implications for instruction would be. Training teachers should include administering EGRA, using EGRA for monitoring progress, interpreting data, grouping for instruction, and applying strategies with students to improve skills.

113. One important note: EGRA, or a modified version of it, should NOT be used as a high-stakes accountability tool that would punish students, teachers, or schools for “underperforming.” If used in the classroom, a modified EGRA could help teachers know what students know, track progress, and make changes to instruction. But chastising students who do not perform well on the measure would defeat the purpose of the instrument. EGRA was created

43

44

to help countries and ministries of education know where to improve instruction, and it should be used for this purpose in the classroom as well.

114. For sample approaches to using EGRA as a classroom tool, please see the Liberia and Kenya materials on www.eddataglobal.org,

http://www.eddataglobal.org/

Annex A: EGRA Student Response Form

45

56

Annex B: Guyana EGRA—Student Stimuli Booklet

57

GUYANA EARLY GRADE READING ASSESSMENT

Student Stimuli Booklet

October 2008

58

Section 1; Section 3

Example : A v L

L i h R S y E O n T i e T D A t a d e w h O e m U r L G R u g R B E i f m t s r S T C N p A F c a E y s Q A M C O t n P e A e s O F h u A t R q H b S i g m i L L i N O e o E r p X N A c D d I O j e n

59

Section 4

Example : cat sick made

go sad up find come help two run see down red and play at you

chair man when now under please soon like they good thank going are know him jump once ask fly want must green sing those always many which upon sit clean stop big me house girl

60

Section 5

Example : ut dif mab

fut lus dit leb gak huz jod kib lek tob

nom rop hig reg san tup ral wix nep nad lut yod sim tat sig en mon nup sen kad

taw lew paf sal zuv ved kag vom riz gof maz kol ver et beb tib lef yag lim dov

61

Section 6

Good morning. My name is Pat.

I am seven years old. My brother’s name is Sam.

He is five years old. I also have a sister.

Her name is Ann. We live in Rose Hall.

We like to read stories. My father is a farmer.

We go to the farm every Saturday.

My mother sells fruits at the market in town.

NOTE: THIS PAGE IS FOR THE ADMINISTRATOR ONLY.

DO NOT SHOW IT TO THE STUDENT.

Section 8. Dictation

Turn the student response form to the last, lined page for writing, and place it in front of the student. Say,

I am going to read you a short sentence. Please listen carefully. I will read the whole sentence once. Then I will read it in parts so you can write what you hear. I will then read it again so that you can check your work. Do you understand what you are to do? The student will write the dictation sentence on the lined page of the response form. Read the following sentence aloud ONCE at about 1 word per second. Then give the child a pencil, and repeat a SECOND time, grouping the words “Go to the shop” - “and buy some rice” - “and sugar”. Wait 10 seconds after each group, allowing the student to write. Then repeat the sentence a THIRD time while the child is writing. Give the child up to 15 seconds to complete writing after the third reading

Go to the shop and buy some rice and sugar.

Go to the shop (wait 10 seconds)

and buy some rice (wait 10 seconds)

and sugar. (wait 10 seconds)

Go to the shop and buy some rice and sugar. (wait 15 seconds)

62

Annex C: Teacher Questionnaire

63

Ministry Of Education National Centre for Educational Resource Development Teacher Questionnaire October 2008

• NCERD is conducting a study to better understand how children learn to read. Your school was selected through a process of statistical sampling. We would like your help in this. But you do not have to take part if you do not want to.

• Your name will not be recorded on this form, nor mentioned anywhere in the survey data. The results of this survey will be published in the form of collective tables. The information acquired through this instrument will be shared with the Ministry of Education with the hope of identifying areas where additional support may be needed.

• The name of your school and the grade level and class you teach will be recorded, but only so that we can correctly link school, class, and student data so as to analyse relationships between children’s learning and the characteristics of the settings in which they learn. Your school’s name will not be used in any report or presentation. The results of analysis will be used by the Ministry of Education to help identify additional support that may be needed.

• If you agree to help with this study, please read the consent statement below, check the “Yes” box, and answer the questions in this questionnaire as completely and accurately as you can, regarding your teaching preparation and activities. It should take you no more than 10 minutes. Return the completed form to the NCERD study team before the team leaves your school.

• If after reading this message you prefer not to participate, please return this form with no markings to the study team.

CONSENT STATEMENT: I understand and agree to participate in this reading research study by filling out this questionnaire as completely and accurately as possible.

YES

Please answer all questions truthfully. Write each response in the space on the right across from each item. Where response options are given, clearly circle the number on the far right of the option that corresponds most closely to your response. For example, 3

1 Name of Region: 2 Name of School:

64

3 Grade level(s) you are teaching this year (Circle numbers for ALL grades that apply):

Grade 1 ................................................1 Grade 2 ................................................2 Grade 3 ...............................................3 Grade 4 ................................................4 Grade 5 ..............................................5 Grade 6 ..............................................6

4 Name of your Class:

5 Your gender: Male .....................................................1 Female ................................................2

6 Enrolment of your class Number of boys:

(indicate numbers by gender) Number of girls:

7 Your age at last birthday (years) ______ years

8 Do you hold a Trained Teacher’s Certificate?

No.........................................................0 Yes .......................................................1

9 What is your highest professional qualification?

None.....................................................1 Trained Teacher’s Certificate ...............2 Certificate in Education ........................3 Bachelor’s of Education........................4 Master’s in Education ..........................5 Other (specify:__________________) .6

1 How many years have you been teaching overall?

______ years

1 How many years have you been teaching as a trained teacher?

______ years

1 Does your school have a functioning Library?

No.........................................................0 Yes .......................................................1 Don’t know ...........................................9

1 If yes to question 12, about how many book titles are in the library?

____ book titles

1 If yes to question 12, do you supervise your students as they use the library?

No.........................................................0 Yes .......................................................1

65

1 Do you have a learning corner (at the school or in the classroom)?

No.........................................................0 Yes .......................................................1 Don’t know ...........................................9

1 Does your school have a functioning Parent / Teacher Association?

No.........................................................0 Yes .......................................................1 Don’t know ...........................................9

1 Do you have class meetings with the parents of your students?

No.........................................................0 Yes .......................................................1

1 If yes to Question 17, about how often do you have class meetings with parents?

About once per term or less .................1 About twice per term.............................2 About thrice per term............................3 About once a month or more ...............4

1 How long would it take you to walk to school, approximately?

More than 60 minutes...........................1 46 to 60 minutes ...................................2 31 to 45 minutes ...................................3 16 to 30 minutes ...................................4 15 minutes or less ...............................5

2 Do you use the recommended Reading Texts during reading lessons?

No.........................................................0 Yes .......................................................1

2 If yes to Question 20, how often do you use the recommended Reading Texts during reading lessons?

Rarely ...................................................1 About half the time................................2 Most but not all lessons ........................3 Every lesson.........................................4

2 If yes to Question 20, how useful do you find the recommended Reading Texts?

Not very useful......................................1 Moderately useful .................................2 Very useful ...........................................3

2 Do you have a teacher’s guide for the reading instruction curriculum?

No.........................................................0 Yes .......................................................1

2 If yes to Question 23, how useful do you find this guide?

Not very useful......................................1 Moderately useful .................................2 Very useful ...........................................3

66

2 If yes to Question 23, what one or two improvements to the guide would you recommend most strongly? (Describe):

Following are different activities you might do with your students. Think about the last 5 school days and indicate how often each of the following activities took place,

by circling the number on the right that corresponds to the closest frequency: Never

On 1 or 2 days

On 3 or 4 days Daily

2 The whole class repeated sentences that you said first.

0 1 2 3

2 Students copied down text from the chalkboard.

0 1 2 3

2 Students retold a story that they had read.

0 1 2 3

2 Students sounded out unfamiliar words. 0 1 2 3

3 Students learned meanings of new words.

0 1 2 3

3 Students read aloud to teacher or to other students.

0 1 2 3

3 Students were assigned reading to do on their own during school time.

0 1 2 3

Which of the following methods do you use to measure your students’ reading progress? Indicate how often you use each method by circling the number on

the right that corresponds to the closest frequency: Never

Once per

term or less

Once or twice

monthly Weekly or more often

3 Written evaluations 0 1 2 3

3 Oral evaluations 0 1 2 3

3 Review of portfolios and other projects

0 1 2 3

3 Review of copy books 0 1 2 3

67

3 Review of homework 0 1 2 3

3 Other methods (please describe):

During what grade level should students FIRST be able to demonstrate each of the

following reading skills? Circle number of option corresponding most closely to your response for each skill.

Before Grade 1

Grade 1

Grade 2

Grade 3

Not impor-

tant

3 Read aloud a short passage with few mistakes

7 1 2 3 8

4 Write name 7 1 2 3 8

4 Understand stories they read 7 1 2 3 8

4 Recognize letters and say letter names

7 1 2 3 8

4 Sound out unfamiliar words 7 1 2 3 8

4 Understand stories they hear 7 1 2 3 8

4 Recite alphabet 7 1 2 3 8

4 Have you attended any in-service training or professional development sessions such as workshops in the last year?

No .........................................................0 Yes .......................................................1

4 Have you received teacher training on how to teach reading within the last three years?

No .........................................................0 Yes .......................................................1

4 If yes to Question 47, indicate year(s) and for how many hours total (approx.)

Which Year(s): _________________

Total Hours: ______

4 If yes to Question 46 or Question 47, what was the most useful aspect of these trainings?

Thank you for your participation! You have been very helpful.

68

Annex D: Psychometric Testing of the Instrument

The Early Grade Reading Assessment (EGRA) ability instrumentation underwent a psychometric evaluation to determine areas of improvement or change before further analyses or data collection efforts. The standard assessment of internal consistency was followed by a more rigorous analysis using the Rasch measurement model. Below, each instrument section (listening comprehension, dictation, etc.) is discussed separately and recommendations for modifications are presented. Note that floor and ceiling effects, while typically undesirable in testing, are somewhat expected in this approach as the assessment tests a large age and grade range of students in order to establish a baseline for future assessment and analysis.

Reliability

An assessment of internal consistency was conducted on all the sections within the instrument, as well as the individual items within a section. For the continuous between-section analysis, Cronbach’s alpha was utilized and a value above 0.70 is considered internally consistent. For the dichotomous items within each section, a Kuder-Richardson Formula 20 (KR-20) analysis cutoff of 0.80 indicates good reliability.

Rasch Analysis

The next component of the EGRA ability instrumentation assessment involved the Rasch measurement model. The items within each section were assessed individually. The Rasch model makes an assessment of each item based on the difficulty of the item itself and the ability level of the individual taking the item. The discussion below refers to high student ability (i.e., responding correctly to most if not all of the items in a section) and low student ability (i.e., responding incorrectly to most if not all of the items in a section). In addition, items are discussed in terms of difficulty, where more difficult items have a low frequency of correct responses and easy items have a high frequency of correct responses. The Rasch model is not explained in detail in this particular report;5 however, the analytic output is discussed generally in relation to possible changes for future applications of the instrument.

Missing vs. Incorrect:

Before we proceed with the discussion of the Rasch analysis, it is important to note that there are two schools of thought with regard to the treatment of items not reached in timed testing situations. Some adamantly claim that the unreached items should be treated as missing, since it is possible that the participant could have provided a correct response had he/she reached the item. However, there is the opposite view that if the participant was proficient at the skill being assessed, then the item would have been answered. The two schools of thought can provide very different assessments of the same set of items. Since the purpose of the current analysis was to evaluate the instrument and not make theoretical testing stipulations, analyses were conducted with both missing and incorrect data, and recommendations for next steps take 5 Instead, see Bond, T. G., & Fox, C. M. (2001). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, New Jersey: Lawrence Erlbaum Associates.

69

into account the possible differences in the treatment of missing data. Each section therefore was analyzed with incomplete items (those items student did not reach in the assessment) both as missing and as incorrect.

Differential Item Functioning

Within the context of the Rasch analysis, an assessment of item-level bias can be obtained in the form of Differential Item Functioning (DIF). A DIF analysis evaluates the possibility that certain items have significantly different “meanings” for different groups. Typically, this analysis is utilized to detect racial bias in standardized testing items. If DIF is found, it means that two individuals at the same level of ability have a disproportionate probability of getting the item correct. If an item assessment shows a significant DIF result, it is recommended that the item be examined carefully within context, and then either removed or modified before further analyses or data collection. For the purposes of this evaluation, gender was the focus for analysis and all item nonresponse was converted to incorrect, thereby easing analytic complexity.

Section Results

Correct per minute, all section reliability = 0.92 Overall, the Cronbach’s alpha for the Guyana EGRA is 0.92, considered excellent in terms of test reliability. Individual section results are analyzed below using each of the individual words, letters, etc., as items. All test sections (evaluated with item nonresponse converted to incorrect) had an alpha near 0.9.

Item Observa-

tions SignItem–test

correlationItem–rest

correlation

Average inter-item

correlation AlphaLetter Name Knowledge (correct letters per minute) 2699 + 0.763 0.691 0.554 0.909 Initial Sound Identification 2699 + 0.644 0.547 0.584 0.918 Letter Sound Knowledge 2698 + 0.671 0.579 0.577 0.916 Familiar Word Reading (correct words per minute) 2699 + 0.906 0.875 0.518 0.896 Invented Word Decoding (correct nonsense words per minute) 2699 + 0.814 0.756 0.541 0.904 Oral Passage Reading 2699 + 0.904 0.872 0.518 0.896 Reading Comprehension (correct answers to comprehension questions) 2699 + 0.840 0.788 0.534 0.902 Listening Comprehension (correct answers to comprehension questions) 2699 + 0.588 0.481 0.598 0.922 Dictation (correct answers to dictation questions; weighted score) 2576 + 0.852 0.804 0.533 0.901

Test scale 0.551 0.917

70

Letter Name Knowledge: Reliability = 0.97

Missing: The findings from the letter name knowledge analyses showed a ceiling effect of students, where there is a substantial cluster of individuals near the top of the ability distribution. As expected (given that this skill is typically acquired in nursery school or first grade) the items taken were too easy for most participants. The three easiest items were s2l94, s2l97, and s2l99. In the case of item s2l94 (“D”), 322 of the 2699 students responded to the item, and 321 provided the correct response. Items s2l97 (“O”) and s2l99 (“e”) were similar in that there were a great deal of missing responses, but those who were able to reach those items were able to respond correctly. The most difficult item was s2l72 (“q”), where 946 students provided a response and 32% of those responded incorrectly. In an examination of item content, s2l72 was the first and only time the lower-case “q” made an appearance on the stimuli; therefore it would be expected that students found it difficult to identify. In contrast, the upper-case “O” appeared seven times in the stimuli. Finally, several items produced outlying responses (high-ability persons providing incorrect responses), s2l96 (“I”), s2l100 (“n”), s2l98 (“j”), and s2l95 (“d”). Since all of these items were at the end of the stimuli, missing data may play a large role in response peculiarities.

Incorrect: By converting the missing items to incorrect, we now account for the timing element involved. The evidence of the ceiling effect translates over from the first assessment of this section, where very proficient students not only would make it to the end of the section within the time limit, but also would provide correct responses to those later items. The floor effect is augmented by the conversion of nonresponse to incorrect. Item s2l100 (“n”) is now the most difficult item, as would be expected, as it was the last item in a timed test. The easiest item, s2l8 (“O”) had 94% of students answering correctly. The remaining items addressed a range of ability levels, with several items representing the same level. This is referred to as “stacking” and indicates that there was a redundancy in the items. Since there were 100 items in this particular section, it could be cut down and still effectively measure the range of individuals. Also, some repetitive items could be modified to address the extreme high and low ability persons or be dropped.

DIF: Two items in this section showed differential item functioning, or bias. The items, s2l1 (“L”) and s2l46 (“A”), showed a female advantage (i.e., a female student had a higher probability of responding correctly than a male student with the same ability). It is recommended that these items be examined more closely by cognitive interviewing to determine the source of the complexity.

Initial Sound Identification:** Reliability = 0.94

This set of items appears to have had a floor effect, where many individuals were clustered at the lower end of the ability spectrum. However, it also appears that despite all the individuals in the lower end of the ability spectrum, the items were slightly too easy for some of the participants. This particular section would benefit from the inclusion of more difficult items, especially for children in the higher grades, including perhaps words with first sound exceptions or slightly more complex words. In addition, the population of interest might benefit from some more simplistic items as well. The easiest item, s3f2 (“say”) showed about a 50% split of correct and incorrect responses. This indicates that the item was still relatively difficult. The most difficult item, s3f5 (“now”) showed only 36% of the students responding correctly. An

71

examination of outliers indicates that while there were extreme cases in student ability (both high and low), no items elicited extreme unexpected responses.

DIF: A single item, s3f8 (“pot”) in the initial sound identification section showed gender bias in the male direction (i.e., a male student had a higher probability of responding correctly than a female student with the same ability). Further investigation of how the students perceive this item may assist with determining the source of bias.

Letter Sound Knowledge:** Reliability = 0.97

It appears that this particular section was very difficult for students. Perhaps the participants misunderstood the instructions, or sounds were of particular difficulty for them. No matter the reason, this section could benefit from a modification to more simplistic items as well as item reduction. As this was an experimental section, future iterations could reflect these suggestions. In addition, greater student familiarity with the task through additional instruction would likely change the results significantly.

DIF: Six of the 100 items in the letter sound knowledge section showed DIF, all leaning toward a female advantage. The items showing DIF were s3l2 (“i”), s3l6 (“y”), s3l11 (“i”), s3l84 (“O”), s3l86 (“o”), and s3l87 (“E”). An assessment of the actual responses that were given when a student failed the item might establish where the difficulty and the bias came from.

Familiar Word Reading: Reliability = 0.85

Missing: There were many students at the bottom of the ability distribution (floor effect) for this section. However, the items in this section did effectively measure the ability of mid-range students. The three most difficult items, s4w29 (“know”), s4w39 (“those”), and s4w43 (“upon”), as well as the easiest, s4w50 (“girl”) had a very high instance of missing data, which may overshadow the realistic difficulties of the items.

Incorrect: The conversion from missing to incorrect preserved the floor effect. There was

some evidence of stacking, where multiple items assessed the same level of ability and did not yield additional information. Therefore, some items may be modified to address underrepresented levels in the ability continuum. For example, this section would benefit from item modifications to address the low-ability students. Perhaps more simple words (i.e., those with only two letters) could be included. In addition, some more difficult words could be added to better represent those at the high end of the ability spectrum. Interestingly, the most difficult item, s4w43 (“upon”) not only was at the end of the stimuli, which may account for its difficulty, but also is potentially a less common word. It is possible that while this list was intended to contain only familiar words, “upon” is not as common as would be expected. Finally, the investigators may want to consider ordering the items in this section according to difficulty. The instance of “easy” (i.e., two-letter) words at the end of the list may have been confounded by the time limit.

DIF: Eight items of the 50 within this section showed bias, with no definitive pattern of gender advantage. The items showing a female advantage were s4w2 (“sad”), s4w11 (“red”), s4w23 (“like”), s4w24 (“they”), and s4w32 (“once”). The items showing a male advantage were

72

s4w9 (“see”), s4w16 (“chair”), and s4w20 (“under”). It is recommended that this section be examined more closely by cognitive interviewing to determine the source of the bias.

Invented Word Decoding: Reliability = 0.94

Missing: This section also showed a floor effect. There also seemed to be quite a bit of stacking, where items s5w20 (“nad”), s5w25 (“sig”), s5w17 (“ral”), and s5w40 (“gof”), seemed to replicate measuring the same ability level. This means that there were several seemingly redundant items, capturing the same ability level. Several of these items could be modified to better capture the range of ability, or could be deleted. Again, there was quite a bit of “missingness” in these data, with almost 1000 students not responding past item s5w7, which is further confirmation of the difficulty of this section.

Incorrect: These items were simply too difficult for most of the students studied. The investigators may want to consider reducing the number of items, and simplifying some of the redundant items (i.e., those that showed evidence of stacking) into two-letter words. Those modifications should alleviate some of the magnitude of the floor effect.

DIF: Four of the items in the invented word decoding section showed bias: s5w1 (“fut”), s5w3 (“dit”), s5w9 (“lek”), and s5w13 (“hig”), all of which favored males. This section could benefit from a few sessions of cognitive interviewing to determine what prompted the students’ responses and where the sources of error were.

Oral Passage Reading: Reliability = 0.81

Missing: In some instances there was evidence of stacking in this section, and there appeared to be some clustering of individuals both at the top and bottom of the ability distribution. This indicates that while these items seem to have adequately measured mid-level students, there need to be more items representing the extremes of high and low ability. There were some oddities in the easiest item s6w58 (“the”) and the most difficult item s6w35 (“Hall”), which may be due to missing values present in the data. Over 50% of the participants did not respond past item s6w26, further confirming this section’s difficulty.

Incorrect: The change of missing information to incorrect preserved the initially found ceiling and floor effect. This indicates that for some individuals, this reading section was too easy, and perhaps a more difficult sentence or two would better address their ability level. However, for the majority, the reading passage was too difficult, and the investigators might want to add a few more simplistic sentences. Interestingly, the most difficult item, s6w35 (“Hall”) was not necessarily a difficult or unfamiliar word. Perhaps it was the context of the proper noun that made this particular item difficult. Considering that there were many more complex words in this section, further evaluation with cognitive interviewing might be helpful.

DIF: Thirteen of the 61 items in this section showed significant differential item functioning, four of which favored males. The items favoring females were s6w2 (“morning”), s6w6 (“Pat”), s6w7 (“I”), s6w9 (“seven”), s6w10 (“years”), s6w23 (“also”), s6w27 (“Her”), s6w34 (“Rose”), and s6w35 (“Hall”). The items favoring males were s6w22 (“I”), s6w46 (“we”), s6w47 (“go”), and s6w48 (“to”).

73

74

Reading Comprehension:** Reliability = 0.83

The reading comprehension section showed a definite floor effect, with the vast majority of the students getting the items incorrect. The most difficult item, s6p6_m (“How many children are in Pat’s family?”) required that the student remember and calculate the response rather than simply remembering the answer straight from the text. Because the passage was removed from the child, there was a memory component involved in responding to these items and that may account for the difficulty of this section.

DIF: Half of the items in the reading comprehension section showed DIF. It is recommended that this section be examined at length with cognitive interviewing and reevaluated. The item favoring females was s6p6_m (“How many children are in Pat’s family?”). The items favoring males were s6p9_m (“How often do the children go to the farm?”) and s6p10_m (“Where does Pat’s mother work?”).

Listening Comprehension:** Reliability = 0.61

The listening comprehension section also showed a substantial floor effect, in which over 45% of the students surveyed responded incorrectly on each item. The most difficult item, s7p3_m (“What did the duckling see that was important?”), required that the child not only understand the term “important” but also be able to interpret that definition in the context of the story. This is an intriguing item in that there were some students who were able to make this assessment, despite its difficulty.

DIF: There was no indication of differential item functioning (i.e., bias) in these items.

Dictation:**

Reliability = 0.87. This section did an adequate job of assessing students in the mid to high range of the ability distribution. There was still evidence of a floor effect, and the bulk of the items were effective at targeting the high end of the ability distribution. However, the easiest item, s8p6 (“Used appropriate direction of text”), followed by s8p5 (“Used spacing between words”), attempted to evaluate the most basic components of the written language. The most difficult item, s8p9 (“Used full stop at end of sentence”—a period punctuation mark), which is often troublesome for adults, had only 8% of students respond correctly. This section could benefit from additional simplistic assessments of writing, such as correct letter formation.

DIF: The single item showing bias in this section favored females: item s8p7 (“Used capital letter for the word “Go”).

sample and organization of work - global reading network · added to make up for differences in...

Documents