1CRESST ONR/NETC Meetings, 17-18 July 2003, v1

ONR Advanced Distributed Learning

Language Factors in the Assessment of English Language Learners

Jamal Abedi

University of California, Los AngelesNational Center for Research on Evaluation, Standards, and Student Testing

(CRESST)

July 18, 2003

2CRESST ONR/NETC Meetings, 17-18 July 2003, v1

The “No Child Left Behind Act” mandates inclusion The “No Child Left Behind Act” mandates inclusion

of of ALLALL students students

• Goals 2000• Title I and VII of the Improving America’s School

Act of 1994 (IASA)• However, language factors create a major obstacle

in including English language learners (ELLS)• Because of possible English language deficiencies,

ELL students have been traditionally excluded from large-scale National and State assessments.

3CRESST ONR/NETC Meetings, 17-18 July 2003, v1

CRESST Studies on the Impact of Language

Factors on the Assessment of ELL Students:

4CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Study #1

Analyses of extant data (Abedi, Lord, & Plummer, 1995).

Used existing data from NAEP 1992 assessments in math and science.

SAMPLE: ELL and non-ELLs in grades 4, 8, and 12 main assessment.NAEP test items were grouped into long and short and linguistically complex/less complex items.

Findings

ELL students performed significantly lower on the longer test items. ELL students had higher proportions of omitted and/or not-reached items.ELL students had higher scores on the linguistically less-complex items.

5CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Study #2

Interview study (Abedi, Lord, & Plummer, 1997)

37 students asked to express their preference between the original NAEP items and the linguistically modified version of these same items. Math test items were modified to reduce the level of linguistic complexity.

Findings Over 80% interviewed preferred the linguistically modified items over the original version.

6CRESST ONR/NETC Meetings, 17-18 July 2003, v1

The revised items need less time to respond:

“It’s easier to read, and it gets to the point, so you won’t have to waste time.”

“I might have a faster time completing that one cause there’s less reading.”

“Less reading; then I might be able to get to the other one in time to finish both of them.”

“Cause it’s, like, a little bit less writing.”

7CRESST ONR/NETC Meetings, 17-18 July 2003, v1

The vocabulary in the revised items was more familiar:

“This one uses words like ‘approximation,’ and this one uses words that I can relate to.”

“It doesn’t sound as technical.”

“I can’t read that word.”

“Because it’s shorter and doesn’t have, like, complicated words.”

8CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Study #3

Impact of linguistic factors on students’ performance (Abedi, Lord, & Plummer, 1997).

Two studies: testing performance and speed.

SAMPLE: 1,031 grade 8 ELL and non-ELL students.41 classes from 21 southern California schools.

FindingsELL students who received a linguistically modified version of the math test items performed significantly better than those receiving the original test items.

9CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Study #4The impact of different types of accommodations on students with limited

English proficiency (Abedi, Lord, & Hofstetter, 1997)

SAMPLE: 1,394 grade 8 students. 56 classes from 27 California schools.

FindingsSpanish translation of NAEP math testSpanish-speakers taking the Spanish translation version performed significantly lower than Spanish-speakers taking the English version. We believe that this is due to the impact of language of instruction on assessment.

Linguistic ModificationContributed to improved performance on 49% of the items. Extra TimeHelped grade 8 ELL students on NAEP math tests.Also aided non-ELL students. Limited potential as an assessment accommodation.

10CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Study #5

Impact of selected background variables on students’ NAEP math performance(Abedi, Hofstetter, & Lord, 1998).

SAMPLE: 946 grade 8 ELL and non-ELL students. 38 classes from 19 southern California schools.

Findings

•Four different accommodations used (linguistically modified, a glossary only, extra time only, and a glossary plus extra time).•The glossary plus extra time was the most effective accommodation.

Glossary plus extra time accommodation•Non-ELLs showed a greater improvement (16%) than the ELLs (13%). •This is the opposite of what is expected and casts doubt on the validity of this accommodation.

11CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Study #6The effects of accommodations on the assessment of LEP students in NAEP

(Abedi, Lord, Kim, & Miyoshi, 2000)

SAMPLE: 422 grade 8 ELL and non-ELL students. 17 science classes from 9 southern California schools.

Findings •Some forms of accommodations may help the recipients with the content of assessment. For example, a dictionary defines all the words in a test, both content and non-content.

A Customized Dictionary Easier to use than a published dictionary Included only non-content words in the test. ELL students showed significant improvement in performance. No impact on the non-ELL performance.

12CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Study #7

Language accommodation for large-scale assessment in science(Abedi, Courtney, Leon, Mirocha, & Goldberg, 2001).

SAMPLE: 612 grades 4 and 8 students. 25 classes from 14 southern California schools.

Findings

A published dictionary was both ineffective and administratively difficult as an accommodation. Different bilingual dictionaries had different entries, different content, and different format.

13CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Study #8

Language accommodation for large-scale assessment in science (Abedi, Courtney, & Leon, 2001)

SAMPLE: 1,856 grade 4 and 1,512 grade 8 ELL and non-ELL students.132 classes from 40 school sites in four cities, three states.

Findings Results suggested: linguistic modification of test items improved performance of ELLs in grade 8. No change on the performance of non-ELLs with modified test. The validity of assessment was not compromised by the provision of an accommodation.

14CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Study #9 Impact of students’ language background on content-based performance: analyses of

extant data (Abedi & Leon, 1999). Analyses were performed on extant data, such as Stanford 9 and ITBSSAMPLE: Over 900,000 students from four different sites nationwide.

Study #10Examining ELL and non-ELL student performance differences and their relationship to

background factors (Abedi, Leon, & Mirocha, 2001). Data were analyzed for the language impact on assessment and accommodations of ELL students.SAMPLE: Over 700,000 students from four different sites nationwide.

FindingsThe higher the level of language demand of the test items, the higher the performance gap between ELL and non-ELL students. Large performance gap between ELL and non-ELL students on reading, science, and math problem solving (about 15 NCE score points). This performance gap was zero in math computation.

15CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Study #11

Research-supported accommodations for English language learners in NAEP (Abedi, Courtney & Leon, 2002)

SAMPLE: 607 grade 4 students (46% ELLs and 54% non-ELLs) and 542 grade 8 students (47% ELLs and 53% non-ELLs)

Accommodations: Computer testing, customized dictionary, and extra time. A reading composite score was used as a covariate. Student responses to accommodation follow-up questionnaires and background questionnaires were analyzed.

Findings

The computer testing was the most effective accommodation. It provided an alternative test item delivery and an easy-to-access gloss of non-math lexicon. The customized dictionary was also shown to be effective. Since non-ELLs who were accommodated performed the same as non-ELLs who were

not accommodated, the two effective accommodations are deemed valid.

16CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Study #12

Opportunity to Learn for English Language Learners (Abedi, Courtney, & Leon, 2002)

SAMPLE: 607 grade 4 students (46% ELLs and 54% non-ELLs) and 542 grade 8 students (47% ELLs and 53% non-ELLs)

Findings •Student self-reported OTL correlated with their actual performance in math.•Teacher-reported OTL (their indication that they taught the materials) did not correlate as high with the student performance.

17CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Study #13

Issues and problems in classification of students with limited English proficiency (Abedi & Leon, 2002)

This study examined the validity of LEP classification scheme by analyzing extant data.

LEP classification codes correlated poorly with test scores.

In lower grades, low-performing ELLs tend to remain classified as LEP.

There appears to be a tendency to reclassify these students in higher grades.

Correlation between test scores and LEP classification varies substantially among districts.

The results of longitudinal analyses indicated that in addition to language proficiency, student background variables were also predictors of LEP classification.

18CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Study #14

Opportunity to learn for English language learners: OTL and language interaction

(Abedi, Courtney, & Leon, 2002) This study examines the differences, if any, in opportunity to learn (OTL) between ELLs and their non-ELL peers in grade 8 math. SAMPLE: 700 grade 8 algebra students (in the 2-year track)

Research Questions:•Do ELL students receive the same level of OTL as non-ELL students? (Observation/teacher interview/ student OTL questionnaire and field testing)•Are the OTL factors influenced by student level of English language proficiency? (Observation/teacher interview/ student OTL questionnaire)•Are the OTL factors for ELL students influenced by the teacher’s impression of the ELL students’ ability to learn? (Observation/teacher interview/ student OTL questionnaire)

19CRESST ONR/NETC Meetings, 17-18 July 2003, v1

References to CRESST Studies

20CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Reports

21CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Abedi, Courtney & Leon (2002) Research-Supported Accommodations for English Language Learners in NAEP. Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing.

Abedi, J., Lord, C., & Hofstetter, C. (2001). Impact of Selected Background Variables on Students’ NAEP Math Performance. National Center for Education

Statistics (NCES), Working Paper, Publication #: (NCES 200111).

Abedi, J. (2001). Assessment and Accommodations for English Language Learners: Issues and Recommendations. Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing. Policy Brief 4.

Abedi, J.; Courtney, M. and Leon, S. (2001). Language Accommodation for Large-scale Assessment in Science: Assessing English Language Learners. Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing.

22CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Abedi, J.; Lord, C.; Kim, C. & Miyoshi, J (2001). The effects of accommodations on the assessment of LEP students in NAEP. National Center for Education Statistics (NCES), Working Paper, Publication #: (NCES 200113).

Abedi, J., Courtney, M., Mirocha, J., Leon, S., & Goldberg. J. (2000). Language Accommodation for Large-scale Assessment in Science. Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing.

Abedi, J., Lord, C., Kim, C., & Miyoshi, J (2000). The effects of accommodations on the assessment of LEP students in NAEP. Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing. CSE Technical Report #537.

Abedi, J., Leon, S., & Mirocha, J. (2001). Students’ performance differences in standardized achievement tests and background factors: Analyses of

Extant Data. University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing.

23CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Abedi, I. Leon, S. (1999). Impact of students’ language background on content-based performance: Analyses of extant data. University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing.

Abedi, J., Hofstetter, C., Baker, E. & Lord, C. (1998). NAEP math performance and test accommodations: Interactions with student language background, Draft Report. Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing. CSE Technical Report #536.

Abedi, J., Lord, C., & Hofstetter, C. (1997). Impact of selected background variables on students’ NAEP math performance. Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing. CSE Technical Report #478.

Abedi, J., Lord C., & Plummer, J. R. (1997). Language Background as a Variable in NAEP Mathematics Performance. Los Angeles: Center for the Study of Evaluation, CSE Technical Report # 429.

24CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Publications

25CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Abedi, J. (2002). Standardized achievement tests and English language learners: Psychometrics and linguistics issues. Educational Assessment (accepted for publication in Educational Assessment).

Abedi, J., Hofstetter, C., & Lord, C . (2002) Assessment Accommodations for English Language Learners: A Review of Empirical Research and Policy Issues. Review of Educational research (submitted for publication).

Abedi, J. (2002). Issues and problems in classification of students with limited English proficiency. Educational Measurement: Issues and Practice. (submitted for publication).

Abedi, J. (2002). Assessing and Accommodations of English language learners: Issues, concerns and recommendations. Journal of School Improvement. v3, n1, Spring 2002.

26CRESST ONR/NETC Meetings, 17-18 July 2003, v1

Abedi, J., Lord, C (2001). The Language Factor in Mathematics Tests. Applied Measurement in Education, 14, 3, June 2001.

Abedi, J. (2000). Loaded Questions? American Language Review, The Magazine for Language Teaching Professional. July/August 2000.

Abedi, J., Lord, C., Hofstetter, C., & Baker, E. (2000) Impact of accommodation strategies on English language learners’ test performance. Educational Measurement: Issues and Practice, 19, 3, pp. 16-26.