standards based classroom assessments of english proficiency

LORENA LLOSA

PREPARED BY: SAEED SHIRI , AMIR HAMID FOROOGH AMERI

AHFAMERI@GMAIL .COM

Standards-based classroom assessments of English proficiency: A review of issues,

current developments, and future directions for research

Abstract

Little attention has been paid to the role of standards-based assessments in the classroom.

The purpose of this paper is to discuss key issues and challenges related to the use of standards-based classroom assessments to assess English language learners’ English proficiency.

First, the paper describes a study of a standards-based classroom assessment of English proficiency in a large urban school district in California. Second, using this study as an example and drawing from the literature in language testing on classroom assessment, this paper highlights the major issues and challenges involved in using English proficiency standards as the basis for classroom assessment. Finally, the article outlines a research agenda for the field given current developments in the areas of English proficiency standards and classroom assessment.

Intro

Aim

Study

Why classroom-based assessments?

Classroom assessments make sense in a standards-based system. Without the limitations of a typical high-stakes test, classroom assessments can be used to assess students’ mastery of more standards, even those that are difficult to gauge with a high-stakes test. Assessments can be conducted in more authentic and meaningful ways in the classroom, and can be based on a wider and more informative range of student performance. Perhaps most importantly, teachers and students have immediate access to classroom assessment results and can use them to improve teaching and learning (Popham, 2003; Taylor, 2002).

The purpose

The purpose of this paper is to discuss key issues and challenges surrounding the use of standards-based classroom assessments (SBCA) of English proficiency and to explore avenues for future research given current policies and developments in the field. As Rea-Dickins (2007) points out, there is considerable variation in the meaning of class-room assessment. For the purpose of this paper, classroom assessment will be broadly defined to include any assessment carried out by teachers in the classroom for summative or formative purposes.

Procedure

First (I), the paper describes a study of a SBCA of English proficiency in a large urban school district in California (Llosa, 2005, 2007, 2008). Second (II), using this study as an example and drawing from the literature in language testing on classroom assessment, this paper highlights some of the major issues and challenges involved in using ELP standards as the basis for classroom assessment. The paper then discusses current policies and developments in the areas of ELP standards and classroom assessment that have the potential to impact SBCA practices. Finally (III), the article outlines a research agenda for the study of classroom assessments based on ELP standards. More research is needed to determine whether and how SBCA can effectively promote the goals of standards-based reform in education, and in English language education in particular.

(I) The ELD Classroom Assessment study: An example

Llosa (2005, 2007, 2008) focused on the English Language Development (ELD) Classroom Assessment, a SBCA of English proficiency based on teacher judgments.

This classroom assessment was used in a large urban school district to make high-stakes decisions about students’ progress from one ELD level to the next (Levels 1–5) and served as one criterion for reclassification into the mainstream. It was also meant to be used to inform teaching and learning.

There is a separate set of assessments for different grade levels: kindergarten through grade two, grades three through five, grades six through eight, and grades nine through twelve. Each ELD Classroom Assessment consists of a list of all the California ELD standards for a given ELD level divided into three sections – Listening/Speaking, Reading, and Writing – following the same organizational structure as the California ELD Standards. The Reading standards are further divided into the following sections: Word Analysis, Fluency and Vocabulary, Comprehension, and Literary Response. The Writing standards are divided into Strategies, and Applications and Conventions.

(I) The ELD Classroom Assessment study: An example(Contd.)

Teachers are expected to score student progress toward each standard using the following scale by taking into account the overall performance of the student in the class:

4 Advanced Progress: Exceeds the standards for the identified ELD level. 3 Average Progress: Meets the standards for the identified ELD level. 2 Partial Progress: Demonstrates some progress towards mastery of the standards. 1 Limited Progress: Demonstrates little or no progress towards mastery of the standards.

The purpose of the study was twofold: to determine the extent to which the ELD Classroom Assessment measures the same language constructs as the statewide standardized test, which is also aligned to the California ELD standards; and to examine how teachers make decisions about students’ language ability when using this classroom assessment. The study used confirmatory factor analysis of MTMM data to examine the constructs assessed by the ELD Classroom Assessment in relation to the standardized test, and verbal protocols of teachers while scoring the assessment to investigate their decision-making processes. The overall purpose of the study was to gather evidence in support of a validity argument that linked performance on the ELD Classroom Assessment to interpretations about a student’s English proficiency.


The main finding of the study was that teachers, when using the ELD, are good judges of students’ overall English proficiency. The quantitative analysis revealed that the ELD measures the same constructs as the statewide standardized test of English proficiency. The qualitative analysis, however, revealed that there are inconsistencies in teacher scoring. It turns out that, in addition to students’ language proficiency, teachers’ scores on the ELD reflect other factors, such as students’ personality and classroom behavior, the teachers’ beliefs about assessment and grading, and external pressure to advance students to the next level. The scores also reflect how teachers interpret the standards themselves and how they interpret the scoring criteria (Llosa, 2005, 2008).


Most relevant to this paper is the finding that teachers did not interpret the standards consistently and, as a result, the extent to which a student was determined to master a standard was largely based on a particular teacher’s interpretation of that standard.

The conclusion of the study was that the usefulness of this SBCA of English proficiency based on teacher judgments depends in large part on the purpose for which it will be used. The fact that teachers are good judges of students’ overall language ability suggests that scores can be used to make overall, summative decisions. On the other hand, the fact that teachers are not very good judges of students’ mastery of individual standards representing specific aspects of language ability calls into question the usefulness of this classroom assessment for some formative purposes. Without a common understanding of the standards and what constitutes mastery of a particular standard, teachers would not be able to provide students with the necessary instruction to develop mastery.

(II) Issues and challenges

This section introduces a number of different issues that emerged from this study and relates these to research and writing in the field.

Many of the issues and challenges involved in using SBCAs of English proficiency revealed by the ELD Classroom Assessment study (Llosa, 2005, 2007, 2008) have been identified by Brindley (1998). Although the focus of his paper is not on classroom assessments, the outcomes or standards frameworks he describes in these contexts were often used by teachers in the classroom. The challenges he identifies include tremendous variability in the way in which standards are interpreted and used by teachers, lack of teacher professional development, the tension between summative and formative uses of SBCAs, and concerns about the quality of the standards themselves. The analysis of these issues that follows is organized around three interrelated components in SBCA: the teacher, the assessment purpose, and the standards.

(II.I) The role of the teacher

By definition teachers play a key role in classroom assessment. Teacher judgments, however, have often been found to be inconsistent and unreliable (Brindley, 1998). As Llosa (2005, 2008) illustrates, a number of factors other than language ability affected teachers’ assessment of students when using the ELD CA. Among these factors were the teachers’ beliefs about assessment, external pressures, and other characteristics of the students being assessed. How teachers interpreted and applied the standards and the rating scale also affected their scores.

This variability in teachers’ assessment of students and in their interpretations of standards is not uncommon. This variability can be attributed in part to lack of professional development.


Llosa (2005) suggests that lack of professional development might explain teachers’ inconsistencies in scoring and differences in how the standards were interpreted on the ELD Classroom Assessment.

The need for adequate preparation of teachers has been well-documented in educational measurement (Cizek, 2007) and in language testing (Brindley, 1998, 2001). Brindley (2001) argues that ‘unless greater attention is given to providing adequate time allocation and appropriate forms of professional development, the many potential benefits of involving teachers in assessment will not be realized’. In the case of SBCA, effective professional development on the standards and assessment principles could, among other things, help reduce variability and improve the overall quality of teacher judgments.


Some argue, however, that variability in teacher judgments is not necessarily a problem and question the assumption that classroom assessment should be required to meet conventional psychometric standards of reliability and validity, especially when the assessment is used for formative purposes.

In this approach to classroom assessment, the teacher’s main role is to support and facilitate learning, whereas in the case of the ELD CA, the teacher’s role was as rater/examiner. The vital role of the teacher in classroom assessment is unquestionable; what shape that role takes, however, depends mainly on assessment purpose.

(II.II) The role of assessment purpose

They were not particularly interested in the extent to which a student had mastered a specific standard or the progress that a student had made towards mastery of a standard. On the other hand, to use the ELD CA to inform the teaching and learning process, a teacher had to attend to a student’s mastery and progress towards mastery of each standard in order to provide helpful feedback to the student and work with them to help them develop mastery. As Brindley (1998) explains, different audiences concerned with assessment for different purposes, have different information needs and ‘system information needs will override those of formative assessment’, which is exactly what happened with the ELD CA. In practice, most teachers used it to satisfy accountability requirements only.


One of the most frequently discussed tensions surrounding classroom assessment has to do with assessment purpose. The discussion is framed in terms of the relationship between formative and summative purposes for assessment.

It is important to explore the relationship between formative and summative assessment because there are tensions inherent in the purposes and audiences of SBCAs (Brindley, 1998). The ELD CA provides a good illustration of this tension since the ELD CA was intended to be used for district accountability purposes and to inform teaching and learning in the classroom. School and district administrators, however, were only interested in getting from the teacher one score that would tell them whether a particular student could be moved from one ELD level to the next.


Another issue of concern the meaning of “formative”. The concern is that as interest in using formative assessment grows in the U.S. so do the misinterpretations of the concept. Among the misinterpretations is the notion that any ongoing, frequent assessment, any alternative form of assessment (such as an oral presentation or a writing portfolio), and any detailed, fine-grained assessment are by definition formative.

As mentioned earlier, consideration of the purpose of assessment will impact the role teachers play in the assessment activity. It also determines the measurement paradigm adopted to evaluate the quality of the assessment, with many arguing that psychometric criteria such as reliability and validity need to be reinterpreted in the context of formative classroom assessment (Davison & Leung, 2009; Leung, 2005; Shepard, 2009; Teasdale & Leung, 2000).

(II.III) The role of standards

It is important to examine the relevance of ‘standards’ to SBCA practices because the ELP standards represent the operational definition of the English proficiency construct.

One of the main criticisms of ELP standards is that they lack empirical foundation . Standards documents are typically developed by groups of teachers and experts in a particular context based on their experience and intuition, and ‘may or may not be checked against actual progress in the classroom’. As a result, the various level descriptors in some standards documents lack clarity and internal coherence.

(II.III) The role of standards

In spite of the many problems surrounding the standards themselves, they provide teachers, students, and other stakeholders with shared goals and guidelines and a common language for discussing English language proficiency and development. As mentioned earlier, in the ELD Classroom Assessment study, teachers were found to be, on average, good judges of their students’ language ability despite the inconsistencies in scoring.

Finally, perhaps the most important issue revealed by the ELD Classroom Assessment study is the scarcity of research that has specifically focused on SBCAs of English proficiency in the US context. This gap is particularly problematic given current developments in the areas of classroom assessment and standards that have the potential to impact SBCA practices in schools. These developments are discussed in the next section.

(III) Current developments

The increasing interest in formative assessment as a means for improving student achievement has prompted test publishing companies to develop standards-based assessments for classroom use. These companies are working on benchmark or interim assessments meant to help teachers monitor student progress towards mastering the standards and getting a sense of how students will perform on the high-stakes tests (Cizek, 2007). Some of these assessments consist of pools of items that address state standards that schools and districts can draw from to develop their own assessments (Pearson, 2005). Recent national policies are also encouraging the inclusion of formative assessment as a component of assessment systems for accountability. The Race to the Top Assessment Program has funded two state consortia to develop new assessment systems that measure student skills against the Common Core Standards, a common set of standards in math and language arts (Heritage, 2010).


The surge of interest in formative assessment will produce many and different types of standards-based assessments for classroom use, although it is unclear at this point to what extent these new SBCA will actually be ‘formative.’ Nevertheless, there will be numerous opportunities to study SBCA practices.


Another important development is in the area of ELP standards. The ELP standards developed by the WIDA consortium (2004, 2007), for example, directly address assessment purpose and are based on a different conceptualization of academic English proficiency. These standards, so far adopted by 26 states, include five standards regarding language use in both social and academic contexts presented in two separate frameworks.

The summative framework identifies model performance indicators that represent the ‘outcomes of learning,’ whereas the formative framework ‘is geared toward guiding student learning and teacher instruction on an ongoing basis’ and is ‘intended to capture those aspects of instruction that are less typically measured by a test but are important to teaching and learning’.


The formative framework includes interactive supports such as ‘opportunities for students to work as partners or in small groups, receive immediate feedback from peers or teachers, engage in self assessment during long-term projects, and integrate technology into their assignments’ (Gottlieb, Cranley, & Cammilleri, 2007, p. 9). The separate formative and summative frameworks have the potential to give teachers very helpful guidance in the development of SBCAs for various purposes.

Most importantly, these standards developed by WIDA and then augmented and adopted by TESOL, differ from those developed pre-NCLB in the ways in which they operationalize the construct of academic English. Unlike pre-NCLB standards that were designed to align to English Language Arts standards, the new standards link ELP to four content areas – language arts, mathematics, science, and social studies – thus better representing the construct of academic English. The nature of these new ELP standards, however, might present new and unique challenges for their use as the basis for classroom assessment.

Defining a research agenda

Based on their observations, Rea-Dickins (2007) and Davison and Leung (2009) call for further on five interrelated areas of research for classroom and teacher-based assessment in English language education.

Future research on SBCA is needed in five interrelated areas: First, studies

should focus on the nature of post-NCLB ELP standards, specifically whether and how this operationalization of the academic English construct impacts SBCA practices. Second, there is a need for descriptive studies that document how teachers use standards as the basis for classroom assessment, whether they develop the assessments themselves, rely on templates that are provided to them, or use commercially available products. Third, in addition to descriptive studies that capture authentic assessment practices in classrooms, research should be designed that systematically and experimentally examines the impact of different SBCA practices on teachers and students. Fourth, these studies need to be guided by appropriate validation frameworks that take into account assessment purpose. Finally, an important line of research should focus on teacher knowledge and professional development needs.

Post-NCLB ELP standards and teachers’ SBCA practices

Post-NCLB ELP standards operationalize the construct of academic English by directly linking ELP to the four content areas of language arts, math, science and social studies. The close linking of language proficiency with content area knowledge in these standards may present challenges to teachers. One such potential difficulty has to do with the relationship between these ELP standards and content area standards.

Teachers often assumed that by addressing the ELA standards they would automatically address ESL standards, but that was not necessarily so. The post-NCLB ELP standards might run a similar risk of being ignored at the expense of content standards.

The WIDA Resource guide explains that the content topics are ‘intended to illustrate how language lessons can be embedded in content lessons’ and ‘are not meant to imply that language learning is automatic when content topics are taught’ (Gottlieb, Cranley, & Cammilleri, 2007, p. 25). This is a critical distinction, but the extent to which teachers will be able to attend to this distinction and pay adequate attention to both ELP standards and content standards simultaneously is unclear.

Descriptive studies on teachers’ use of SBCA

Regardless of whether states use pre or post-NCLB ELP standards, observational and descriptive studies are needed that document the extent to which SBCAs are used by teachers and how.

More specifically, - How do teachers go about developing classroom assessments to

assess students’ mastery of ELP standards? -How do teachers work with standards-based formative assessment

systems, such as FLARE, to develop and engage in classroom assessments?

-H ow do teachers use commercially developed benchmark and interim assessments?

Experimental studies of SBCA

Research is needed to examine the use and impact of carefully implemented classroom assessments in systematic ways. The increasing interest in various forms of SBCA, those used for formative purposes in particular, affords the opportunity to conduct experimental or quasi-experimental studies that examine specific characteristics of the various assessments and assessment practices, and the extent to which the information they provide and the decisions that are made on the basis of that information serve to accomplish the intended purpose, whether it be informing teachers, increasing student motivation, or improving student learning.

Validation frameworks

Careful attention to assessment purpose will be critical in any validity investigation that focuses on SBCAs. These assessments can serve a number of functions including assigning grades, monitoring progress, helping to plan instruction, predicting future performance and enhancing student learning. Justifying assessment interpretations and uses will require a different type of validity argument and supporting evidence depending on the purpose.

Research on teacher knowledge and professional development needs

Finally, it has been stressed throughout this article that effective classroom assessment depends on teacher knowledge and, therefore on professional development that will help foster this knowledge in teachers. Thus, conversation about the meaning of the standards at the classroom, school, district and state levels will be essential to deep and meaningful work with the standards and the ability to use them as the basis for classroom assessment.

Studies could examine the levels of expertise and/or experience required for teachers to become competent developers and users of SBCAs of English for summative and formative purposes.

Research could also investigate the types of professional development that help teachers acquire expertise in both the standards and assessment that would allow them to effectively engage in various SBCA practices.

Final comment

Finally, the research agenda outlined above needs to take place simultaneously with research on the validity and impact of the standards themselves. Bailey and Huang (this issue) provide helpful recommendations for revising, augmenting and validating ELP standards.

standards based classroom assessments of english proficiency

Education