strengthening assessment accountability for arrived english learners essa · 2017-06-28 · english...

Strengthening Assessment & Accountability for Recently Arrived English Learners Under ESSA: Policy and Technical Innovations

Linquanti, Cook, Yoon, & Miller

CCSSO National Conference on Student Assessment6/28/2017 Austin, TX 1


Robert Linquanti, WestEdH. Gary Cook, WCERKim Miller, Oregon Department of EducationIlhye Yoon, Maryland State Department of Education

CCSSO National Conference on Student Assessment Austin, TX June 29, 2017

Some Key Questions

Who are recently arrived English learners (RA ELs)?

How can states and districts develop and implement services responsive to their strengths and needs?

What policy options and challenges does ESSA offer states and LEAs serving RA ELs?

How can states build more nuanced and equitable assessment and accountability systems for RA ELs?




Symposium Overview

Share project background and context

Discuss conceptual, policy, and technical issues in assessment and accountability for Recently Arrived English Learners

Review RA EL assessment and accountability model development guidelines

Empirically examine and compare results from different RA EL model options

Reflect on applications to date in two project states (MD, OR)

Project Background & Context• December 2014: ED approves Florida’s NCLB flexibility waiver for including RA ELs in Title I accountability 9 other states file waivers

▪ Argument: RA ELs need additional time to learn English before R/LA can be measured accurately in English; relief from accountability needed

• November 2015: ED OSS launches community of practice among 7 states, experts, stakeholders to explore issues and generate guidance for states

• December 2015: Every Student Succeeds Act (ESSA) signed into law

• May‐December 2016: ESSA implementing regulations drafted, finalized

• January 2017: RA EL Accountability Guide for states published by ED

• March 2017: ESSA implementing regulations rescinded via CRA; empirical analysis working paper published by WCER




Community of Practice Members

▪ Recently Arrived English Learners (RAEL) Working Group

Subject Matter Experts Colorado Florida Georgia Maryland New York Ohio Oregon

Project Co‐Directors▪ H. Gary Cook▪ Robert Linquanti

Partner Organizations

▪ CCSSO▪ CGCS▪ LCCR

▪ NCLR▪ MALDEF▪ LULAC

▪ Thomas Ahn▪ Diane August▪ Melissa Bowles▪ David Francis▪ Eugene Garcia▪ Pete Goldschmidt

▪ Brian Gong▪ Kenji Hakuta▪ Andrew McEachin▪ Charlene Rivera▪ Ed Roeber

States:

Key Issues

CONCEPTUAL

• Who is a RA EL?

• What does it mean for an RA EL to meaningfully participatein Title I assessment/ accountability?

POLICY / TECHNICAL

• How to determine ESSA RA EL policy option(s)?

• How to develop, implement, and evaluate RA EL model?




ESSA Title I Assessment and Accountability (§1111)

• RA EL is an EL who has been “enrolled in a school in one of the 50 States…or DC for less than 12 months”

• Provides options for including recently arrived ELs §1111(b)(3)(A)

▪ (i) Exclude from R/LA assessment and accountability in Year 1; test & incorporate proficiency result in accountability Year 2

OR

▪ (ii) Assess and report R/LA (but not accountability) in Year 1; test and incorporate progress result in Year 2; incorporate proficiency result in Year 3

ESSA Title I RA EL Exception Options

Final ESSA regulations* clarified “option 3”:

• Either apply one option uniformly to all or assign most appropriate for each RA EL taking into account:

▪ Initial ELP level

▪At State’s discretion: Grade, age, L1 proficiency level, and limited or interrupted formal education (SIFE)

• Must establish a uniform procedure for making this student‐level determination

*Rescinded in March 2017 via Congressional Review Act




RA EL Guide Objectives and Uses

• Provide evidence‐based / best‐practice guidelines for developing RA EL accountability model grounded in state’s theory of action

• Use guidelines to

▪ Develop an RAEL accountability model

▪ Formatively evaluate proposed or established approaches (Linquanti & Cook, 2017)

Guidelines (1 of 7)

1. Establish a clear, statewide definition of a RA EL

▪ Operationalize “12 months” – cumulative enrolled in any/all US public schools

▪ Standardize questions and process for parent report on total months student enrolled

▪ Explain reasons for asking, set protocols and guidance to avoid “restarting clock” or discouraging enrollment




Maryland’s Definition

▪ Recently arrived ELs (RAELs)▪A recently arrived EL is an English learner who has attended schools in the US for less than 12 cumulative school months.

▪ ELs with interrupted education▪ Students who have missed six (6) months or more of formal schooling prior to enrollment in a US school above the age of 7.

▪ Recently arrived ELs (RAELs)▪A recently arrived EL is an English learner who has attended schools in the US for less than 12 cumulative school months.

▪ EL Students with interrupted formal education (EL‐SIFE)▪ Meet at least one of the following two categories:

▪ 1. Come from a home where a language other than English is spoken and enter a school in the US after grade two;

▪ 2. Are immigrant students who enter a school in the United States after grade 2

▪ And meet the following conditions:

▪ a. Have had at least two years less schooling than their peers; and,

▪ b. Function at least two years below expected grade level in reading and in mathematics; and,

▪ c. May be pre‐literate in their native language.

Oregon’s Definition




Guidelines (2 of 7)

2. Develop a theory of action for RAEL accountability models

▪ Focus on expected positive outcomes for students and educators

▪ Optimize meaningful, appropriate participation

▪ Examine fit with larger TOA for ELs, all students

▪ Consider timing, availability, and precision of initial ELP assessment results for option 3

Reading Performance by RA EL Status/ELP Level

Figure 1. Grade 5 Reading/Language Arts Performance by EL Status/ELP Level, Test A

Figure 2. Grade 5 Reading/Language Arts Performance by EL Status/ELP Level, Test B

Note: Newcomer 0 = EL; Newcomer 1 = RA EL; EO = monolingual English speakers




RA EL vs. Non‐RA EL vs. Non‐EL R/LA Performance (Grades 3‐8, 11) in One State

Figure 3. Non‐RA EL R/LA Performance by EL Status/ELP level (Grades 3‐8, 11)

Figure 4. RA EL R/LA Performance by EL Status/ELP Level (Grades 3‐8, 11)

Note: NonEL = monolingual English speakers

L5 Mean = 740 L5 Mean = 743

Guidelines (3 of 7)

3. Carefully consider RA EL options relative to the state’s assessment and accountability model

▪ States have discretion; flexibility on growth

▪ Consider how each RA EL option may fit (or not) with larger EL assessment and accountability, and State’s overall school accountability system

▪ Type of growth model

▪ Primary language arts assessment use




Guidelines (4 of 7)

4. Determine appropriate option(s) based on theory of action and aligned RA EL accountability model

▪ Ensure educators appropriately and consistently determine and apply option(s)

▪ Relationship of ELP level to content performance

▪ Timing of ELP assessment results, sequence of academic content assessments

▪ Option 1, 2, 3 (combo), or neither

▪ Primary language arts assessment

Guidelines (5 of 7)

5. Validate assessment use in the state’s chosen RA EL accountability model

▪ Provide sufficient validity evidence to support inferences

▪ Several validity‐related questions and types of evidence

▪ Initial ELP screener validated to distinguish ELP levels used to determine options




Guidelines (6 of 7)

6. Establish procedures for implementing the state’s chosen RA EL accountability model

▪ Document and communicate purpose, goals, development process

▪ Set policies and procedures for transition to model

▪ Set rules and procedures, including business rules for unusual cases

▪ Monitor school and district implementation

Maryland State Profile

▪ Assessments: PARCC for ELA and math, ACCESS for ELLs 2.0 for ELP ▪ Accommodations: PARCC accommodations include extended time, word‐to‐word bilingual dictionary, mathematics response speech‐to‐text or human scribe, Spanish math, and general administration read aloud and/or clarified in native language.

▪ Annual Measurable Objectives: The School Progress data identifies if all students as well as special service and racial student groups met or did not meet the achievement, attendance, and graduation rate AMOs.

▪ Tracking recently arrived students: Maryland collects a snapshot of the Date First Enrolled in US School for active ELs on the Pretest File in November.

▪ Reports: Maryland can generate reports based on summative ELP level, date first enrolled in U.S. school, length of time in English Language Development program, literacy in EL’s home language, and interrupted education.




Maryland’s Theory of Action Considerations

▪N size

▪Option 1 or 2▪Option 1 with weighted % proficiency model for recently‐arrived ELs

▪Reclassified (Released) ELs

Oregon State Profile

• State ELP assessment: Oregon uses ELPA21 as the state ELP assessment and SBAC for ELA/Math

• State exit criteria: Proficiency on ELPA21 is the main criterion. Additional state‐approved evidence may also be used in specified conditions.

• State exit procedure: Students exited if they score proficient on ELPA21. Students who do not score proficient on all language domains of ELPA21 may be exited using additional state‐approved criteria aligned to relevant domains.

• Other background information: Oregon intends to use two measures for the ELP indicator: (1) on track to ELP and (2) ELP growth. The rationale behind using multiple measures is to support a comprehensive view of progress to English language proficiency via the use of two distinct but complementary measures. Note that Oregon will only establish a baseline measure, MIPs, and a long‐term goal for on track to ELP.

• Accommodations: Oregon’s Accessibility Manual provides guidance on all accessibility supports for all of Oregon’s state assessments




Oregon’s Theory of Action Consideration

▪ Minimum n: Oregon intends to use a minimum n‐size of 20 ELs in the current school year or 20 ELs across three consecutive years (i.e., aggregating across years). The latter represents Oregon’s attempt to include as many schools as possible in the accountability system.

▪ Option 1 or Option 2 Recently arrived English learners:

▪ Exclusion from 1st administration of ELA assessment (this is business as usual for Oregon).

▪ Current/monitored English learners:

▪ Student group includes current English learners as well as former English learners monitored for four years (as opposed to two years).

▪ Considering the inclusion of the English language proficiency as a prior score in the ELA and math growth models.

Guidelines (7 of 7)

7. Establish evaluation criteria and process for the state’s chosen RA EL accountability model

▪ Focus on definitions, theory of action, impact

▪ Does model meaningfully differentiate schools?

▪ How does State limit gaming and ensure inclusive procedures?

▪ Is adopted RAEL model consistent with larger system?

▪ Are policies in place for transitioning to model?

▪ Plan to monitor consequences, incentive structures?




Empirical Illustration of RA EL Accountability Models

▪ Provide an example application of ESSA’s RA EL accountability options to a state’s ELP assessment dataset

▪ Empirically examine and compare results from different RA EL model options

▪ Identify trends and issues learned from the analyses (Cook, Sahakyan, & Linquanti, 2017)

RA EL Accountability Models – Option 1

▪ Approach: Exclude students in year #1

▪ Theory of Action: Provides a more meaningful reflection of RA EL student performance. Calculating without growth or any proficient adjustments will motivate schools to support better RA EL outcomes.

▪ Model




RA EL Accountability Models – Option 2

▪ Approach: Include students in year #1; use growth in year #2

▪ Theory of Action: R/LA proficiency per se is not a meaningful reflection of RA EL student performance, growth in R/LA is a better indicator of RA EL performance.

▪ Model

∑

Option 2 Models

▪ Value table (a): break R/LA proficiency levels into multiple categories. Students making progress receive a “1” otherwise they receive a “0.”

▪ Percentile growth (b): ranks R/LA growth scores into percentiles. RA ELs growing at or above the 40th percentile are considered to make growth and are assigned a “1” otherwise they receive a “0.”

▪ Residual gains model (c): post‐on‐pre (R/LA score) regression analysis. If RA ELs make better‐than‐predicted growth they receive a “1” otherwise they receive a “0.”




RA EL Accountability Models – Option 3▪ Approach: Incorporate options 1 and 2 based on a RA EL’s initial ELP level.

▪ Theory of Action: RA EL’s initial ELP level affects R/LA performance, and effects are different for those at different ELP levels. Accounting for this difference better RA ELs’ performance.

▪ Model

∑

30

WIDA Consortium: 36 States, 3 territories have adopted WIDA Standards and use ACCESS for ELLs

Data Used for Analyses




ACCESS for ELLs 2.0

ENTERING BEGINNING DEVELOPING EXPANDING BRIDGING

54321REACHING

6

Annually administered ELP Test aligned to WIDA

Standards

WIDA State A

RA EL = An EL student with the first observed ELP score in the 2014-2015 school year




Characteristics of ELs

Non SIFE & SIFE ELs




ELA Proficiency by RA EL and SIFE Status

35

Results from Analyses – All Students




Results from Analyses – EL Subgroup

Findings

▪When analyzed using the all students group, very little difference in school outcomes between model options

▪When analyzed using the EL subgroup, observable differences but generally small between model options

▪ Based on above, a state’s theory of action is critical to model option selection

▪ CAVEAT‐This is only one state. Results will likely vary…a lot.

6/27/2017 38

strengthening assessment accountability for arrived english learners essa · 2017-06-28 · english...

Documents