what item level data tell us about universal design:

National Center on Educational

Outcomes

What Item Level Data Tell Us About Universal Design:

Fantasy, Foolishness, or

Fuel for Fire? in Large-Scale

Assessments


Outcomes

Presenters:Martha Thurlow, NCEO

Liru Zhang, Department of Education, Delaware

Karen Barton, Research Triangle Institute, International

Brent Garrett, Alliance for Systems Change, MSRRC


Outcomes

Title I Regulations introduced the need for universally designed assessments –

[Assessments must be] designed to be accessible and valid with respect to the widest possible range of students, including students with disabilities and students with limited English proficiency.

Sec. 200.2(b)(2)


Outcomes

Proposed Reauthorization of IDEA includes universal design concepts as well –

UNIVERSAL DESIGN- The State educational agency (or, in the case of a districtwide assessment, the local educational agency) shall, to the extent possible, use universal design principles in developing and administering any assessments….

Sec 612, Senate Bill 1248


Outcomes

A Quick Definition

Universally designed assessments are built from the beginning to be accessible and valid for the widest range of students.


Outcomes

Five categories for consideration of item bias grew out of an international effort (SENDD subgroup of OECD):1.Bias due to item layout and design

2.Bias due to differential acculturation/lack of experience directly related to disability

3.Bias due to unnecessary or extraneous complexity of item language or length, including bias due to memory load and bias due to verbal load


Outcomes

Five categories for consideration of item bias grew out of an international effort (SENDD subgroup of OECD):4.Bias due to presentation or

response design that differentially affects users of assistive technology or alternate formats

5.Bias due to use of manipulatives that differentially affect students, irrelevant to the construct being measured


Outcomes

Another effort involved reviewing and piecing together information from other fields:

Vision

Ergonomics

Graphic Design

Architecture


Outcomes

From this review, NCEO identified a set of elements of universally designed assessments• They relate to different points in

the item development process – some of these are the responsibility of item writers, some are the responsibility of test contractors; some are the responsibility of both

• Some have more research behind them than others – so far


Outcomes

Elements of Universally Designed

Assessments Inclusive assessment

population

Precisely defined constructs

Items developed and reviewed for bias and accessibility

Amenable to accommodations


Outcomes

Elements of Universally Designed

Assessments

Simple, clear, and intuitive instructions and procedures

Maximum readability/ comprehensibility

Maximum legibility: text, graphs, tables, illustrations, and response formats


Outcomes

“Four holds on one of the rock climbing walls are labeled on the diagram below. Matthew first climbs vertically 10 feet from Hold A to Hold B, horizontally 25 feet from Hold to Hold C, and then vertically 15 feet from Hold C to Hold D. How many fewer feet would Matthew have climbed if he had climbed directly from Hold A to Hold D?”

Is the use of “hold” as a noun familiar to students?Is the concept of a “rock climbing wall” familiar to most students?Will students be distracted by the odd shapes on the diagram?


Outcomes

Amenable to Accommodations

Could this item be presented in an alternate format? Braille?Is the high number of items on the map and long list of cities necessary to respond to this item?

“According to this weather page, which place is the warmest on December 28?”

If you were flying to Chicago the day this weather page was printed, what information could you learn for your trip from this page?


Outcomes

Legible graphs, tables, illustrationsWhat is that big black rectangle?


Outcomes

Is the border distracting?


Outcomes

Element #5: Simple, clear, and intuitive instructions and procedures

Are the swimmers at the bottom of the page distracting?


Outcomes

Abedi research suggested that linguistic complexity of test items was a significant source of measurement error for ELL students (and students with disabilities)Examples of Linguistic Modifications:

Census Video game

Familiarity or frequency of non-math vocabulary (unfamiliar or infrequent words changed):

Last year’s class vice president The vice president

Length of nominals (long nominals shortened):


Outcomes

Many of the Universal Design elements directly address the characteristics of test items.

Is there a way to tell which items may be a problem without looking at all of the items?

We have procedures that we use to identify potential bias for other subgroups – will these work for students with disabilities?


Outcomes

Universal Design Project at NCEO has suggested three ways to flag items:• Statistical analysis of test

results (or field test results)

• Expert review using universal design considerations

• Think aloud (cognitive labs) with students


Outcomes

Focus Today – Statistical Analysis of Test or Field Test ResultsDo these approaches work

for identifying problematic items? Pros and cons of approaches?

Should we instead do as one TAC said – just look at all items?

what item level data tell us about universal design:

Documents