the cbal reading assessment · model-building skills • the skills needed to form an accurate...

®

Kathleen M. Sheehan,John Sabatini, Heather Nadelman

The CBAL Reading Assessment

®

Copyright © 2009 by Educational Testing Service

2

Overview

• Goals • The Groundwork

– The CBAL Reading Competency Model• CBAL Reading Summative Assessments

– current testing assumptions – an example

• CBAL Reading Formative Materials

®


3

CBAL Reading Goals

®


4

Has a positive

impact on teaching

and learning

Provides high quality evidence

about what students know

and can do

A Computer-Delivered Reading Comprehension Assessment that …

Measurement Goals

Learning Goals

®


5

Approach• Specifically address the tradeoffs

between measurement and learning goals by combining features from – Traditional Standards-based Assessments

• Standardized testing conditions• Selected response items that have

high reliability and low scoring costs

– Performance Assessments• Scenario-based projects designed to enhance motivation • Stronger links to modern theories of how students learn• Closer alignment with effective instruction • Broader construct coverage

®


6

The Groundwork:The CBAL ReadingCompetency Model

®


7

Conceptual Framework

• Developed from a review of– Cognitive literature on reading comprehension– Literature on reading strategies– State reading standards– Work with classroom teachers in ME, NJ, PA

®


8

Text Activity

Reader

Framework is Consistent with the View of Reading Proficiency Outlined

in the Rand Report (Snow, 2002)Comprehension entails three elements:

the text that is to be comprehended

the reader who is doing the

comprehending

the activity in which comprehension is a part

®


9

The CBAL Reading Competency Model

Reading Proficiency

Prerequisite Reading Skills

Model Building Skills

Applied Comprehension Skills

Pre-Reading Strategies

Model Building Strategies

Strategies for Going Beyond the Text

Required Skills

Reading Strategies

Know. of Text Conventions

Informational Text

Literary Text

Persuasive Text

(Top Two Layers)

®


10

Distinction Between Skills & Strategies

Reading Proficiency







Required Skills

Reading Strategies


Informational Text

Literary Text

Persuasive Text

Skills:

Strategies:Deliberate, conscious, effortful actions designed to repair breaks in comprehension and enhance understanding.

Automatic, internalized abilities, unconsciously applied during comprehension

®


11

A Closer Look at the Skills Branch

Reading Proficiency







Required Skills

Reading Strategies


Informational Text

Literary Text

Persuasive Text

> Learning to read

> Reading to learn

> Reading to do

®


12


• Verbal Efficiency Theory (Perfetti & Hart, 2002) – Efficient word recognition and decoding

is essential to skilled reading– Permits faster, more accurate access to word meanings– So less demand on working memory resources– More resources available for higher-level processing

• Implications for Test Design– When reading problems surface, it may be important

to determine whether the problem is traceable to a deficiency in lower-level prerequisite reading skills or higher-level comprehension skills since these two situations imply different remediation strategies.

®


13

Model-Building Skills• The skills needed to form an accurate mental

representation of the information presented in a text (“the gist”) – The Construction Integration Model (Kintsch, 1998)– The Structure Building Framework (Gernsbacher, 1990)– The Mental Model Theory (Johnson-Laird, 1983)

• Key components– Successful readers seek to create mental models

that are coherent both locally and globally.– But text is frequently incomplete (or inconsiderate)

so abilities such as integrating information from multiple parts of a text and generating accurate text-based inferences are needed.

®


14


• The skills needed to go beyond the literal and inferential meaning of text in order to achieve particular goals such as writing a report or making a decision

• 21st century skills such as – Integrating information from multiple texts– Reconciling alternative viewpoints– Evaluating the quality of information

• Is the text inaccurate, outdated, or biased?

®


15

A Closer Look at the Strategies Branch

Reading Proficiency







Required Skills

Reading Strategies


Informational Text

Literary Text

Persuasive Text

Examples:

Preview the text, make predictions

Use Graphic Organizers

Adopt a critical stance

Research Basis:

Successful readers tend to employ reading strategies more frequently than do less successful readers (Chi, et al., 1989)

Explicit training in reading strategy selection and use can lead to significant improvements in comprehension (Chi, et al., 1994)

®


TextText

Feature 4

Feature 5

Feature 6

. . .

Feature n

Feature 1

Feature 2

Feature 3

Dim. Score Calcu-lator

SR Scoring Engine

Academic

Inf. Grade Level

Vocabulary

Sent. Cmpl.

Spoken L.

Cohesion

Par. Length

Negation Lit. Grade Level

Complexity Scoring via SourceRater: Distinct Prediction Models for Informational

and Literary Texts

SourceRater is trained to reflect grade level classifications provided by professional test developers.

(Sheehan, Kostin & Futagi, 2008a, 2008b, 2009)16

®


17

CBAL Reading Summative

Assessments (PAAs): Current Testing

Assumptions

PAA = Periodic Accountability Assessment

®


18

All CBAL Reading PAAs …• are administered in two sections, each 50 minutes

– Section 1: Scenario-based tasks culminating in an extended integrated task that requires accessing multiple related texts (focused either on literary or informational/persuasive skills)

– Section 2: Short, stand-alone questions used to equate across forms (both literary and informational/persuasive )

• include a mixture of selected-response (SR) and constructed response (CR) items

• are scored via a combination of automated and human scoring

Copyright © 2008 Educational Testing Service.

18

®


What Each PAA Looks LikeLiterary PAA Inf./Persuasive PAA

Section I: ● Introductory Scenario● 20 items designed to tap

Literary Skills (SR & CR)

Section I: ● Introductory Scenario● 20 items designed to tap Informational/Persuasive Skills (SR & CR)

Section II: ● 40 short items● Designed to tap both Literary & Informational/Persuasive Skills (SR)

Section II: ● 40 short items● Designed to tap both Literary & Informational/Persuasive Skills (SR)

19

®


20

Sample Design for a Year’s Sequence of PAAs

Prerequisite Skills

Reading Strategies

Literary Skills &

Knowledge

Informational/ Persuasive

Skills & Knowledge

PAA # 1

PAA # 2

PAA = Periodic Accountability Assessment

®


21

A Sample PAA Targeted at Middle School

Readers: The E-waste Project

®


The E-Waste Scenario

In order to raise money for your annual class trip, you and other members of your class fundraising committee have decided to sponsor E-Waste Day, a one-day electronics recycling project. People in your community can bring in old cell phones, MP3 players, computers, and other electronic products, which your class will send to a recycling company in exchange for cash. Your committee will need to research the issue of e-waste in order to help the school select the best recycling company for E-Waste Day and learn as much as it can about the issue.

22

®


The E-Waste Stimulus Materials

• Simulated Web search output• A news article about e-waste • An editorial about e-waste• Advertisements about e-waste

disposal companies with student evaluations

23

®


A Sample Item: Collect evidence for the “Apply efficient search techniques” Standard

24

®


A Sample Item: Collect evidence for the “Distinguish Fact from Opinion” Standard

25

®


A Sample Item: Collect evidence for the “Compare Multiple Viewpoints” Standard

26

®


A Sample Item: Collect evidence for the “Detect Bias” Standard

27

®


28

Some Items Target Fundamental Learning Progressions

• Select performance standards that represent the desired high level of achievement in a modest number of curricular areas

• Using the Competency Model and previous research as a guide, generate a theory about how learning is likely to develop in each area; such theories are sometimes called Learning Progressions(Popham, 2008)

• Create Task Models that provide mastery evidence relative to each of the subskills and bodies of enabling knowledge specified in the progression

®


29

A Sample Learning Progression

The target curricular

goal

The sequence of subskills

and knowledge that, according to the theory,

students must master en route

to mastering the targeted

curricular goal

(Popham, 2008)

Provide a Task Model for each Level

Target Curricular Aim:

Use an Understanding of Text Structure to Enhance Comprehension of

Informational Text

Level 3:

Infer categories that span two or more texts

Level 3:

Infer categories that span two or more texts

Level 1:

Group details into appropriate categories

Level 1:

Group details into appropriate categories

Level 2:

Infer appropriate categories from details

Level 2:

Infer appropriate categories from details

Starting Point:

Mastery of Critical Prerequisite Skills

®


This Item also Targets a Key Reading Strategy: Use Graphical Organizers to Organize/Chunk Information

A Sample Level 2 Item

Infer Categories from details

Prop. Correct = 0.45

30

®


A Sample Level 1 Item:

Group Details into appropriate categories

Prop. Correct = 0.69

31

®


32

Percent of Examinees at Each Observed “Text Structure” Pattern

00 10 11 01

010

2030

4050

60

Observed Pattern

Per

cent

25

30

39

6

94% of the observed response vectors are consistent with the

hypothesized progression.

®


33

CBAL Formative Reading Materials: Component Tasks

(Evaluating Arguments)

®


34

Formative Instructional

Sequence

®


35

®


36

®


37

Formative Instructional Sequence

®


38

®


39

®


Results: Measurement Goals• Administered multiple prototype assessments• Individual assessments demonstrated our ability to

– Maintain standardized testing conditions for all students– Target a broad array of skills

• not just Model-Building skills, also Applied Comprehension skills– Obtain total test scores with acceptable levels of reliability– Develop automated scoring models

for a wide array of innovative new item types • lower scoring costs• more timely delivery of assessment results

40

®


Results: Learning Goals• Developed and administered extended,

scenario-based tasks designed to– be more like reading in non-testing contexts– promote retention and transfer

• Developed hypothesized Learning Progressions intended to– reduce reliance on unhelpful “test prep” strategies– encourage use of empirically-validated reading strategies– help teachers conceptualize the pathways along which students

are expected to progress (Heritage, 2008)– provide a starting point for subsequent

classroom-based formative assessments

• Planning for needed validity studies is underway41

the cbal reading assessment · model-building skills • the skills needed to form an accurate...

Documents