university of colorado at boulder lorrie a. shepard school of education university of colorado at...

University of Colorado at Boulder

Lorrie A. ShepardSchool of Education


Conceptual Coherence between Large-Scale and Classroom Assessments

CRESST ConferenceUCLA

September 9, 2004


Contrasts between large-scale and classroom level assessments that imply that different assessments are needed

• standardized vs. dynamic

• uniform date vs. variable dates

• independent performance vs. assisted performance

• delayed vs. immediate feedback

• stringent requirements for technical accuracy vs. less stringent requirements

The most important shared characteristics of the two levels of assessment should be alignment with the same curriculum standards and a shared model of learning.


For an assessment system to support learning, it must have coherence. The conceptual base or models of student learning underlying both external and classroom assessments must be compatible.

Knowing What Students Know

Pellegrino, Chudowsky, & Glaser NRC, 2001


Committee on Assessment in Support of Instruction and Learning A joint committee of Board on Testing and Assessment Committee on Science Education K-12 Mathematical Sciences Education Board

Bridging the Gap criteria that characterize an ideal system, distilled from:

NRC Science Education Standards NCTM Assessment Standards Commission on Instructionally Supportive Assessment


Shared Characteristics of Large-Scale and Classroom Assessments

A shared model of student learning including developmental progressions over time

A shared conception of disciplinary knowledge and competence Focus on evaluating understanding and reasoning Offer a clear vision of what constitutes mastery Target both general and domain-specific forms of cognition Select complex, context-based tasks that are open to multiple approaches


This idea of alignment between large-scale and classroom assessments refers

Not to the limited alignment where test publishers show fit within test blueprints,

But to the more complete and substantive alignment that occurs when the tasks, problems, and projects in which students are engaged represent the range and depth of what we say we want students to understand and be able to do.

Perhaps a better word would be embodiment.

Shepard, 2003, NSTA


A Sampler of Science Assessment© California Department of Education

Part B


A Sampler of Science Assessment© California Department of Education


Note that good assessment tasksare interchangeable with good instructional tasks.

(The exact same task should not be used for both purposes, however.)

We also have evidence that “teaching to”

problem types like these improves learning.


Shepard, Flexer, Hiebert, Marion, Mayfield, & Weston (1996).


Effect Size: AP – PaceSetter Students = .64 on SAT I, 1.58 on AP Calculus, and .21 on Open-Ended Tasks


Purposes Served by Large-Scale Assessment • Data for monitoring and accountability

• Exemplification of learning goals

• Program “Diagnosis”

• Professional Development


Program Diagnosisnotes from Delaware’s Rachel Wood and Julie Schmidt

…an 8th-grade weather assessment revealed that students across the state had over-generalized their knowledge of the movement of all air masses as having to go from west to east.

…lead teachers were stunned that their own students could not answer a question that involved drawing conclusions from a simple graph.

A commentary around a released item on Punnett squares attempts to turn teachers’ attention to thinking about why students have acquired only a mechanical sense.


Professional Development

Assessment-focused professional development deepens understanding of standards and directs attention to what students are getting from new activities.

Given time burdens, teachers can learn formative assessment strategies while working to raise achievement on accountability tests.

Assessment efforts only make sense if they are intimately tied to content learning.


A developmental framework and progress variables provide a substantive basis for monitoring student progress over time.

“Large-scale assessment tools and supporting instructional materials should be developed so that clear learning goals and landmark performances along the way to competence are shared…”


Towards Coherence Between Classroom Assessment and AccountabilityNSSE Yearbook 2004M. Wilson, Editor

Forster & Masters (Australia)

• Closely articulated, classroom-level developmental assessments

• National survey assessments

• Both tied to progress maps and accompanying curriculum materials

• Progress maps are broad, general descriptions of how proficiency typically develops in a curricular area.

• Accountability data collection embedded in classrooms with teachers as judges


NSSE Yearbook 2004

Wilson & Draney (BEAR Assessment System)

BEAR follows a developmental perspective, is built around a set of progress variables that reflect

the way that increasing proficiency develops.

(In contrast to cross-sectional and smorgasbord type assessments typical in the U.S.)

Coherence between classroom and external tests requires a shared understanding of the construct at the level of the progress dimension and the level of specific assessment tasks and scoring guides.


Note that KWSK said that progress variables provide a substantive basis for monitoring student progress.

Developmental continua should not rely on either statistically derived trajectories or aspirational benchmarks. Instead progress maps should represent the pathways of real students’ learning gains under conditions of reasonably adequate instruction.

Caution in Developing Growth Trajectories


Queensland

Shavelson

Queensland replaced A levels with formative assessments to meet students’ needs and to serve

as a basis for summative information beyond the classroom.

Queensland’s system is intended to audit local implementation of curriculum, assessment, and accountability.

A process for moderation of scores ensures consistency across classes and schools.

Australian examples are quite different from U.S. context because of the feasibility of a shared curriculum and trust in teachers’ judgments.


Delaware: Comprehensive Science Assessment

Rachel Wood

Curriculum materials developed to support state standards.

Prerequisites for meeting curricular objectives were identified grade-by-grade.

Then, end-of-unit assessments were developed.

Generic rubrics were not diagnostic, replaced with specific rubrics to identify misconceptions.

State, with some outside resources, provided intensive professional development for a cadre of teachers.


Déjà Vu All Over Again Assessment reformers used the terms

“Performance,” (Resnick & Resnick, 1992), “Direct,” (Frederiksen & Collins, 1989), and “Authentic” Assessment (Wiggins, 1989)

to convey the idea that assessments must capture real learning goals. Embodying worthy goals is still a core issue. KWSK’s ideal system will require much more than token

constructed-response items on accountability tests.

university of colorado at boulder lorrie a. shepard school of education university of colorado at...

Documents