1

Today’s Topics

Research-based assessment to improve learning: models, templates, and tools

Definitions

Skill Base

2

Strategy ChoicesTop down conceptual design

Bottom up empirical design

Present practice: the top down linkage is weak mediated exclusively by standards and test specs

Decisions re tests and improvement driven by empirical findings against a weak conceptual base

CRESST improvement strengthen and operationalize the conceptual base so that it drives assessment design and instruction

3

“Theory of Action” of Accountability

Problem 1:What is alignment? Standards-Tests

At best, only some standards are measured

Sampling is poor

No common technical approach to document relationship

Wrong metaphor (geometric congruence)

Goals are aligned with instruction and testsmeasure goals. Feedback on student results improves the instruction and learning, because teachers know what to do, have resources, and sanctions are effective

4

Theory of Action of Accountability

Problem 2: Alignment of instruction & tests

Adequate sampling of content and skill domains, spiraled

Performance standards as point of entry

Measure success of alignment by analyzing nature of teacher assignments, student performance

Analyze, review, and share lessons and marked student work that exhibit standards and promote transfer of learning to new settings

5

Theoryof Action of Accountability

Problem 3: Assessment Design and

Reporting

Multiple purposes, uses, and audiences

Limited designs and types

Traditional tests start with subject matter

A learning based assessment path

6

What Should a Coherent Assessment System Do?

Provide a conceptual link among goals, instruction, measures, and subsequent actions so to increase the likelihood of success for assessments of different purposes

Detect differences in instruction

Impact positively on instructional practice

Reflect current views of learning and sustained performance

Support fairness and morale

7

Floating Formative Evaluation

Formative assessment weakly linked to standards is a loser

Formative assessment strongly linked to single test is a short term winner but a big risk

Formative assessment, test design rules for external measures, need to be made explicit

Trade-off how many can you measure?

Answer: choose power tasks that require other skills

8

Model Based Assessment: Intellectual Capital Cognitive Families

ContentUnderstanding

ProblemSolving

Teamwork andCollaboration

MetacognitionLearning to LearnCommunication

Learning

9

From Science to Models

DOMAIN-INDEPENDENT

PRINCIPLES

CONTENT CONTENT CONTENT

CBA

TEMPLATE TEMPLATE TEMPLATE

MODEL

SCIENTIFICFINDINGS

COGNITIVE DEMANDS

Domain DependentFINDINGSSUBJECT MATTER

SPECIFIC MODELS-pedagogical and ontological

10

From Templates to Tasks To reduce error, increase

comparability, and control cost

CBA

TEMPLATE TEMPLATE TEMPLATE

TASK TASKTASK

TASKTASK

TASK

TASK

TASKTASK

TASK

TASK TASK

11

Domain-independent set of Principles: Deep Understanding of

Content

Based on demonstrated relationships among principled declarative and procedural types of knowledge

Prior knowledge is key

Ability to express critical relationships

The quality of the relationships is judged from an expert knowledge perspective

12

Domain-Independent Definition:Problem Solving

Depends upon finding the problem (if masked)

Using knowledge to identify critical barriers and ways around them

Selecting procedures to follow, recognizing impasses, and adjusting plan

Has knowledge, metacognitive, motivational, analytic, and feedback components

13

Model-Based Assessment Rubric Deep Understanding of Content

(Domain Independent)

Principles or themes (big ideas)

Key prior knowledge

Explicit relationships

Avoid misconceptions

Expert performance-based scoring

14

Template Ingredients (Specifications)

Task(s)

Format(s)

Prompt(s) and requirements

Expert Based Scoring

Directions

Sample

15

Example of Template for Deep Understanding of Content

Present primary source materials in each domain, usually two points of view

Student required to integrate prior knowledge and principles

Scored by using performance by subject matter experts

16

Content UnderstandingTemplate #1 Explanation

An array of primary source materials (2 sides)

A prompt that asks for an explanation in context

Constructed (written) answer

Evaluated by means of a scoring rubric that embodies key elements of learning model

17

Content Understanding 7th Grade Hawaiian History Writing

Assignment—BayonetConstitution

Be sure to show the relationships among your ideas and facts.

Your essay should be based on two major sources:

1. The general concepts and specific facts you know about Hawaiian history, and especially what you know about the period of the Bayonet Constitution.

2. What you have learned from the readings yesterday.

Imagine you are in a class that has been studying Hawaiian history. One ofyour friends, who is a new student in the class, has missed all the classes.Recently, your class began studying the Bayonet Constitution. Your friend isvery interested in this topic and asks you to explain everything that you havelearned about it.

Write an essay explaining the most important ideas you want your friend tounderstand. Include what you have already learned in class about Hawaiianhistory, and what you have learned from the texts you have just read. Whileyou write, think about what Thurston and Liliuokalani said about the BayonetConstitution, and what is shown in the other materials.

18

Excerpts from Hawaiian HistoryPrimary Source Documents

LILIUOKALANI

For many years our sovereigns had welcomed the advice of American residents who had established industries on the Islands. As they becamewealthy, their greed and their love of power increased. Although settledamong us, and drawing their wealth from resources, they were alien to usin their customs and ideas, and desired above all things to secure their own personal benefit.

Kalakaua valued the commercial and industrial prosperity of his kingdomhighly. He sought honestly to secure it for every class of people, alien ornative. Kalakaua’s highest desire was to be a true sovereign, the chiefservant of a happy, prosperous, and progressive people.

And now, without any provocation on the part of the king, having maturedtheir plans in secret, the men of foreign birth rose one day en masse, calleda public meeting, and forced the king to sign a constitution of their ownpreparation, a document which deprived [him] of all power and practically took away the franchise from the Hawaiian race.

19

Content Knowledge Prompt (Cont’d)

*From Hawaii’s Story by Hawaii’s Queen, Liliuokalani (Boston: Lee and Shepard Publishers, 1898).

It may be asked, “Why did the king give them his signature?” I answerwithout hesitation, because he had discovered traitors among his mosttrusted friends and because the conspirators were ripe for revolution, andhad taken measures to have him assassinated if he refused.

It has been known ever since that day as “The Bayonet Constitution,” and the name is well-chosen; for the cruel treatment received by the king from the military companies. [text continues]

Explain to your friend who missed class the reasons and differences for the Queen and the Senator’s approach to Hawaii’s future.

Scoring Rubric •General impression (on task)•Principles and themes•Prior knowledge•Relevant concrete examples•Avoidance of misconceptions

20

Template #2Prior Knowledge and

Explanation

Explicit measurement of knowledge domain in the explanation

Adds short-answer or selected response

Helps interprets explanation performance

Used in authoring system for teachers

21

Environmental Science

22

Genetics

23

CRESST Validity Criteria for Tests and Assessments

Fairness

Cognitive complexity

Content domain

Instructionally sensitive

Transfer and generalization

Learning-focused

Validity evidence reported for each use

Trustworthy

Cost sensitive

24

Measuring Transfer

Construct new templates or menus that vary model elements; or select credible measures of the domain:

Content complexity

Number of task elements to address, including distracters or irrelevant content

Graphical support or distraction

Need to prioritize requirements

Linguistic demands

25

Criteria for Useful Assessments in Classrooms

Detects differences in instruction

Provides information about where to focus attention rather than success/lack of success

Integrates cognitive skills and content

Includes transfer for situations and response types

Economical, transparent, and usable by teachers and students

Develops rather than constrains teacher growth

26

Measuring Transfer (Cont’d)

Vary response modes

Oral, written and other constructed response modes

Length

Response support/prompts

Individual and group performance

Degree of stringency in scoring

27

CRESST Model Based Assessment Quality Studies

Score reliability

Task and rater generalizability

Stability of student performance over time

Relationships among measures

Instructional sensitivity

Opportunity to Learn (OTL)

Effect of school composition on performance

Cut-score modeling

28

Evidence for Model-Based Assessment

Across topics, content areas, and age ranges (preschool to adult)

Reliable scores

Teachable

Impact long-range outcomes (HS exit exam)

Cost low, quality maintained

Reusable elements, automation

To allow aggregation of assessments from classrooms for formal accountability

Measuring Transfer

Construct new templates or menus that vary model elements:

Content complexity

Number of task elements to address, including distracters or irrelevant content

Graphical support or distraction

Need to prioritize requirements

Linguistic demands

29

Continuing R&D Areas

New contexts

Trade-offs (limited number of templates vs. wide range of formats)

Performance over time

Scalability in the long run

Authoring systems to support teacher-developed assessments linked to large-scale assessment