c r e s s t / u c l a eva l. baker international congress for school effectiveness and improvement...

C R E S S T / U C L A

Eva L. Baker

International Congress for School Effectiveness and ImprovementJanuary 2003

Sydney, Australia

UCLA Graduate School of Education & Information StudiesCenter for the Study of Evaluation

National Center for Research on Evaluation, Standards, and Student Testing

Technology-Based Assessment for

High-Performance Learning


Premises and Goals

Assessment is central to the effectiveness of classroom and distance learning and accountability—some basics

What are advances in assessment? How does technology fit?

How can technology weave instructional and external testing into a coherent system?

Can assessment be cost-sensitive and valid?

Many examples throughout


Order of Topics

Basics about assessment

CRESST research-based models

Attributes, benefits

Examples

Template 1: Paper and pencil

Template 2: Computer

Template 3: Authoring

Technology benefits and requirements


About Assessment Role of external tests—accountability, evaluation,

and system monitoring

Instructional uses of tests in classrooms and schools: diagnosis, modeling, formative assessment as a core teaching strategy

Variations in curriculum require strands supporting coherence among goals (standards), content, cognitive demands (what thinking skills are required?)

Coherence perceived from student, teacher, administrator, and expert views


External Instructional Exams Assessments

Motivated performance

Time-sensitive

Standardized

Shallow sampling

Stand-alone

Embedded in learning

Adapted to learners

Extended time

Opportunity to revise

Contextualized results


External and Instructional Assessments Must Be More

Coherent

Horizontally and vertically

Conceptual and psychological linkage

For accountability systems, classroom measures may be good supplements


CRESST Assessment Models

Do not start with content

Focus on aspects of learning and assessment that transfer from content to content area

Multi purpose: both formative learning and outcomes

Emphasize student-constructed answers

Expert performance defines scoring

Research substantiates these models across different subjects and people


Families of Cognitive Demands:

Model-Based Assessment

ContentUnderstanding

ProblemSolving

Teamwork andCollaboration

MetacognitionCommunication

Learning


CRESST Assessment Models

Research-based, and sites are exclusively classrooms, schools, and systems

Focus on cognition and learning

Combine content-independent and content-dependent knowledge and strategies

Reusable and cost-sensitive


Examples in Knowledge Understanding and Problem

Solving

Paper-pencil templates

Technology-based administration, scoring, and reporting

Technology-supported authoring templates and menus

For teachers, curriculum experts, test makers


Assessment of Understanding

Deep understanding of primary source materials or key processes

Standard reading as part of task

Standard directions

Standard scoring rubrics based on experts’ performance


Hawaiian History Assessment Task:

Bayonet Constitution

Be sure to show the relationships among your ideas and facts.

Your essay should be based on two major sources:

1. The general concepts and specific facts you know about Hawaiian history, and especially what you know about the period of the Bayonet Constitution.

2. What you have learned from the readings yesterday.

Imagine you are in a class that has been studying Hawaiian history. One ofyour friends, who is a new student in the class, has missed all the classes.Recently, your class began studying the Bayonet Constitution. Your friend isvery interested in this topic and asks you to explain everything that you havelearned about it.

Write an essay explaining the most important ideas you want your friend tounderstand. Include what you have already learned in class about Hawaiianhistory, and what you have learned from the texts you have just read. Whileyou write, think about what Thurston and Liliuokalani said about the BayonetConstitution, and what is shown in the other materials.


Excerpts from Hawaiian HistoryPrimary Source Documents

LILIUOKALANI For many years our sovereigns had welcomed the advice of American residents who had established industries on the Islands. As they becamewealthy, their greed and their love of power increased. Although settledamong us, and drawing their wealth from resources, they were alien to usin their customs and ideas, and desired above all things to secure their own personal benefit.

Kalakaua valued the commercial and industrial prosperity of his kingdomhighly. He sought honestly to secure it for every class of people, alien ornative. Kalakaua’s highest desire was to be a true sovereign, the chiefservant of a happy, prosperous, and progressive people.

And now, without any provocation on the part of the king, having maturedtheir plans in secret, the men of foreign birth rose one day en masse, calleda public meeting, and forced the king to sign a constitution of their ownpreparation, a document which deprived [him] of all power and practically took away the franchise from the Hawaiian race.


History ExplanationScoring Rubric

1. General impression of content quality2. Principles or concepts (DD)3. Prior knowledge (DD)4. Use of available resources (DD)5. Misconceptions (DD)6. Argument (DD?)7. English mechanics (DI)


Mathematics Explanation

Task (4th or 5th Grade) Imagine a person from a television station has asked you to give a demonstration on TV. You will be on a show to help other students learn about maths. You are asked to explain everything 10-year-old students should know about fractions.

Below are some questions you should try to answer. These are questions that students in the TV audience will ask you.

For each question you should draw as many pictures as you can to show what you mean. Then write down what you would say about your pictures on TV. Use as many words and pictures as you need.

What is a fraction? Why are there two numbers in a fraction? How many fractions are there between 0 and 1? How many fractions are equal to 1/2? What other important ideas should students know about fractions? Show how you would explain these ideas. Use as many pictures and words as you need.


Scoring for Maths Task

Principles

Prior knowledge

Resources

Misconceptions

Argument/explanation


Technology for What?

Most available technology-based assessment promotes efficiency rather than expands the boundaries for measurement of learning

Tendency to limit attention to data processing requirements

We should use technology to extend our understanding of student accomplishment and program and school quality


Template 2: Knowledge Representation to Assess Content Knowledge and

Problem Solving

Same tasks for content knowledge

Responses not essay but representation of relationships, hierarchies

Scored by expert maps

Efficient but expands breadth and depth of knowledge measurement


History: U.S. Depression (CK)


Genetics: High Performance (CK)


Bicycle Pump—High Performance (PS)


Integrating Knowledge Map with Web Search Strategy


MAIN Story: Integrating Formative and External Assessments

Authoring systems (computer-supported guidance) to help teachers create and share various sorts of assessments intended to measure goals included, not covered by, or that need deeper attention than given in external measures


Strategy to Link External and Formative Assessments

Cognitive demands (domain independent) and content knowledge and strategies (domain dependent)

Authoring system allows teachers to assess goals not externally measured, or to connect their assessments to external measures but in a more contextualized setting


How It Works

Teacher will get guidance, using research-based templates

What is the purpose of the test? What is to be measured? What content? What conditions? What intellectual skills?

Scoring rubrics (based on expert performance) will be provided but can be edited

Graphical representation, simulations, and essays are current options


Assessment Authoring Benefits

Common floor on assessments created by teachers

Systems start with easy, fixed formats related to learning demands, and as teacher sophistication develops, move to more choices (mix and match)

Will allow summaries of student work bubbling up from teachers’ formative assessments to validate the external scores

Supports collaboration across different teachers


Authoring Systems Issues

Scored work made public (within the school, with privacy provisions, and among schools)

Success depends upon teacher subject matter knowledge, access to needed information, and sharing

Success may depend on the realistic link to external examinations

Generation of paper- or computer-based tasks


Minimum Requirements

Infrastructure

Capacity—subject matter

Orientation to learning and to results

Congruence with external mandates

Availability of smart tools

Lead to a culture shift


Distance Learning Applications

High-quality performance demanded

No bottleneck in scoring

Basis of comparing courses

Generates online assessments with instant scoring, feedback to student and instructor

Aggregates student work


Window on the Development of Problem-Solving Template 2—

Author Screen Assessment Purpose(s)

Diagnostic, readiness monitoring, certification

Scenario Context, constraints, situation

Problem Characteristics Fix, change usual sequence, improvise step(s), combination

Problem Identification Menu Stated, embedded, multiply masked, barriers, inconsistent

data from multiple sources, time bound, partial identification, prior knowledge requirements


Author Screen Template 2 (Cont’d)

Macro Planning Menu Explicit courses of action, problem subdivision, backup

strategies, help seeking, mix of domain-independent and domain-dependent cognitive strategies

Trial and Feedback Menu Data capture of process, process feedback, help,

iteration

Solution(s) Convergent (right answer), multiple acceptable, partially

acceptable, divergent—with scoring criteria, sequential, mixed


Sample Examinee Screen Components

Scenario

Problem

Information acquisition

Macro strategy

Micro strategies (domain specific)

Solution trials

Report of performance


Sample Examiner Information

Time spent on problem

Trials to criterion

Help access

Solution paths

Generalizability of solution

Acceptability of solution

Likely explanation for errors (e.g., lack of prior knowledge)

Metric (standards for performance)


Summary

Technology-based assessment needs to extend what we can do

Authoring systems can help teachers design better, more sensitive tests and projects

Technology can help us share findings

Technology-based assessment requires the same evidence of technical quality

Demand evidence, not business claims, before you buy

c r e s s t / u c l a eva l. baker international congress for school effectiveness and improvement...

Documents

c r e s s t u c

formative assessment

goals assessment

costsensitive slide

expert views slide

formative learning

content focus

distance learning