ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

6 February 2003

ONR/NSF Technology Assessment of Web-Based Learning

Eva L. Baker

UCLA/CRESST

Harry O’Neil

USC/CRESST

Assessment of Distance

Learning

2ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

Why Are Some Organizations Successful in Using Knowledge?

• Focus on learning

• Constant use of appropriate information (formal and informal)

• Focus on feedback and change

• Public display and exchange

• Community pride in outcomes of learners

• Select knowledge to foster these ends

3ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

Assessment in Support of Evaluation

Distance Learning

IntellectualCapital

SocialCapital

+

LearningOrganization

4ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

Goals for CRESST Model-Based Assessment (MBA)

• Assessment components share a common framework. Model-based assessment starts with intellectual capital and applies it to content domains to support– Coherent, sustained learning

– Spiral design—common language

– Transfer (application to new situations)

– Multipurpose

5ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

CRESST Model-Based Assessment (MBA) Features

• Research-based

• Focus on cognition and learning

• Abstracted in models based on key learning elements—principles guiding test design and instruction

• Operationalized in templates

• Reusable and cost-sensitive design/training/scoring

6ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

Model-Based Assessment Families of Cognitive Demands

ContentUnderstanding

ProblemSolving

Teamwork andCollaboration

MetacognitionCommunication

Learning

7ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

Expert Model—Deep Understanding of Content

(Domain Independent)

• Came from syntheses of research on learning

• Principles or themes (big ideas)

• Use of prior knowledge

• Explicit relationships

• Avoiding misconceptions

• Expert performance-based scoring

8ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

Model-Based Assessment Design

• Models to templates (to specification) to tests

• Template contains domain-independent (transfer) and domain-specific (strategy and knowledge) components

• Templates that allow common domain-specific design approaches to be used, e.g., primary sources in history

• Scoring requirements

9ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

Template

• Task(s)

• Format(s)

• Prompt(s) and requirements

• Scoring

• Directions

• Sample

10ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

One Template for the Model of Deep Content Understanding

• Graphical representation of relationships

11ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

TemplateKnowledge Representation

• Prompts

• Key aspects of ideas, supporting facts and views and their relationships

• Relationship is explicit

• Organizational options– Core and peripheral

– Hierarchical

– Cause-and-effect

– Chronological

• Expert scoring

12ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

Genetics

13ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

Bicycle Pump

14ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

Economics

15ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

Brief History of MBA in LAUSD

• Content understanding and problem-solving models

• Explanation templates

• 4 subjects, 3 grade levels, 2 languages– Math, science, history, literacy

• Purposes: (1) to clarify expectations; (2) to provided instructionally embedded assessment; (3) to get a measure of performance

• CRESST-managed instructor involvement

16ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

CRESST Validation Studies

• Score reliability

• Task and rater generalizability

• Stability of student performance over time

• Relationships among measures

• Instructional sensitivity

• Opportunity to Learn (OTL)

• Cut-score modeling

17ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

Model-Based Assessment ofSocial Capital

Trust

EfficacyNetworks

EffortTransparency

LearningOrganization

18ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

Context for Success ofKnowledge-Based Reform

• Measures of intellectual and social capital

• Psychometrics of team-based simulation

• Infrastructure and stability

• Capacity to investigate—6 Sigma organizational

• Learning

• Congruence or peace with external mandates

19ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

Summary of Assessment Knowledge Requirements

• Knowing why

• Knowing what to assess: intellectual and social capital components

• Knowing how: transfer (application to other topics and situations)

• Applying MBA to distance learning

20ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California

CRESST Web Site

http://www.cse.ucla.edu