1

Assessing Student Understanding

David Niemi

UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing (CRESST)

CRESST ConferenceLos Angeles, CASeptember 9, 2005

2

The mile wide, inch deep curriculum

How wide and deep should it be?

1 inch wide and 1 mile deep?

1/2 mile wide and 2 inches deep?

3

Example of Instruction

eπi + 1 = 0

4

Assessment

eπi + 1 =

5

Answer

eπi + 1 = 0

6

What would you have to know in order for this to be meaningful

and useful?

eπi + 1 = 0What do the symbols mean? What is this equation about? What can you do with it?What’s important about it?How does it connect to other topics in mathematics?

7

Our approach

Analyze and map the domain based on previous research and knowledge elicitation

Identify central principles

Used to build schemas

Enable inferences, complex problem solving, and learning

Provide a foundation for advanced learning

Use models to build assessments at right level of cognitive demand

8

Research on the Structure of Knowledge

--Chi, Glaser, & Rees, 1983

9

Eliciting the Structure of Knowledge

• Experts (scientists, mathematicians, historians, writers) identify organizing concepts and principles (“big ideas”)

• Determine related ideas and skills: facts, problems, situations, etc.

• Map all ideas and skills

10

Structure of Algebra I Knowledge:Central Principles

1 Number

2 Expressions, Equations, Inequalities

3 Functions

4 Problem Solving

5 Reasoning

6 Sets

11

Ideas About Functions

A function is a mapping between inputs and outputs such that each input is mapped to one and only one output.

Many events in the physical world can be modeled as functions.

Many functions can be represented by algebraic equations or graphs.

The equation of a linear function can be written in the form f(x) = mx + b.

The graph of a linear function is a line.

12

Map of Algebra I Knowledge

13

Task Components for Assessing Complex Learning

• Text or other representation of information requiring domain knowledge (concepts, principles, and factual knowledge)

• Respond with complex performance • E.g., explanation, problem solving, knowledge map

• Scoring based on expert performance

14

Examples of Models in Action Assessments of history and math understanding tested with hundreds of students

Scaling up with 300-400,000 students per year in the district’s 2nd largest district

IERI: online tools to build assessments (Assessment Design and Delivery System or ADDS)

New CRESST: assessments of mathematics understanding in grades 6-8, including worked examples

15

Study of Fifth Grade Students’ Understanding of Fraction

Representations 500 students randomly assigned to instruction on:

• Fractions as parts of wholes

• Fractions as rational numbers

After instruction, students given

Students performed better with the types of representations and meanings they had been taught.

Students who understood the representations performed better on measures of complex problem solving.

16

Fractions

17

Fractions

18

Equivalent Fractions

19

Equivalent Fractions

20

Fractions

21

ADDS Designer:Select Information Source

22

Screenshot from an Animation

23

Two Screenshots From an Animation

1 2

24

Screenshot from a Simulation

25

Results from Studies of Teachers Using ADDS

Series of experimental studies

In one study, 33 middle school science teachers randomly assigned to ADDS and non-ADDS groups

ADDS teachers were more likely to focus on central ideas and developed more cognitively complex assessments.

In another experimental study with 17 teachers, ADDS enhanced the ability of teachers to develop comparable assessments on the same topic.

26What have we learned (in the effort to “make

evidence based practice a reality”)?

Complex, research-based assessments:

Can be instructionally sensitive, valid measures of challenging state standards, predictive of performance on state tests

Can provide info to guide and improve instruction,

Can be reliably scored.

27

What have we learned?

Deep understanding is rare or non-existent, but understanding and use of big ideas can be improved in a relatively short time.

Information on student understanding and complex performance is a powerful tool for change and can be used to help focus and propel capacity building efforts.