Practical Strategiesfor Assessing Student Learningin Departments and Programs:

A Utilization-Focused Approach

Jo BeldSt. Olaf College

Guiding questions

• Who are the users of the evidence?

• How might the evidence be used?

• How will you gather the evidence?

• How will you animate the evidence?

Theoretical perspectives

Utilization-focused assessment

(Patton, 2008):

Focus on intended uses

by intended users

Theoretical perspectives

Backward design

(Wiggins & McTighe, 2005):

“Beginning with the end in mind”

Theoretical perspectives

Traditional instructional design:

Choose texts

Develop classroom activities

Make up tests

Theoretical perspectives

Backward instructional design:

Identify intended learning outcomes

Determine appropriate evidence

Plan instruction and practice

Theoretical perspectives

Traditional assessment design:

Choose an assessment instrument

Gather and summarize evidence

Send a report to someone

Theoretical perspectives

Backward assessment design:

Identify intended users and uses

Define and locate the learning

Choose assessment instrument

Who are the users?

Faculty roles, commitments, and disciplinary identities offer both incentives and disincentives to engage assessment

What are the uses?

What kind of outcome do you want to investigate…

–Knowledge (“Understanding of…”)

–Proficiencies (“The ability to…”)

–Practices (“The habit of…”)

–Values or attitudes (“A concern for…”)

What are the uses?…and why?

• Affirming current practices• Tweaking the content of key courses• Extending a specific pedagogy• Enhancing “scaffolding”• Piloting innovations• Supporting grant applications• Setting future assessment agendas

How will you gather evidence?

Assessment is like any other kind of investigation; use strategies that fit the questions you are trying to answer

How will you gather evidence?

“Direct” Assessment”

“Direct” Assessment

Evidence of what students actually know, can do, or care about

“Indirect”Assessment

Evidence of learning-relatedexperiences or

perceptions

How will you gather evidence?

Common direct assessment “artifacts”

Papers, essays, abstracts

Presentations and posters

Oral or written examination items

Analytic journals

Responses to survey or interview questions that ask for examples of knowledge, practice, or value

How will you gather evidence?

Common indirect assessment “artifacts”

Course-taking patterns or transcript analysis

Responses to survey or interview questions about experiences, perceptions, self-reported progress, or impact of program experiences

Reflective journals

How will you gather evidence?

But wait!! Aren’t we observing student work all the time anyway? What’s the difference between grading and assessment?

How will you gather evidence?

Gradingsummarizes

many outcomes

for

one student

Assessmentsummarizes

one outcome

for

many students

How will you gather evidence?

The purpose of an assessment instrument is to provide systematic, summarized information about the

extent to which a group of students has realized one or more intended learning

outcomes

How will you gather evidence?

Options to consider:

Use an instrument developed by someone else

Adapt an existing instrument

Add to something you’re already doing

Connect to institutional-level evidence

Invent something new

How will you gather evidence?

Where possible, pair indirect observations of processes and perceptions with direct observations of outcomes (pp. 16-17).

How will you gather evidence?

The dual goal of sampling:

Representativenessand

Manageability

How will you gather evidence?

Examples involving comprehensive sampling:

Survey of all senior majors

Application of rubric to all research abstracts in all seminars

Application of rubric to all work submitted for senior art show

How will you gather evidence?

Examples involving selective sampling:

Application of rubric to randomly-selected subset of final papers in capstone course

Pre/post administration of locally-developed quiz in required sophomore methods course

End-of-course survey in one introductory and one senior-level course

Aggregation of results on selected items in an evaluation form for student work

How will you animate the evidence?

Evidence never speaks for itself; be intentional in summarizing, interpreting, and responding to the findings

How will you animate the evidence?

Plan intentionally for summarizing and interpreting findings:

• Delegate and distribute• Enlist help from staff and students• Use technology to save time• Balance individual and collective work• Dedicate time for discussion

How will you animate the evidence?

Plan intentionally for responding to the findings:

• Borrow strategies from past successes in collective departmental action

• Focus reporting on planned actions, not on the evidence itself

• Weight Watchers trumps The Biggest Loser

• Dedicate resources for action

A final thought….

Less really is more!