innovation in action: a simpler path for meaningful assessment … · 2018. 1. 22. · a simpler...

assessment ∙ accreditation

e-portfolios

Innovation in Action: A Simpler Path for Meaningful Assessment Data

Ben Coulter, Ed.D.Senior Director, Campus Solutions Taskstream

Outcomes

• Understand the goals and scope of the Multi-State Collaborative (MSC) to Advance Learning Outcomes Assessment

• Understand the benefits and experiences for participants

• Appreciate how simple technology can reduce barriers to doing direct assessment of student learning outcomes (SLO’s)

• Consider how the technology could be used on your campuses to engage faculty and start or improve assessment initiatives

• Founded in 2000

• 500+ client institutions

• Proven, reliable technology

• Experienced, caring team

• Dedicated to client success

Taskstream advances effective assessment to improve student learning and institutional quality.

AMS

Assessment Planning & Documentation

LAT

ePortfolio-Based Assessment

Aqua

Streamlined Direct Assessment

The Multi-State Collaborative to

Advance Learning Outcomes

Assessment

MSCGates-funded project to create a scalable model to assess Essential Learning Outcomes based on authentic student work using VALUE rubrics

VALUE Rubrics

• ACC&U initiative in 2007-2009

• Faculty and other professionals from over 100 higher education institutions developed common rubrics

• 16 Essential Learning Outcomes

• Available for institutional use

Inquiry and Analysis

Critical Thinking

Creative Thinking

Written Communication

Oral Communication

Reading

Quantitative Literacy

Information Literacy

Teamwork

Problem Solving

Civic Engagement

Intercultural Knowledge

Ethical Reasoning

Lifelong Learning

Global Learning

Integrative Learning

The MSC Challenge: Scaling Direct Assessment

• Most participants were first first-time users of Taskstream

• Each had a specific role and goals

• All with day jobs and other responsibilities

• Limited opportunity to train

• Limited time to complete their work

• Focused on one activity at a time

PROJECTLEADS

DATALEADS

FACULTYSCORERS

The Technical Challenge: Engaging Distributed Users

We asked the question:Why is it so difficult for so many institutions

to engage in meaningful, sustainable assessment of student learning?

Why is it so difficult for many institutions to engage in meaningful, sustainable

assessment of student learning?

motivation

outcomes

rubrics

exams

papers

portfolios

surveys

calibration

reliability

gradesculture

politics

reputation accreditation status

mission

governance

improvement

students

faculty

administrators

fearvalue

abilitytime

funding training

identity

recognition

trustautonomy

leadership

Outcomes assessment is complex.

And…

Software doesn’t do assessment.People do assessment.

If people aren’t engaged in the work, software isn’t going to help.

We asked another question:Why is it so difficult for so many institutions

to engage in meaningful, sustainable assessment of student learning?

How easy can we make this?

Visual tracking reports to show progress of collection and scoring

PROJECTLEADS

DATALEADS

FACULTYSCORERS

Engaging the MSC users: How we did it

use space for screenshots or other large images

Track progress of collection across the project and by statePROJECTLEADS

use space for screenshots or other large images

Track progress of scoring overall and by outcomePROJECTLEADS

Visual tracking reports to show progress of collection and scoring

PROJECTLEADS

DATALEADS

FACULTYSCORERS

Engaging the MSC users: How we did it

Simple tools for tagging assignments and bulk uploading artifacts

Create and tag assignments with key metadataDATALEADS

Bulk upload student artifacts (papers)DATALEADS

Visual tracking reports to show progress of collection and scoring

PROJECTLEADS

DATALEADS

FACULTYSCORERS

Engaging the MSC users: How we did it

Basic tools for tagging assignments and bulk uploading artifacts

Simple, intuitive and tablet-friendly evaluation experience to make scoring intuitive

A simple queue of workFACULTYSCORERS

Easy interaction with the contentFACULTYSCORERS

Quick access to the rubricFACULTYSCORERS

Clear way to review scoresFACULTYSCORERS

Minimal User Training Needed

• Demonstrated the “VALUE Rubric Scoring System” to MSC participants

• Training was a 30-minute demo and 15 minutes of questions

How easy did we make it?

“I was impressed by the design of this process. Functions were intuitive, everything was easy to find and use. The interface made

the most of screen real estate for easy reading and referencing of criteria.”

MSC pilot study participant feedback

“Taskstream took any concerns about technology out of the

scoring experience.”

“Taskstream was a useful system that allowed me to assess the student artifacts quickly.”

“I was pleased with the ease of using and dependability of Taskstream.”

“Taskstream was great. It was an easy system to use, understand, and navigate.”

“Taskstream is very user-friendly.”

MSC pilot study participant feedback

Reflections and Evolution

• Improve user experience for uploading student work.

• Present assessment results with visually engaging reports.

• Offer additional ways to bring work into the system.

• Support multiple artifacts per student, assessment of multiple outcomes for each assignment.

• Provide the means for institutions to engage in similar initiatives locally.

Headwaters of Aqua…Why is it so difficult for so many institutions

to engage in meaningful, sustainable assessment of stunt learning?

Developmental TrajectoryWhy is it so difficult for so many institutions

to engage in meaningful, sustainable assessment of stunt learning?

Motivation = Expectancy x Value - Cost

Expectancy-Value Theory (Eccles, 1983)

Motivation = Expectancy x Value - Costan individual’s anticipated ability to successfully accomplish a task

Motivation = Expectancy x Value - Costan individual’s perceived importance for the task

Motivation = Expectancy x Value - Costhow much an individual perceives that he/she has to sacrifice

to accomplish the task

Expectancy x Value – Cost

Motivation

=

A breakthrough response to the complexities ofoutcomes assessment that provides a simpler pathto meaningful assessment and lets you focus moreclearly on improving student learning.

A simple path for direct assessment of learning

Critical Thinking

Written Communication

Intercultural Knowledge

What?Which outcome(s)

will you assess?

Where?Where will you get

the evidence?

course assignments

Who?Who will score

student work?

SimpleSetup

A simple path for direct assessment of learning

What?Outcome(s)

Where?Assignments

Who?Scorers

SimpleSetup

Data!Streamlined

Scoring Experience

Create Projects Simply

Easily create and track your assessment projects, and provide context to faculty about the project goals and the outcomes to be assessed

Manage Multiple Projects

View the status and progress of all of your assessment projects to see where things stand and what actions you need to take

Control How Scoring Will Take Place

Designate evaluators by outcome and distribute student work to calibrated evaluators, with or without student anonymity

Collect Work In The Way That Makes Sense For Your Community

Assemble student work for scoring through a variety of collection methods

Expectancy

Streamlined, intuitive workflows make it clear what you need to do and

how to do it

Value

Clean displays of outcomes data make it easier to focus on results

Cost

User-centered design improves efficiency, saves faculty time

Benefits of Aqua

1. Gets to actionable data more easily

2. Makes the assessment process more efficient, saves time

3. Allows you to start small and grow as it makes sense for your community

4. Offers an intuitive user experience for even the most technologically-challenged users

5. Does not require student interaction with the system

6. Cost/Pricing…

Live Site Walkthrough

assessment ∙ accreditation

e-portfolios

Innovation in Action: A Simpler Path for Meaningful Assessment Data

Ben Coulter, Ed.D.Senior Director, Campus Solutions Taskstream