bridging the gap from knowledge to action: putting analytics in the hands of academic advisors

USE LabDigital Media Commons

Bridging the Gap from Knowledge to Action: Putting Analytics in the Hands of Academic Advisors

Steven LonnAndrew Krumm

R. Joseph WaddingtonStephanie Teasley

University of Michiganwww.umich.edu/~uselab

USE LabDigital Media Commons

Research Setting:M-STEM Academy

• Undergraduate engineering mentoring program• Historically underrepresented students • 200 Engineering students in 4 cohorts

2

USE LabDigital Media Commons

Goals of the project

• Utilize data stored in campus learning management system to:

• Provide timely and targeted data on student performance to M-STEM mentors

• Shorten the timespan from problem identification to intervention

3

USE LabDigital Media Commons

Supporting M-STEM mentors

• Iteratively develop • Metrics for comparing

students using LMS data• Classification schemes• Visualizations of student

performances• Send mentors weekly updates

Photo%Credit:%h,p://teacherrogers.wordpress.com

4

USE LabDigital Media Commons

How does mentor’s use of EWS affect student outcomes?

5

ImprovedPerformance

Analysis

Product

StudentAudience

Action

Student

Data

Mentor

academic resources, study strategies

6

Communication

EWS

Classification

Face-to-Face / Email

ImprovedPerformance

Analysis

Product

StudentAudience

Action

Student

Data

Mentor

academic resources, study strategies

6

Communication

EWS

Classification

Face-to-Face / Email

Assignments

Gradebook

“Presence”

Weekly%query%of%LMS%for%courses%that%include%an%MASTEM%student%and%use%the%Gradebook%or%Assignments%tool

8

USE LabDigital Media Commons

Measures from LMS data

• Gradebook and Assignments tools allow up-to-date tracking of student performances

• Report student-level information for M-STEM students• Percent of available points earned• Course averages (all students)

9

USE LabDigital Media Commons

• “Presence” events serve as a proxy for effort and are events common to all courses• Cumulative and week-to-week “Presence”

10

Measures from LMS data

11

75%

25%

mean

USE LabDigital Media Commons

11

75%

25%

mean

Cumulative “Presence” events can be highly

predictive for students’ final course grade performance

USE LabDigital Media Commons

12USE LabDigital Media Commons

12USE LabDigital Media Commons

Classification scheme• Absolute grade thresholds• Difference from course average • Presence cutoff

12USE LabDigital Media Commons

Classification scheme• Absolute grade thresholds• Difference from course average • Presence cutoff

Comparisons• Grades (course average)• Percentile Ranks (presence)

Classification schemeStudent % Relative Distance Presence Percentile Rank E3

>=0.85 . . Encourage0.75<=X<0.85 <-0.15 . Explore0.75<=X<0.85 >=-0.15 <0.25 Explore0.75<=X<0.85 >=-0.15 >=0.25 Encourage0.65<=X<0.75 <-0.15 <0.25 Engage0.65<=X<0.75 <-0.15 >=0.25 Explore0.65<=X<0.75 >=-0.15 . Explore0.55<=X<0.65 >=-0.10 . Explore0.55<=X<0.65 <-0.10 . Engage<0.55 . . Engage

13USE LabDigital Media Commons

Mentor summary

14USE LabDigital Media Commons

15USE LabDigital Media Commons

Student Detail Report

USE LabDigital Media Commons

Benefits of EWS use

• Contacting students• Shortening time to intervention• Viewing longitudinal trends• By individual course• Across all courses

• Contextualizing M-STEM student performance

16

USE LabDigital Media Commons

How mentors use the EWS

17

USE LabDigital Media Commons

Next Steps

• New infrastructure• New versions• Instructor• Students

• Messaging system• Recommendations (from person, from system)

18

ImprovedPerformance

Analysis

Product

StudentAudience

Action

Student

Data

Mentor

academic resources, study strategies

19

Communication

EWS

Classification

Face-to-Face / Email

USE LabDigital Media Commons

USE LabDigital Media Commons

Conclusion• Closing the gap between problem identification and

intervention

• Organizational capacity and the success of learning analytics• “Analytics” is but a small part

• Information is always subject to interpretation• How can we scaffold interpretation and effective

action-taking?20

USE LabDigital Media Commons

Collaborators

M-STEM• Cinda-Sue Davis• Guy Meadows• James Holloway• Daryl Koch• Mark Jones• Debbie Taylor

ITS• Bryan Hartman• Jeff Jenkins• Dan Kiskis

USE Lab• Gierad Laput

21

USE LabDigital Media Commons

Questions

www.umich.edu/~uselab

slides: www.slideshare.net/stevelonn

22

Steve slonn@umich.edu @stevelonnStephanie steasley@umich.edu @stephteasley