bridging the gap from knowledge to action: putting analytics in the hands of academic advisors
DESCRIPTION
Short Paper Presentation at Learning Analytics and Knowledge Conference 2012, May 1. #LAK12 This paper presents current findings from an ongoing design- based research project aimed at developing an early warning system (EWS) for academic mentors in an undergraduate engineering mentoring program. This paper details our progress in mining Learning Management System data and translating these data into an EWS for academic mentors. We focus on the role of mentors and advisors, and elaborate on their importance in learning analytics-based interventions developed for higher education.TRANSCRIPT
USE LabDigital Media Commons
Bridging the Gap from Knowledge to Action: Putting Analytics in the Hands of Academic Advisors
Steven LonnAndrew Krumm
R. Joseph WaddingtonStephanie Teasley
University of Michiganwww.umich.edu/~uselab
USE LabDigital Media Commons
Research Setting:M-STEM Academy
• Undergraduate engineering mentoring program• Historically underrepresented students • 200 Engineering students in 4 cohorts
2
USE LabDigital Media Commons
Goals of the project
• Utilize data stored in campus learning management system to:
• Provide timely and targeted data on student performance to M-STEM mentors
• Shorten the timespan from problem identification to intervention
3
USE LabDigital Media Commons
Supporting M-STEM mentors
• Iteratively develop • Metrics for comparing
students using LMS data• Classification schemes• Visualizations of student
performances• Send mentors weekly updates
Photo%Credit:%h,p://teacherrogers.wordpress.com
4
USE LabDigital Media Commons
How does mentor’s use of EWS affect student outcomes?
5
ImprovedPerformance
Analysis
Product
StudentAudience
Action
Student
Data
Mentor
academic resources, study strategies
6
Communication
EWS
Classification
Face-to-Face / Email
ImprovedPerformance
Analysis
Product
StudentAudience
Action
Student
Data
Mentor
academic resources, study strategies
6
Communication
EWS
Classification
Face-to-Face / Email
Assignments
Gradebook
“Presence”
Weekly%query%of%LMS%for%courses%that%include%an%MASTEM%student%and%use%the%Gradebook%or%Assignments%tool
8
USE LabDigital Media Commons
Measures from LMS data
• Gradebook and Assignments tools allow up-to-date tracking of student performances
• Report student-level information for M-STEM students• Percent of available points earned• Course averages (all students)
9
USE LabDigital Media Commons
• “Presence” events serve as a proxy for effort and are events common to all courses• Cumulative and week-to-week “Presence”
10
Measures from LMS data
11
75%
25%
mean
USE LabDigital Media Commons
11
75%
25%
mean
Cumulative “Presence” events can be highly
predictive for students’ final course grade performance
USE LabDigital Media Commons
12USE LabDigital Media Commons
12USE LabDigital Media Commons
Classification scheme• Absolute grade thresholds• Difference from course average • Presence cutoff
12USE LabDigital Media Commons
Classification scheme• Absolute grade thresholds• Difference from course average • Presence cutoff
Comparisons• Grades (course average)• Percentile Ranks (presence)
Classification schemeStudent % Relative Distance Presence Percentile Rank E3
>=0.85 . . Encourage0.75<=X<0.85 <-0.15 . Explore0.75<=X<0.85 >=-0.15 <0.25 Explore0.75<=X<0.85 >=-0.15 >=0.25 Encourage0.65<=X<0.75 <-0.15 <0.25 Engage0.65<=X<0.75 <-0.15 >=0.25 Explore0.65<=X<0.75 >=-0.15 . Explore0.55<=X<0.65 >=-0.10 . Explore0.55<=X<0.65 <-0.10 . Engage<0.55 . . Engage
13USE LabDigital Media Commons
Mentor summary
14USE LabDigital Media Commons
15USE LabDigital Media Commons
Student Detail Report
USE LabDigital Media Commons
Benefits of EWS use
• Contacting students• Shortening time to intervention• Viewing longitudinal trends• By individual course• Across all courses
• Contextualizing M-STEM student performance
16
USE LabDigital Media Commons
How mentors use the EWS
17
USE LabDigital Media Commons
Next Steps
• New infrastructure• New versions• Instructor• Students
• Messaging system• Recommendations (from person, from system)
18
ImprovedPerformance
Analysis
Product
StudentAudience
Action
Student
Data
Mentor
academic resources, study strategies
19
Communication
EWS
Classification
Face-to-Face / Email
USE LabDigital Media Commons
USE LabDigital Media Commons
Conclusion• Closing the gap between problem identification and
intervention
• Organizational capacity and the success of learning analytics• “Analytics” is but a small part
• Information is always subject to interpretation• How can we scaffold interpretation and effective
action-taking?20
USE LabDigital Media Commons
Collaborators
M-STEM• Cinda-Sue Davis• Guy Meadows• James Holloway• Daryl Koch• Mark Jones• Debbie Taylor
ITS• Bryan Hartman• Jeff Jenkins• Dan Kiskis
USE Lab• Gierad Laput
21
USE LabDigital Media Commons
Questions
www.umich.edu/~uselab
slides: www.slideshare.net/stevelonn
22
Steve [email protected] @stevelonnStephanie [email protected] @stephteasley