Adding predictive elements to student and instructor

dashboardsRobert Bodily

Brigham Young University

Context• Class

• First year chemistry course• Blended – class 3x per week,

• Resources• 150 videos (avg. 2 min long, supplemental resources)• 15 weekly quizzes (unlimited question attempts)

• Participants• 200 students (online interactions)• 96 students took the self-report resource use survey

Nature of the data• Quiz

• Confidence in answer (just a guess, pretty sure, very sure)• Time spent on quiz• Correct/incorrect• Number of attempts per question• Leave tab (still open, but inactive), come back to tab (active again)

• Video• Play, pause, skip forward/backward, change play rate, change volume,

• Dashboard• Number of times students follow recommendations given in dashboard• Number of clicks within the dashboard

Student Dashboard

Instructor dashboard

GoalsOur dashboards are completely descriptive, so I want to add predictive elements to both the student and instructor dashboards1. Understand what course elements are predictive of student

achievement (grade on final exam)2. Develop an early course prediction of student success3. Determine what student profiles exist based on online behavior4. Develop a model to classify future students into groups

What course elements are predictive of student success?

Variable Beta P-valueOnline homework score 0.366 0.000In-class IClicker scores 0.154 0.024# of attempts/question -0.411 0.000Amount of question navigation -0.206 0.040# of online activity sessions -0.195 0.020

Variable Beta P-valueRead the textbook 2.443 0.059Ask professor questions in class 7.363 0.000Watch Khan Academy -2.738 0.051Use the internet -3.199 0.010Skip recitation -4.820 0.041

Model 1 – regressing online interaction data on final exam score.

Model 2 – regressing self-report resource use on final exam score.

Develop an early course prediction of student achievement

Online student interaction data Online student interaction data AND exam scores

There is significant improvement in both models until week 3 or 4, so that seems to be a good time to make predictions for instructors and students.

Clustering to find student profilesCluster 1: Higher prior knowledge, good study skills, uses email, does not use office hours

Cluster 2: Efficient, game the system, ok losing some points, office hour students, does not use email

Cluster 3: Work hard but inefficiently, use tutor/friends, low self-regulation, bad study habits

3

Develop a classification model to classify future students into groups

Classes AIC BIC SSA BIC Log Likelihood

2 8299 8460 8305 -4101

3 7870 8086 7877 -3869

4 7502 7774 7511 -3668

5 7139 7467 7150 -3470

6 6813 7196 6826 -3290

7 6612 7051 6626 -3172

8 6523 7017 6539 -3110

Group counts3. 21. 202. 1195. 246. 84. 22

Group 3: efficient smart learnersGroup 1: low online activity, efficiency driven, just getting byGroup 2: average students, lower effort, low online activityGroup 5: average students, high effort, high online activityGroup 6: low learning skills, low knowledge awareness, high effortGroup 4: low learning skills, low knowledge awareness, but put forth less effort than group 6

Student dashboard suggestions• Provide students with recommendations on the things that good

students do to succeed in the course, as well as the potential effect of these things.

• Give students feedback on how they will do on the final based on historical students similar to them. Give them something to click to act on this information (e.g. meet with TA, meet with instructor, etc.).

• Show students some examples of their online behaviors to make them more aware of their online activity. Provide recommendations to help them improve.

Instructor dashboard suggestions• Provide a list of things successful students do along with their effect

on student final exam grade so the instructor can encourage students to do them to improve in the course

• Provide a predicted pass/fail score for each student at week 3 or 4 in the course so the instructor or teaching assistants can intervene with potentially struggling students

• Provide a student profile for each student so the instructor can better personalize feedback to students