predicting student performance in solving parameterized exercises

Predicting Student Performance in Solving Parameterized Exercises

Shaghayegh Sahebi (Sherry)1, Yun Huang1, and Peter Brusilovsky1,2

1 Intelligent Systems Program, University of Pittsburgh2 School of Information Sciences, University of Pittsburgh

Predicting Student Performance in Solving Parameterized Exercises 2Shaghayegh Sahebi (Sherry)

Parameterized Questions

• One question template with multiple parameter sets– One template generates many questions– Each question be repeated multiple times by the

same student– Makes cheating difficult– The student can learn by practicing over time


A parameterized question from QuizJET


The Challenge

• Unproductive repetitions – Students who are not good in managing their

learning [Hsiao et. al, 2009]

• How to avoid this?– Personalized e-learning system– Predict the success of students’ future attempts

the same way as recommender systems– Predicting students’ performance (PSP)


PSP for parameterized questions: how is it different from static questions?

• In static questions, the student solves a problem once– No attempt sequence on each question– Time-ignorant methods work well• Collaborative filtering approaches

• Assumption in parameterized questions: the student can learn by practicing over time– Attempt sequence for each student on each

question


Our Goal

• To study the – recommender systems approaches – effect of attempt sequence

in PSP for parameterized questions

• Approaches: – Bayesian Knowledge Tracing (BKT)– Performance Factor Analysis (PFA)– Bayesian Probabilistic Matrix Factorization (BPMF)– Bayesian Probabilistic Tensor Factorization (BPTF)– Max baseline


Bayesian Knowledge Tracing (BKT)

• Markov Model with two states• Models attempt sequence explicitly

K K K

Q Q Q

Initial knowledge

LearningP(T)

P(G),P(S)


Performance Factor Analysis (PFA)

• Regression model

• No attempt sequencing but implicitly models attempt history


Matrix Factorization (BPMF)

• From collaborative filtering • No attempt sequence modeling• We use Bayesian Probabilistic Matrix

Factorization (BPMF) [Xiong et al., 2010]

• Other models were used for static questions [Thai-Nghe et al., 2011]

1 0 0 01 1 0 10 0 1 10 0 0 1St

uden

ts

Questions/ topics

0.9

0

1.5

0.4

0 1.4

0 0.9

Stud

ents

KCs

0.8

0.5

0 0.3

0 0 0.5

0.8

KCs

Questions/ topics


3D-Tensor Factorization (BPTF)

• Adds attempt sequence modeling to BPMF• We use Bayesian Probabilistic Tensor

Factorization (BPTF)• Other models used for static questions

Stud

ents

Time

Questions/ topics

…


Max Baseline

• Predicting success (majority class) for every attempt


Data

• From QuizJET system• Java Programming Questions• Six semesters• 166 Students• 103 questions• 69.04% success records (majority class)


Study Setup

• Time-aware methods:– BKT: explicitly– PFA: counting previous success/failure– BPTF: student’s performance changes smoothly over time

• Time-ignorant methods:– Matrix factorization (BPMF)– Max baseline

• Collaborative filtering approaches:– Tensor factorization (BPTF)– Matrix factorization (BPMF)

• Knowledge component: question• 5-Fold user-stratified cross validation

– 80% of users in train data, rest in test data


Results


Time-aware methods perform better that matrix factorization

BKT PFA BPTF BPMF Max-Baseline66

67

68

69

70

71

72

73

74

75

76

Accuracy


BKT overestimate student’s performance

BKT PFA BPTF900

950

1000

1050

1100

1150

1200

False Positive


PFA and BPMF underestimate student’s performance

BKT PFA BPTF0

50

100

150

200

250

300

350

400

450

False Negative


PFA predicts success better

BKT PFA BPTF BPMF0

20

40

60

80

100

120

140

Minority RecallMajority Precision


BKT predicts failure better

BKT PFA BPTF BPMF0

20

40

60

80

100

120

140

160

180

Majority RecallMinority Precision


Conclusion

• Attempt sequence is important in PSP for parameterized questions

• Recommender systems approaches are as good as the pioneers PSP methods – if they consider attempt sequence– Do not need to know the exact Knowledge

Components– Encourages more research on applying more

recommendation techniques in PSP


Future work

• Other collaborative filtering approaches

• Ensemble of approaches

• Effect of knowledge structure (our AIEDCS paper)

• Personalize students’ experience according to our results


Thank You!


Implementation

• EM algorithm for BKT and set the initial parameters as follows: p(L0) = 0:5 , p(G) = 0:2 , p(S) = 0:1 , p(T) = 0:3 . For running PFA, we use

• the implementation of logistic regression in WEKA [3].

• For BPTF and BPMF: Matlab code prepared by Xiong et. al. We experimented with different latent space dimensions for BPTF and BPMF (5, 10, 20 and 30) and chose the best one, which has the latent space dimension of 10


Predicting Students’ Performance

• Predicting the student’s capability to solve a problem or perform an educational task, mostly based on her performance in the past

• Predicting success/failure in solving a question

• Questions can be related to topics (Here, each topic can have multiple questions and each question is related to one topic)


Results

No significant accuracy difference between all methods except BPMF and Max Baseline (P<0.05)


Results


Results

PFA tends to predict more failures for the students.


Results

If BKT predicts a failure for a student, this prediction is more likely to be true compared to the other methods


Results

if PFA predicts a success for a student, this prediction is more likely to be true compared to the other methods


• Maj. Prec: TP/(TP+FP)• Min Prec: TN/(TN+FN)

• Maj. Recall: TP/(TP+FN)• Min Recall: TN/(TN+FP)

• Accuracy: (TP+TN)/all

predicting student performance in solving parameterized exercises

Data & Analytics

predicting student performance

parameterized exercises

students performance

time attempt sequence

attempt sequence modeling

kcs questions topics

effect of attempt sequence

students kcs