On the Performance Characteristics of Latent-Factor andKnowledge Tracing Models

Severin KlinglerDepartment of Computer

ScienceETH Zurich, Switzerlandkseverin@inf.ethz.ch

Tanja KäserDepartment of Computer

ScienceETH Zurich, Switzerlandkaesert@inf.ethz.ch

Barbara SolenthalerDepartment of Computer

ScienceETH Zurich, Switzerlandsobarbar@inf.ethz.ch

Markus GrossDepartment of Computer

ScienceETH Zurich, Switzerlandgrossm@inf.ethz.ch

ABSTRACTModeling student knowledge is a fundamental task of an in-telligent tutoring system. A popular approach for modelingthe acquisition of knowledge is Bayesian Knowledge Trac-ing (BKT). Various extensions to the original BKT modelhave been proposed, among them two novel models thatunify BKT and Item Response Theory (IRT). Latent Fac-tor Knowledge Tracing (LFKT) and Feature Aware Studentknowledge Tracing (FAST) exhibit state of the art predic-tion accuracy. However, only few studies have analyzed thecharacteristics of these different models. In this paper, wetherefore evaluate and compare properties of the models us-ing synthetic data sets. We sample from a combined stu-dent model that encompasses all four models. Based on thetrue parameters of the data generating process, we assessmodel performance characteristics for over 66’000 parame-ter configurations and identify best and worst case perfor-mance. Using regression we analyze the influence of differentsampling parameters on the performance of the models andstudy their robustness under different model assumption vi-olations.

KeywordsKnowledge Tracing, Item Response Theory, synthetic data,predictive performance, robustness

1. INTRODUCTIONA fundamental part of an intelligent tutoring system (ITS)is the student model. Task selection and evaluation of thestudent’s learning progress are based on this model, andtherefore it influences the learning experience and the learn-ing outcome of a student. Thus, accurately modeling andpredicting student knowledge is essential.

Approaches for student modeling are usually based on twopopular techniques: Item Response Theory (IRT) [36] andBayesian Knowledge Tracing (BKT) [9]. The concept ofIRT assumes that that the probability of a correct responseto an item is a mathematical function of student and itemparameters. The Additive Factors Model (AFM) [7, 8] fitsa learning curve to the data by applying a logistic regres-sion. Another technique called Performance Factors Analy-sis (PFA) [27] is based on the Rasch item response model [12].BKT models student knowledge as a binary variable that canbe inferred by binary observations. Performance of the orig-inal BKT model has been improved by using individualiza-tion techniques such as modeling the parameters by studentand skill [23, 35, 39] or per school class [34]. Clustering ap-proaches [25] have also proven successful in improving theprediction accuracy of BKT. Furthermore, hybrid modelscombining the approaches of IRT and BKT have been pro-posed. In [17] a dynamic mixture model has been presentedto trace performance and affect simultaneously. The KT-IDEM model extends BKT by introducing item difficultyparameters [22]. Other work focused on individualizing theinitial mastery probability of BKT by using IRT [38]. Logis-tic regression has also been used to integrate subskills intoBKT [37]. Recently, two models have been introduced whichsynthesize IRT and BKT. Latent Factor Knowledge Tracing(LFKT) [18] individualizes the guess and slip probabilitiesof BKT based on student ability and item difficulty. FeatureAware Student Knowledge Tracing (FAST) [14] generalizesthe individualized guess and slip probabilities to arbitraryfeatures.

Lately, the analysis of properties of BKT has gained increas-ing attention. It has been shown [5] that learning BKTmodels exhibits fundamental identifiability problems, i.e.,different model parameter estimates may lead to identicalpredictions about student performance. This problem wasaddressed by using an approach that biases the model searchby Dirichlet priors to get statistically reliable improvementsin predictive performance. [33] extended this work by per-forming a fixed point analysis of the solutions of the BKTlearning task and by deriving constraints on the range ofparameters that lead to unique solutions. Furthermore, ithas been shown that the parameter space of BKT models

can be reduced using clustering [30]. Other research focusedon analyzing convergence properties [24] of the expectationmaximization algorithm (EM) for learning BKT models andexploring parameter estimates produced by EM [15]. It hasbeen shown that convergence in the log likelihood space doesnot necessarily mean convergence in the parameter space.[11] have studied how good BKT is at predicting the momentof mastery. Different thresholds to assess mastery and theircorresponding lag, i.e., the number of tasks that BKT needsto assess mastery (after mastery has already been achieved),have been investigated. Using multiple model fitting pro-cedures, BKT has been compared to PFA [13]. While nodifferences in predictive accuracy between the models havebeen reported, it has been shown that for knowledge tracingEM achieves significantly higher predictive accuracy thanBrute Force. Findings from other studies, however, suggestthe opposite [1, 2]. In [4], upper bounds on the predictiveperformance have been investigated by employing variouscheating models. It has been concluded that BKT and PFAperform close to these limits, suggesting that other factorssuch as robust learning or optimal waiting intervals shouldbe considered to improve tutorial decision making. The pre-dictive performance of LFKT and FAST has been comparedto KT and IRT models in [19]. The evaluation is based ondata from different intelligent tutoring systems.

In this work, we are interested in the properties of hybridapproaches combining latent factor and knowledge tracingmodels. In extension to previous work and especially to [19],we empirically evaluate the performance characteristics ofthe two recent hybrid models LFKT and FAST on syntheticdata and compare them to the underlying approaches ofBKT and IRT. We sample from a combined student modelthat encompasses all four models. By using synthetic datagenerated from the combined model, we show the robust-ness of the models under breaking model assumptions. Byevaluating the models on 66’000 different parameter configu-rations we are able to rigorously explore the parameter spaceto demonstrate the relative performance gain between mod-els for various regions of the parameter space. Our findingsshow that for the generated data sets FAST significantly out-performs all other methods for predicting the task outcomeand that BKT is significantly better than FAST and LFKTat predicting the latent knowledge state. Furthermore weare able to identify the influence of different properties of adata set on model performance using regression and showbest and worst case performances of the models.

2. INVESTIGATED MODELSIn an intelligent tutoring system a student is typically pre-sented with a set of tasks to learn a specific skill. For eachstudent n the system chooses at time t an item i from a setof items corresponding to a particular skill. The system thenobserves the answer yn,t of the student, which is assumed tobe binary in this work. In the following, we briefly presentfour common techniques to model various latent states ofthe student and the tutoring environment.

BKT. Bayesian Knowledge Tracing (BKT) [9] models theknowledge acquisition of a single skill and is a special caseof a Hidden Markov Model (HMM) [29]. BKT uses twolatent states (known and unknown) to model if a studentn has mastered a particular skill kn,t at time t, and two

observable states (correct and incorrect) to represent theoutcome of a particular task. Therefore, the probabilisticmodel can be fully described by a set of five probabilities.The initial probability of knowing a skill a-priori p(kn,0) isdenoted by pI . The transition from one knowledge statekn,t−1 to the next state kn,t is described by the probabilitypL of transitioning from the unknown latent state to theknown state and the probability pF of transitioning fromthe known to the unknown state:

p(kn,t) = kn,t−1(1− pF ) + (1− kn,t−1)pL. (1)

In the case of BKT, pF is fixed at 0. Finally, the task out-comes yn,t are modeled as

p(yn,t) = kn,t(1− pS) + (1− kn,t)pG, (2)

where pS denotes the slip probability, which is the probabilityof solving a task incorrectly despite knowing the skill, andpG is the guess probability, which is the probability of cor-rectly answering a task without having mastered the skill.Learning the parameters for a BKT model is done usingmaximum likelihood estimation (MLE).

IRT. Item Response Theory (IRT) [36] models the responseof a student to an item as a function of latent student abil-ities θn and latent item difficulties di. The simplest form ofan IRT model is the Rasch model, where each student n andeach item i are treated independently. The outcome yn,t attime t is modeled using the logistic function

p(yn,t) =(

1 + e−(θn−di))−1

. (3)

A student with an ability of θn = di has a 50% chance ofgetting item i correct. In contrast to BKT, IRT does notmodel knowledge acquisition. The model parameters for theRasch model are learned using EM.

LFKT. The Latent Factor Knowledge Tracing (LFKT) [18]model combines BKT and IRT using a hierarchical Bayesianmodel. On the basis of the BKT model, slip and guess prob-abilities are individualized based on student ability and itemdifficulty as

pGn,t =(

1 + e−(di−θn+γG))−1

(4)

pSn,t =(

1 + e−(θn−di+γS))−1

, (5)

where γG and γS are offsets for the guess and slip proba-bilities. The model is fit by calculating Bayesian parameterposteriors using Markov Chain Monte Carlo.

FAST. Feature Aware Student Knowledge Tracing (FAST)[14] allows for unification of BKT and IRT as well, but gen-eralizes the individualized slip and guess probabilities to ar-bitrary features. Given a vector of features fn,t for a studentn at time t the adapted emission probability reads as

p(yn,t) =(

1 + e−(ωT fn,t))−1

, (6)

where ω is a vector of learned feature weights. If a set ofbinary indicator functions for the items and the students areused, FAST is able to represent the item difficulties di andstudent abilities θn from the IRT model. The parametersare fit using a variant of EM [6].

3. SYNTHETIC DATA GENERATIONSynthetic data is needed to have ground truth about theunderlying data generating model, which enables the exper-imental evaluation of various properties of a model.

The sampling procedure starts by generating N student abil-ities θn from a normal distribution N(0, σ). Then, it gener-ates I item difficulties di from a uniform distribution U(−δ, δ).Based on the initial probability pI and the learn probabil-ity pL a sequence of knowledge states kn,0, kn,1, . . . , kn,T issampled based on (1) and we therefore simulate data fromonly one skill. The time t∗ at which kn,t∗ = 1 for the firsttime is considered as the moment of mastery. The numberof sampled knowledge states is then given as T = t∗ + L,where L denotes the lag of the simulated mastery learningsystem. For each student we generate a random sequence ofitems, i.e., item indices i. Arbitrary features from the train-ing environment, such as answer times, help calls, problemsolving strategy, engagement state of the student and gam-ing attempts, can have an influence on the performance of astudent. To simulate those influences in a principled way, asingle feature f is added to the data generating model witha varying feature weight ω (and thus varying correlation tothe task outcomes yn,t).

Based on these quantities, we sample the observations yn,tfrom a Bernoulli distribution with probability

p(yn,t) =(

1 + e−(θn−di−log γn,t+ωfn,t))−1

, (7)

where

γn,t = (kn,t(1− pS) + (1− kn,t)pG)−1 − 1.

Figure 1 gives a graphical overview of the described sam-pling procedure. Our sampling model has the followingnine parameters: pI , pL, pS , pG, δ, σ, ω, I,N . The describedsampling procedure allows sampling of data that exactlymatches the model assumptions of all four models. To sam-ple BKT data we set δ = σ = ω = 0 and (7) simplifies tothe standard BKT formulation. By setting pS = pG = 0.5and ω = 0 we can sample from an IRT model. To samplefrom an LFKT model we set ω = 0 and for FAST none ofthe parameters are restricted.

4. EXPERIMENTAL SETUPParameter space. We generated a vast number of pa-rameter configurations in order to analyze the four models.The set of parameter configurations has been carefully de-signed to match real world conditions. The BKT parameters(pI , pG, pS , pL) are based on the parameter clusters foundon real world data [30]. Using a normal distribution witha standard deviation of 0.02, we sampled up to 30 points(depending on the cluster size) around each cluster mean.According to common practice [16] we scaled the studentabilities θn to have a mean of 0 and a variance of 1 andtherefore σ = 1. We sampled the parameter δ (determiningthe range of the item difficulties) uniformly from [0, 3] (ac-cording to [16]). Despite simulating only one skill, we variedthe item difficulties to account for the fact that skill modelstend to be imperfect in practice [7, 32, 20]. In accordanceto the item difficulties, the feature weight ω was varied uni-formly across [0, 1.5]. Feature values fn,t were sampled fromthe uniform distribution U(−1, 1).

𝑑𝑖

𝑘𝑛,𝑡

𝑓𝑛,𝑡

𝜃𝑛𝑦𝑛,𝑡

time steps 𝑻

students 𝑵

items 𝑰

Figure 1: Combined student model used for syn-thetic data generation. The model corresponds toLFKT with the addition of a single feature. Therelative dependencies of the observable nodes (blue)and the latent nodes (white) are shown. kn,s denotesthe latent knowledge state, di the item difficulty, θnthe student ability, yn,t the observation, and fn,t thefeature value.

For every parameter configuration we generated five foldswith N = 300 simulated students. Each fold was randomlysplit up into two parts of equal number of students. Thefirst part was used as training data and the second partfor testing. Therefore, the training data did contain unseenstudents only. As we simulated data from a mastery learningenvironment the number of tasks simulated for each studentwas determined by the moment of mastery. Based on theresults presented by [11], we set the lag of the simulatedsystem to L = 4 tasks from the moment of mastery. Wesimulated I = 15 different items with random item order.

In total, we generated 66′000 parameter configurations forpI , pG, pS , pL, δ, ω, this amounts total evaluation time (train-ing and test) of 1’280 hours and 1’351 hours for LFKT andFAST respectively. The evaluation time for the BKT was 99minutes and all configurations were evaluated in 58 minutesfor the IRT model.

Implementation. To train BKT models we used our cus-tom code that trains BKT using the Nelder-Mead simplex al-gorithm minimizing the log-likelihood. We thoroughly testedour implementation against the BKT implementation of [39].The IRT models were fit by joint maximum likelihood es-timation [21] implemented in the psychometrics library1.FAST using IRT features was shown to be equivalent toLFKT except for the parameter estimation procedure [19].As this work did not investigate different parameter esti-mation techniques, both models were trained and evaluatedusing the publicly available FAST student modeling toolkit2.

5. RESULTS AND DISCUSSIONUsing the generated data, we investigated the performancecharacteristics of the four models and evaluated their pre-dictive power and robustness under varying parameter con-figurations. For our results we generated 66′000 parameter

1An open source Java library for measurement, available athttps://github.com/meyerjp3/psychometrics.2http://ml-smores.github.io/fast/

configurations, and for each of them we generated syntheticdata for 1′500 students. Note that there are many ways tocharacterize performance differences among student modelsand we only cover a subset of these possibilities.

5.1 Error MetricsThe right choice of error metrics when evaluating studentmodels has recently gained increased interest in the EDMcommunity. In [28] some of the common error metric choicesare discussed, highlighting possible issues with the accuracyand area under the ROC curve (AUC) measure. Correla-tions between various performance metrics and the accuracyof predicting the moment of mastering a skill has been inves-tigated in [26], showing that the F-measure (equaling to theharmonic mean of precision and recall) and the recall are twometrics with a high correlation to the accuracy of knowledgeestimation. The root mean squared error (RMSE) and log-likelihood, on the other hand, are well suited if one wantsto recover the true learning parameters. Similarly, [10] con-cluded from results of 26 synthetic data sets that RMSE isbetter at fitting parameters than the log-likelihood.

In line with this previous work we investigated correlationsbetween accuracy, RMSE and F-measure across all four mod-els. For this, all models were trained and evaluated on datausing 66′000 different parameter configurations. All metricsare strongly correlated |ρ| > 0.75, p � 0.001. Our inspec-tions of the metric correlations revealed no significant differ-ences in the metric correlations among the different models.Thus, to a large extent the measures capture equal char-acteristics for the models we considered in this work. Inthe following, we therefore focus our analysis on the RMSEmeasure.

5.2 Model ComparisonOverall Performance. In a first step we investigated theoverall performance of the models. For every parameter con-figuration, we calculated the average RMSE over the fivegenerated folds. Table 1 summarizes the parameters for thebest and worst data set for every model when model assump-tions are met (see Section 3). Results show that all modelsthat model a knowledge state (all except IRT) perform bestif the slip probability is low and the guess probability ishigh. This leads to a data set that exhibits a high ratio ofcorrect observations. IRT performs best on data that hasvery distinguished item difficulties (δ is high). Notably thebest performance of FAST is achieved on a data set with-out features (ω = 0). We assume that this is due to thedecreased complexity of the data set, compared to one thatexhibits high ω. Consistently, worst case data sets exhibithigh symmetric values for guess and slip probabilities. Inthe case of LFTK and FAST worst case data sets addition-ally do not distinguish between items (difficulty range δ = 0)and for FAST the feature weights are low.

We then performed the non-parametric Friedman test overall parameter configurations to assess performance differ-ences between the models. We found that there is a statisti-cally significant difference in the performance of the models(χ2(3) = 13′065, p < 0.0001). Performing a post-hoc anal-ysis using Scheffe’s S procedure [31] shows all model differ-ences to be significant at p < 0.0001 with mean ranks of1.7156, 2.3017, 2.6898 and 3.2929 for FAST, LFKT, BKT,

Table 1: Parameters of best and worst case data setsfor each model. We only considered data sets thatmeet the model assumptions. Parameters denotedwith * are fixed according to the model assumptions(see Section 3).

Model δ pI pL pS pG ω RMSE

BKT

Best 0.00* 0.71 0.41 0.01 0.47 0.00* 0.25

Worst 0.00* 0.10 0.12 0.50 0.49 0.00* 0.48

IRT

Best 3.00 0.10 0.08 0.50* 0.50* 0.00* 0.42

Worst 0.00 0.10 0.10 0.50* 0.50* 0.00* 0.50

LFKT

Best 0.75 0.69 0.40 0.01 0.46 0.00* 0.25

Worst 0.00 0.53 0.16 0.28 0.29 0.00* 0.51

FAST

Best 0.75 0.67 0.40 0.01 0.46 0.00 0.25

Worst 0.00 0.56 0.16 0.28 0.28 0.00 0.51

and IRT, respectively. FAST therefore significantly outper-forms the other methods on our synthetic data sets. In [19]IRT performed not significantly worse than LFKT and FASTon four different data sets. The good performance of IRTwas attributed to the deterministic item ordering that al-lows IRT to infer knowledge acquisition confounded withitem difficulty. Our results support this hypothesis as in oursynthetic data set the items are in random order and IRTexhibits the worst overall performance.

Parameter Space Investigation. To gain a better under-standing of the performance characteristics of the differentmodels, we analyzed their performances across the parame-ter space. For every pair of parameters pi and pj , we dividedthe parameter configurations into bins with similar valuesfor pi and pj . We used five bins for each parameter (piand pj) resulting in a total of 25 bins. Performance of eachmodel was assessed by calculating the mean RMSE for eachbin. Significance of the observed performance differenceswas computed using the Friedman test and p < 0.05.

Figure 2a shows the relative performance of the best modelfor each parameter pair. The models are color-coded: BKTis shown in red, IRT in green, LFKT in yellow, and FAST inblue. The color gradient indicates the relative improvementof the winning model over the second best model, wheredarker colors indicate higher values. White-colored areasindicate that there is no significant difference between themodels. The plot shows that FAST is robust to parametervariations and outperforms the other models in large parts ofthe parameter space. In parts with low feature weights, i.e.,where the feature f shows only a low correlation with taskoutcomes, LFKT outperforms FAST. When the variance δof item difficulties di is low, BKT is the best model. Alow variance in di implies a good skill model, with all taskshaving approximately the same difficulty.

In contrast to Figure 2a, where we assessed the prediction

diffi

culty

ran

ge δ

learn0.1 1.0

0.0

3.00.0 0.5

0.0

3.00.1 0.5

0.0

3.00.0 1.5

0.0

3.00.1 0.9

0.0

3.0

lear

n

slip0.0 0.5

0.1

1.00.1 0.5

0.1

1.00.0 1.5

0.1

1.00.1 0.9

0.1

1.0

slip

guess0.1 0.5

0.0

0.50.0 1.5

0.0

0.50.1 0.9

0.0

0.5

gues

s

feature weight ω0.0 1.5

0.1

0.50.1 0.9

0.1

0.5

feat

ure

wei

ght ω

initial0.1 0.9

0.0

1.5

(a) Relative improvement in task outcome prediction (RMSE).

diffi

culty

ran

ge δ

learn0.1 1.0

0.0

3.00.0 0.5

0.0

3.00.1 0.5

0.0

3.00.0 1.5

0.0

3.00.1 0.9

0.0

3.0

lear

n

slip0.0 0.5

0.1

1.00.1 0.5

0.1

1.00.0 1.5

0.1

1.00.1 0.9

0.1

1.0

slip

guess0.1 0.5

0.0

0.50.0 1.5

0.0

0.50.1 0.9

0.0

0.5

gues

s

feature weight ω0.0 1.5

0.1

0.50.1 0.9

0.1

0.5

feat

ure

wei

ght ω

initial0.1 0.9

0.0

1.5

(b) Relative improvement in knowledge state prediction (RMSE).

Figure 2: Best performing models (RMSE) regarding prediction of task outcomes (a) and knowledge stateprediction (b). The color for each bin indicates the best performing model, averaged over all other parameters.We investigated BKT (red), IRT (green), LFKT(yellow), and FAST(blue). White-colored bins exhibit nosignificant difference in model performance. The color brightness indicates the relative improvement of thebest performing model over competing models, with dark colors referring to higher values. FAST is robust toparameter variations and outperforms the other models in large parts of the parameter space when predictingtask outcomes (a). BKT is the best model if the variance of the item difficulty is low (a). BKT is superiorto the other models in large parts of the parameter space when predicting knowledge states (b).

of task outcomes, we analyzed the quality of the predictionof knowledge states kn,t using the RMSE in Figure 2b. Ulti-mately, we want to predict whether a student has mastered askill or not [26, 3]. The plot uses the same parameter pairsand color codings as Figure 2a. Interestingly, LFKT andFAST are not superior to BKT when it comes to predictionof the latent state. The additional parameters that LFKTand FAST use have a direct influence on the predicted taskoutcomes and therefore improve performance when predict-ing task outcomes. They have, however, no direct influenceon the latent state kn,t of the model.

Robustness. Next, we tested the robustness of the dif-ferent models against each other. We generated ideal data(meeting the model assumptions) for all the models and theninterpolated the parameter values between these ideal cases.The classes of data sets that meet the model assumptionsfor the four models are described in Section 3. From everyclass of data sets, we selected the extreme case with the leastamount of noise. In the following, we describe these cases.

For BKT, data is generated using δ = ω = 0, assuminga perfect skill model (all tasks with same difficulty) andsetting the influence of additional (not captured) featuresto 0. Furthermore, we removed the randomness by settingpG = pS = 0. For IRT, the extreme case data was generatedusing pG, pS = 0.5, ω = 0 and by additionally setting δ = 3.As LFKT is a combination of IRT and BKT, we set the pa-rameters to pG, pS = 0.25 and δ = 1.5. Furthermore, weset ω = 0, again assuming no influence of not captured fea-tures. For FAST we used the same parameters as for LFKT,but additionally introduced a feature influence by settingω = 1.5. We linearly interpolated the parameter space in-between these extreme cases to asses model robustness whenmodel assumptions are violated. Figure 3 displays the modelwith best RMSE in this subspace that contains the extreme(ideal) cases, where pL and pI are averaged over the BKTparameter clusters presented in [30]. From these results, wecan see that BKT tends to be robust to increased featureinfluence as long as pG, pS ≤ 0.15. If the feature weightω > 0.75, FAST outperforms all the other classifiers. Forlarge differences in item difficulties and large guess and slipprobabilities, LFKT has a slight advantage over IRT.

5.3 Parameter InfluenceTo analyze the influence of the model parameters on the per-formance of the student models, we used linear regression topredict the RMSE based on the parameters of the samplingmodel. This allowed us to identify statistically significantcorrelations between the sampling parameters and the per-formance of the models despite the high dimensionality ofthe parameter space.

The sampling parameters have a direct influence on the ra-tio of correct observations in the data, e.g., a high learningprobability with low guess and slip parameters leads to ahigh ratio of correct observations. Further, if the parame-ters model fast learners then the average number of taskstends to be low since we are simulating a mastery learn-ing environment. The three models IRT, LFKT and FASTwhich explicitly model items are sensitive to this kind oflacking data, as by having fewer observed items per studentthe estimation of item difficulty becomes more difficult. To

(pS,p

G,δ

) =

a *

(0.

5,0.

5,3)

ω = b * 1.5

BKT

LFKT

IRT

FAST

Best performing model under breaking assumptions

b = 0 b = 1

a = 0

a = 1

Figure 3: Relative model performance on ideal datasets generated by linearly interpolating between pa-rameters. The colors refer to the models BKT (red),IRT (green), LFKT (yellow) and FAST (blue). Thecolor gradient indicates the relative performance asin Figure 2a. BKT and FAST are more robust tothe invalid assumptions of our experiment than IRTand LFKT.

investigate the effect of both factors, we added the two vari-ables correct ratio and average number of tasks as predic-tors to the regression model. In order to make correlationcoefficients comparable, all sampling parameters have beennormalized to have mean 0 and standard deviation 1.

Figure 4 shows the regression coefficients for all four models,with red and green denoting statistically significant and notsignificant coefficients, respectively. The variables correctratio and average number of tasks have the largest influenceon the RMSE. Both effects are significant and positive (re-ducing the RMSE). A larger range of item difficulties δ has apositive influence on the performance of all models except forthe BKT model. This is expected as BKT does not accountfor variations in item difficulty and thus larger variations initem difficulties are treated as noise by BKT, which makesprediction harder. IRT, LFKT and FAST, on the otherhand, benefit from larger variations. We assume that this isdue to the better identifiability of the effects of the differentitems. Interestingly, increasing the feature range ω has nosignificant negative effect for the models that do not takefeatures into account (BKT, IRT, LFKT), but has a posi-tive effect for FAST. The initial probability and the learningprobability have a small negative and small positive effecton performance, respectively. While these coefficients arepartially significant they have very small magnitude. Thepositive effect of the slip probability pS for all models ex-cept BKT (the effect is not significant) is rather surprising.However, the effect of a high slip probability in our samplingmodel is that it weakens the influence of the latent knowl-edge state on the task outcomes. This could explain thepositive influence for models that estimate item difficulty,since the difficulty estimates are less convoluted with effectsfrom the knowledge state. Further work is needed to provethis effect.

Regressing RMSE of BKT

Coefficent

Predictor

−0.2 −0.1 0 0.1 0.2

difficulty range (δ)

learn (pL)

slip (pS)

guess (pG

)

feature weight (ω)

initial (pI)

correctRatio

averageNumberOfTasks

Regressing RMSE of IRT

Coefficent

Predictor

−0.2 −0.1 0 0.1 0.2

difficulty range (δ)

learn (pL)

slip (pS)

guess (pG

)

feature weight (ω)

initial (pI)

correctRatio

averageNumberOfTasks

Regressing RMSE of LFKT

Coefficent

Predictor

−0.2 −0.1 0 0.1 0.2

difficulty range (δ)

learn (pL)

slip (pS)

guess (pG

)

feature weight (ω)

initial (pI)

correctRatio

averageNumberOfTasks

Regressing RMSE of FAST

Coefficent

Predictor

−0.2 −0.1 0 0.1 0.2

difficulty range (δ)

learn (pL)

slip (pS)

guess (pG

)

feature weight (ω)

initial (pI)

correctRatio

averageNumberOfTasks

Figure 4: Regression coefficients to predict RMSE based on the sampling parameter values for the modelsBKT, IRT, LFKT and FAST. Parameters with positive coefficients have a negative effect on the performanceand vice versa. Red denotes significant coefficients with p < 0.001, green coefficients are not significant.

6. CONCLUSIONSIn this work, we investigated the performance characteristicsof latent factor and knowledge tracing models by exploringtheir parameter space. To do so, we generated a vast amountof 66’000 synthetic data sets for different parameter config-urations containing data for 1’500 students each. Syntheticdata allowed us to study the model performances under dif-ferent parameter settings, and to test the robustness of themodels against violations of specific model assumptions.

We showed best and worst case performances for all themodels and investigated the relative performance gain invarious regions of the parameter space. Our results showedthat the two recently developed models LFKT and FAST,which synthesize item response theory and knowledge trac-ing, perform better than BKT and IRT. FAST even signif-icantly outperformed LFKT if reasonable features can beextracted from the learning environment. Interestingly, IRTexhibited the worst performance, which supports the hy-pothesis by [19] that random item ordering has a negativeinfluence on the performance of IRT models. However, moreanalyses are needed to investigate this effect thoroughly.Further, we investigated the models’ abilities to predict thelatent knowledge state and demonstrated that LFKT andFAST are outperformed by BKT. This raises the questionof how to adjust the two recent methods LFKT and FAST ifthe aim is to predict knowledge states; we leave this explo-ration for future work. The analysis of the model robustnessrevealed that BKT is robust to increased feature influencefor small guess and slip probabilities. For larger guess andslip, FAST outperformed the other methods.

While all sampling parameters have been carefully chosento match real world conditions, we expect real world datato exhibit more noise and additional effects not covered byour synthetic data. Thus, the achieved performance can beconsidered an upper bound on the performance achievablein real world settings. The performance of BKT dependson the quality of the underlying skill model. We have simu-lated imperfect skill models by introducing item effects, butwe did not take other sources for imperfect skill models intoaccount. Furthermore, the simulated data consisted of afixed set of items. For tutoring systems offering many varia-tions of tasks, reliable estimation of item effects is challeng-ing, which in turn influences the performance of IRT, LFKTand FAST. Moreover, the performance of FAST is driven byfeature quality, which may vary between different tutoringsystems.

Finally, it remains questionable whether and how the perfor-mance of the investigated techniques influences the learningoutcome of students in a tutoring system. We show rela-tive improvements in RMSE between models of up to 6%.However, the effect of small-scale improvements in the ac-curacy of student models on the learning outcome has beendiscussed controversially [4, 39].

Acknowledgments. This work was supported by ETHResearch Grant ETH-23 13-2.

7. REFERENCES[1] R. S. Baker, A. T. Corbett, and V. Aleven. More

Accurate Student Modeling through Contextual

Estimation of Slip and Guess Probabilities in BayesianKnowledge Tracing. In Proc. ITS, 2008.

[2] R. S. Baker, A. T. Corbett, S. M. Gowda, A. Z.Wagner, B. A. MacLaren, L. R. Kauffman, A. P.Mitchell, and S. Giguere. Contextual Slip andPrediction of Student Performance after Use of anIntelligent Tutor. In Proc. UMAP, 2010.

[3] R. S. Baker, A. B. Goldstein, and N. T. Heffernan.Detecting the moment of learning. In Proc. ITS, 2010.

[4] J. Beck and X. Xiong. Limits to accuracy: how wellcan we do at student modeling? In Proc. EDM, 2013.

[5] J. E. Beck and K. M. Chang. Identifiability: Afundamental problem of student modeling. In Proc.UM, 2007.

[6] T. Berg-Kirkpatrick, A. Bouchard-Cote, J. DeNero,and D. Klein. Painless unsupervised learning withfeatures. In Proc. NAACL-HLT, 2010.

[7] H. Cen, K. R. Koedinger, and B. Junker. Is overpractice necessary? - improving learning efficiencywith the cognitive tutor through educational datamining. In Proc. AIED, 2007.

[8] H. Cen, K. R. Koedinger, and B. Junker. Comparingtwo IRT models for conjunctive skills. In Proc. ITS,2008.

[9] A. T. Corbett and J. R. Anderson. Knowledge tracing:Modeling the acquisition of procedural knowledge.UMUAI, 1994.

[10] A. Dhanani, S. Y. Lee, P. Phothilimthana, andZ. Pardos. A comparison of error metrics for learningmodel parameters in Bayesian knowledge tracing.Technical report, UCB/EECS-2014-131, EECSDepartment, University of California, Berkeley, 2014.

[11] S. Fancsali, T. Nixon, and S. Ritter. Optimal andworst-case performance of mastery learning assessmentwith Bayesian knowledge tracing. In Proc. EDM, 2013.

[12] G. H. Fischer and I. W. Molenaar. Rasch models:Foundations, recent developments, and applications.Springer Science & Business Media, 1995.

[13] Y. Gong, J. E. Beck, and N. T. Heffernan. Comparingknowledge tracing and performance factor analysis byusing multiple model fitting procedures. In Proc. ITS,2010.

[14] J. P. Gonzalez-Brenes, Y. Huang, and P. Brusilovsky.General features in knowledge tracing to modelmultiple subskills, temporal item response theory, andexpert knowledge. In Proc. EDM, 2014.

[15] J. Gu, H. Cai, and J. E. Beck. Investigate performanceof expected maximization on the knowledge tracingmodel. In Proc. ITS, 2014.

[16] D. Harris. Comparison of 1-, 2-, and 3-parameter IRTmodels. Educational Measurement: Issues andPractice, 1989.

[17] J. Johns and B. Woolf. A dynamic mixture model todetect student motivation and proficiency. In Proc.Artificial intelligence, 2006.

[18] M. Khajah, R. Wing, R. Lindsey, and M. Mozer.Integrating latent-factor and knowledge-tracingmodels to predict individual differences in learning. InProc. EDM, 2014.

[19] M. M. Khajah, Y. Huang, J. P. Gonzalez-Brenes,M. C. Mozer, and P. Brusilovsky. Integrating

knowledge tracing and item response theory: A tale oftwo frameworks. Personalization Approaches inLearning Environments, 2014.

[20] K. Koedinger, J. Stamper, E. McLaughlin, andT. Nixon. Using data-driven discovery of betterstudent models to improve student learning. In Proc.AIED, 2013.

[21] J. Meyer and E. Hailey. A study of Rasch, partialcredit, and rating scale model parameter recovery inWINSTEPS and jMetrik. Journal of AppliedMeasurement, 2011.

[22] Z. A. Pardos and N. Heffernan. Introducing itemdifficulty to the knowledge tracing model. In Proc.UMAP, 2011.

[23] Z. A. Pardos and N. T. Heffernan. Modelingindividualization in a Bayesian networksimplementation of knowledge tracing. In Proc. UMAP,2010.

[24] Z. A. Pardos and N. T. Heffernan. Navigating theparameter space of Bayesian knowledge tracingmodels: Visualizations of the convergence of theexpectation maximization algorithm. In Proc. EDM,2010.

[25] Z. A. Pardos, S. Trivedi, N. T. Heffernan, and G. N.Sarkozy. Clustered knowledge tracing. In Proc. ITS,2012.

[26] Z. A. Pardos and M. Yudelson. Towards moment oflearning accuracy. In AIED Workshops, 2013.

[27] P. I. Pavlik, H. Cen, and K. R. Koedinger.Performance factors analysis - a new alternative toknowledge tracing. In Proc. AIED, 2009.

[28] R. Pelanek. A brief overview of metrics for evaluationof student models. In Approaching Twenty Years ofKnowledge Tracing Workshop, 2014.

[29] J. Reye. Student modelling based on belief networks.IJAIED, 2004.

[30] S. Ritter, T. K. Harris, T. Nixon, D. Dickison, R. C.Murray, and B. Towle. Reducing the knowledgetracing space. In Proc. EDM, 2009.

[31] H. Scheffe. The analysis of variance, volume 72. JohnWiley & Sons, 1999.

[32] J. C. Stamper and K. R. Koedinger. Human-machinestudent model discovery and improvement usingdatashop. In Proc. AIED, 2011.

[33] B. van de Sande. Properties of the Bayesianknowledge tracing model. JEDM, 2013.

[34] Y. Wang and J. Beck. Class vs. student in a Bayesiannetwork student model. In Proc. AIED, 2013.

[35] Y. Wang and N. T. Heffernan. The student skillmodel. In Proc. ITS, 2012.

[36] M. Wilson and P. De Boeck. Descriptive andexplanatory item response models. 2004.

[37] Y. Xu and J. Mostow. Using logistic regression totrace multiple subskills in a dynamic Bayes net. InProc. EDM, 2011.

[38] Y. Xu and J. Mostow. Using item response theory torefine knowledge tracing. In Proc. EDM, 2013.

[39] M. V. Yudelson, K. R. Koedinger, and G. J. Gordon.Individualized Bayesian knowledge tracing models. InProc. AIED, 2013.