Modeling Student Knowledge Using Bayesian Networks to Predict Student Performance

By Zach Pardos, Neil Heffernan, Brigham Anderson and Cristina Heffernan

To evaluate the predictive performance of various fine-grained student skill models in the ASSISTment tutoring system using Bayesian networks.

Goal

• ASSISTment is a web-based assessment system for 8th-10th grade math that tutors students on items they get wrong. There are 1,443 items in the system.• The system is freely available at www.assistment.org• Question responses from 600 students using the system during the 2004-2005 school year were used.•Each student completed around 260 items each.

The Skill Models

The skill models were created for use in the online tutoring system called ASSISTment, founded at WPI. They consist of skill names and associations (or tagging) of those skill names with math questions on the system. Models with 1, 5, 39 and 106 skills were evaluated to represent varying degrees of concept generality. The skill models’ ability to predict performance of students on the system as well as on a standardized state test was evaluated.

The five skill models used:• WPI-106: 106 skill names were drafted and tagged to items in the tutoring system and to the questions on the state test by our subject matter expert, Cristina.• WPI-5 and WPI-39: 5 and 39 skill names drafted by the Massachusetts Department of Education.• WPI-1: Represents unidimensional assessment.

Background on ASSISTment

Predicting student responses within the ASSISTment tutoring system

•The ASSISTment fine-grained skill models excel at assessment of student skills (see Ming Feng’s poster for a Mixed-Effects approach comparison)•Accurate prediction means teachers can know when their students have attained certain competencies.

1. Skill probabilities are inferred from a student’s responses to questions on the system

Bayesian Belief Network

Student Test Score Prediction Process

This work has been accepted for publication at the 2007 User Modeling Conference in Corfu, Greece.

SponsorsCollaborators

Online Data Prediction Error

05

1015202530

WPI-1 WPI-5 WPI-39 WPI-106

Skill Models

Av

era

ge

Err

or

%

MCAS Test Prediction Error

0

5

10

15

20

25

30

WPI-1 WPI-5 WPI-39 WPI-106

Skill Models

Ave

rag

e E

rro

r %

Result:The finer-grained the model, the better prediction accuracy. The finest-grained WPI-106 performed the best with an average of only 5.5% error in prediction of student answers within the system.

Result:The finest-grained model, the WPI-106, came in 2nd to the WPI-39 which may have performed better than the 106 because 50% of its skills are sampled on the MCAS Test vs. only 25% of the WPI-106’s.

Predicting student state test scores

Conclusions

• A Bayesian Network is a probabilistic machine learning method. It is well suited for making predictions about unobserved variables by incorporating prior probabilities with new evidence.

Bayesian Networks

• Arrows represent associations of skills with question items. They also represent conditional dependence in the Bayesian Belief Network.

•Probability of Guess is set to 10% (tutor questions are fill in the blank)

•Probability of getting the item wrong even if the student knows it is set to 5%

2. Inferred skill probabilities from above are used to predict the probability the student will answer each test question correctly

•Probabilities are summed to generate total test score.

•Probability of Guess is set to 25% (MCAS questions are multiple choice)

•Probability of getting the item wrong even if the student knows it is set to 5%