Predicting Students’ Decisions in a Training Simulation: A Novel Application of TrueSkill

Can TrueSkill predict student choices in graph-based training scenarios? We examine this question using data collected from students who engaged with a dozen branching scenarios using a browser-based training application. Students played through the graph-based scenarios to learn effective interaction skills in unfamiliar situations, during which their performance was measured against predefined learning objectives. Branching points in the scenario graph were annotated by experts as correct or incorrect with respect to one or more learning objectives. The application allowed students to re-run scenarios and presented tutoring information to guide learning. In this paper, we model student performance by giving each student multiple skill ratings, one for each learning objective. Unlike most applications of TrueSkill, students never play directly against each other; instead, each time the student makes a decision at a branch in the scenario graph, we treat this action as if the student played a game with the skill rating of the decision point. Using this model, we are able to predict student choices to a high degree of accuracy. Additionally, we show that training using the browser-based application results in asignificant increase in average student skill rating for a subset of the learning objectives.