Refining Skill Classification with Interactive Machine Learning

This Innovative Practice Full Paper presents our work on using machine learning to estimate student mastery of Calculus skills based on their performance using an online Problem Solving Learning Environment. We have shown in earlier papers that we can make accurate predictions of student performance at an aggregate level using the Performance Factors Analysis (PFA) approach. We had an expert teacher label 243 Calculus problems with the skills required to solve each problem and then use PFA to predict student performance using a dataset with 1609 students. In this paper, we improved our labeling of the skills by adding new difficulty tags as well as showing how to apply interactive machine learning to help the expert teacher improve their labeling of the skills required for each problem. This can then improve the prediction accuracy for future classes doing the same problems. It can also help the expert teacher improve the definition of the skills to make them more effective features for predicting student performance. The technique we present is to use a human-in-the-loop hill climbing process where a skill is added or removed, and we test to see whether this significantly improves the predictions for that problem over all students. If so, then the instructor is asked to verify whether indeed that particular skill should be removed or added. The new skill could be the result of the expert making a mistake in labelling a problem with skills, but it could also suggest that a modification of the skill definition would yield more accurate predictions. The key idea behind this approach is to harness the strengths of both the instructor and the machine learning algorithms to improve the effectiveness of the pedagogical system.