Enabling Intelligent Immersive Learning using Deep Learning-based Learner Confidence Estimation

In today’s world, augmented reality and virtual reality (AR/VR) technologies have become more accessible to the public than ever. This brings the possibility of immersive learning to the forefront of education for future generations. However, there is still much to discover and improve in using these technologies to analyze and understand learning. This paper explores the utilization of data captured through AR/VR headsets during an immersive training program for industrial robotics. This includes data on time spent, eye gaze, and hand movement during a range of activities to track a learner’s understanding of the content and intelligently estimate learner confidence within these environments using deep learning. Leveraging a dataset that comprises responses and confidence levels from 10 individuals across 35 questions, we aim to improve the uses and applicability of confidence estimation. We explore the possibility of training a model using learners’ data to dynamically fine-tune lessons and activities for each individual, thereby improving performance. We demonstrate that a pre-trained compact LSTM classification model can be fine-tuned with relatively small data, for enhanced performance on an individual basis for better personalized learning.