An Immersive Learning System with Multimodal Cognitive Processes Inference

Using multimodal technology to infer learners’ cognitive processes is an important means to support personalized learning. Previous cognitive process analysis is more about subjective reasoning, and less about combining physiological signals. With the rapid development of virtual reality technology, new requirements and more immersive schemes for learner’s cognitive process inferring have been proposed. Based on this, this paper proposes an immersive learning system with multimodal cognitive processes inference, which is composed of macro, mesoscopic and micro environments. The system analyzes learner’s actions, emotions and attention to infer learner’s cognitive process through more comprehensive and objective multimodal data. The system also combines eye movement, brainwaves and other data, and proposes a multimodal data acquisition and analysis scheme using HMD-based VR. This paper also provides an immersive learning cognitive processes inference scheme for individual online learning, offline classroom learning scenarios.