MMKT: Multimodal Knowledge Tracing in Personalized E-Learning Systems for Supporting Lifelong Learning

To effectively support lifelong learning, personalized e-learning systems aim to provide customized educational services tailored to individual learners’ needs. A crucial aspect for enabling these intelligent services is knowledge tracing, which involves dynamically modeling learners’ knowledge acquisition based on their past learning records. Traditional deep knowledge tracing models primarily focus on learners’ responses and the knowledge components covered in exercises, often neglecting other valuable multimodal learning features. In response to this limitation, our contribution is the introduction of the multimodal knowledge tracing (MMKT) model, which leverages four distinct learning modalities: textual, image, cognitive parametric, and knowledge association modalities. MMKT initially employs a combination of pretrained models and fine-tuning to obtain representations for text and image modalities. Using the EM algorithm, it acquires the cognitive parameter modality for exercises and learners. Subsequently, a multimodal fusion is applied to obtain a comprehensive exercise representation. In the next step, the model explores knowledge impact weights through the knowledge association modality. This information is combined with the cognitive parameter modality of the learner and the hidden states tracked by the gated recurrent unit, resulting in a multimodal representation of the learner’s knowledge state. Experiments conducted on publicly available datasets demonstrate the significant improvement of the proposed MMKT model. The proposed model outperforms the widely used baselines, deep knowledge tracing and dynamic key–value memory networks, achieving relative AUC gains of 3.57% and 6.37%, respectively, in predicting student learning performance. This study provides valuable insights that assist learners in understanding their knowledge mastery and contribute to the realization of lifelong learning from a technical standpoint.