Compressing Transfer: Mutual Learning- Empowered Knowledge Distillation for Temporal Knowledge Graph Reasoning

With the widespread application of temporal knowledge graph reasoning (TKGR) models, there is an increasing demand to reduce the memory consumption and enhance the reasoning efficiency. Knowledge distillation (KD) is a classical approach to achieve model compression and acceleration, which has been gradually introduced into the TKGR domain. Through KD, the expertise of a high-capability teacher TKGR model can be transferred to a lightweight student TKGR model. The effective transfer of reasoning knowledge primarily faces two major challenges. The first is how to extract high-quality and high-value knowledge from the teacher to the student to achieve better teaching outcomes. The second is how to encourage the teacher to improve the teaching pattern so that the knowledge is more easily assimilated by the student. Motivated by these challenges, this article firstly designs a soft-label evaluation mechanism to mitigate the problem of anomaly diffusion and knowledge transfer redundancy by measuring the confidence and entropy changes of soft labels, then proposes a mutual learning-empowered KD (MLEMKD) framework for compressing TKGR models. It refines the distribution of knowledge by analyzing the cognitive differences between teacher and student models on training samples, which enhances the acceptability of knowledge. Extensive experiments conducted on four benchmark datasets demonstrate that MLEMKD significantly outperforms existing KD methods.