An Intra-Network Multi-Teacher Distillation Method Towards Lightweight Knowledge Graph Completion

Recently, Knowledge Graph Completion (KGC) based on Pre-trained Language Models (PLM) has made significant advancements. However, PLM typically have a large number of parameters, which makes lightweight research for low-resource challenging. For KGC, knowledge distillation can be an portable method. But traditional knowledge distillation is difficult to achieve efficient knowledge transfer. To solve this issue, this paper proposes an intra-network multi-teacher knowledge distillation, which can effectively reduce knowledge leakage through multi-level information transmission. Specifically, we divide the teacher model into multiple sub-teachers based on network depth, the sub-teachers deliver different knowledge representations. In addition, we use the loss variation of each sub-teacher as a confidence level, which can dynamically regulate the intensity of multi-teacher distillation and enable the student model to perceive distilled knowledge at a finer granularity. A series of experimental results show that our proposed method achieves state-of-the-art performance with the low number of parameters.