Improving Knowledge Graph-based BERT Pre-Training Through Entity Hot Partition and Adapter Fusion

Enhancing pre-training models based on knowledge graphs is an emerging technique used to handle knowledge-intensive tasks. In the latest research presented at EMNLP 2021, an approach to multi-subgraph BERT embedding using knowledge graph partitioning is developed, demonstrating the effectiveness of parallel multiple subgraphs-based BERT embeddings. However, this approach is task-agnostic and only consider topological features during the knowledge graph partitioning process, while in practical applications such as question answering, the graph partitioning can dynamically change as users' interaction, that may neglect entity popularity features and potentially affect the performance of pre-training models. In this paper, we propose an entity hot partition(EHP) approach to multi-subgraph BERT embedding. We improve the performance of the BERT model through integrating entity hot partitioning and adapter fusion. By calculating entity popularity based on knowledge graph triplets, we construct an entity popularity weight vector and introduce it into the classic METIS method to achieve popularity-sensitive graph partitioning. Finally, the partitioned graph results are fed into adapters to facilitate fine-tuning-based BERT pretraining. We conducted extensive experiments on three biomedical BERT models, namely SciBERT, BioBERT, and PubMedBERT, across six downstream tasks. The results demonstrate the effectiveness of our approach in achieving the performance improvements under simulated popularity scenarios.