A Survey of Pre-Trained Language Models IncorporatingKnowledge Graphs

Pre-trained models acquire knowledge from vast amounts of unannotated and unstructured data through self-supervised learning. However, they suffer from limitations such as inadequate performance and limited knowledge reasoning capabilities due to the lack of external knowledge guidance. To address these limitations, integrating structured knowledge from knowledge graphs into pretrained models enables them to acquire both general semantic knowledge from free text and real-world knowledge behind the text, thereby effectively addressing downstream knowledge-driven tasks. This paper introduces the concepts of pretrained models and knowledge graphs, discusses research advancements, provides an overview of methods for integrating knowledge into pretrained models, and proposes three classification approaches based on fusion methods. It also outlines the application domains where these approaches can be applied. Finally, the paper summarizes and discusses future research directions for pretrained models Integrated with knowledge.