Self-supervised Graph-Language Model for Document Information Extraction

Document information extraction is an important research area of document AI, and it requires a deep understanding of the multimodel information of the document, especially the structure. Early research used graphs to model the layout and internal relationship of documents and used graph neural networks to process structural messages. Recently, pretrained large language models have achieved great success in various document AI tasks, including information extraction. However, graph-based methods are still valuable for the ability to model the structure of the document explicitly. In this paper, we introduce Self-supervised Graph-Language Model (SGLM), a novel frame-work that uses graph method to benefit pretrained language models for document information extraction. We apply graph neural networks to enhance the model's capacity for modeling document structure. To achieve this, we design two special tasks, namely, direction prediction and distance discrimination, to guide the model in reconstructing the document graph. Experiments on three public datasets have been conducted to demonstrate that our method significantly improves the performance of language models on information extraction tasks.