GraphTeacher: Transductive Fine-Tuning of Encoders through Graph Neural Networks

We present GraphTeacher for fine-tuning transformer encoders by leveraging Graph Neural Networks (GNNs) to effectively train models when fully labeled training data is unavailable. When different percentages of labeled training data exist, we study popular transformer models, DistilBERT, RoBERTa, and BERT. The proposed approach uses the underlying graph structure of a corpus by allowing the transformer encoders to incorporate GNNs into the fine-tuning process. Using latent patterns and correlations identified in unlabeled data, our method aims to enhance the model’s adaptability to scarcely labeled data scenarios. Moreover, our approach excels in conducting single-instance inference, a capability not inherently possessed by models with a transductive (semi-supervised) training stage. GraphTeacher not only processes the unlabeled data effectively, as in transductive methods, but also offers an inductive inference setup for test samples. Experiments on diverse datasets and various GNN architectures show that integrating GNNs significantly enhances transformer encoders’ robustness and generalization capabilities, in particular under sparsely labeled training conditions. GraphTeacher demonstrates a noteworthy improvement, achieving up to a 10% increase in performance on the GLUE benchmark dataset compared to the baselines¹.