Knowledge Distillation Based on Knowledge Graph Data Augmentation

Two-stage distillation, as the name suggests, first general distillation using a task-independent general corpus, and then task-specific distillation using an augmented taskspecific corpus, is popular because of its practicality and wide applicability. Currently available data enhancement methods include substituting similar words, adding similar sentences, etc., which we believe do not do a good job of mining the relationships between corpora. There are some obvious relationships between the data, and simply replacing similar words or calculating the similarity between whole sentences does not reveal the relationship between them. We propose a new way of data enhancement, i.e. using knowledge graphs to enrich the corpus. In our experiments, we find that by combining the data augmentation approach with knowledge graphs, we can significantly improve the accuracy of the model, and our approach can improve the model performance faster in the same training time. In addition, we propose a method to vectorise computational similarities in the knowledge graph, which improves computational efficiency. These results suggest that data augmentation using knowledge graphs is a more efficient way to better explore the structure of the data and thus improve model performance.