Distill Graph Structure Knowledge from Masked Graph Autoencoders into MLP

In recent years, graph neural networks(GNNs) have been increasingly used in collaborative computing applications such as recommender systems and social networks. GNNs perform message passing, aggregating local neighbors to update node representations, which leads to scalability issues in applications, and they are difficult to deploy in real-world applications. To address this problem, recent approaches use knowledge distillation(KD) to learn to compute efficient multilayer perceptron(MLP) by mimicking the output of GNN. However, existing GNN knowledge distillation methods require an input of specific node location features to capture the structural information of the graph, which limits the knowledge migration from GNN to MLP. Here, we propose a novel framework, GMAE-MLP, that uses representational similarity to bridge the powerful graph representation space of GNN and MLP. We employ the graph mask autoencoder (GMAE) as a structure-aware graph encoder, which is able to represent the local structure of reconstructed mask nodes as meaningful structural features. We then propose a feature similarity-based distillation approach to transfer structural knowledge from GNNs to MLPs adequately. Extensive experiments and analyses demonstrate the superiority of GMAE-MLP, and We achieved state-of-the-art performance for GNN-MLP distillation in four of the five graphical datasets in the transductive setup. We show that GMAE-MLP improves GNNs by an average of 2.95%, MLPs by 26.03%, and the state-of-the-art NOSMOG by 0.95% across 5 datasets in the transductive setting.