Dual Self-Attention Transformer for Graph Node Classification

The Transformer has achieved great success in natural language processing and computer vision, but its use in the graph is limited due to its emphasis on node location relationships instead of structural relationships. We propose the Dual Self-Attention Transformer (DSAT) to address this issue. By incorporating a dual self-attention mechanism, DSAT combines the global feature learning capabilities of the Transformer model with the local feature learning capabilities of CNNs. Our approach utilizes the GAT module to learn the local neighborhood structure around each node, while the Transformer learns the graph data's global information. We also use RWPE for absolute position coding and RMSNorm to improve model training speed and feature distribution stability, which leads to better generalization performance. Our experiments on the CLUSTER and PATTERN datasets show that DSAT outperforms GNNs and graph transformer architectures, achieving the best result in graph node classification tasks.