Molecular Structure Learning with Graph Transformers: A Graph Reduction Approach for Improved Efficiency and Accuracy

Transformer models have demonstrated impressive performance across various domains, yet their application to non-NLP fields, such as chemical and biological informatics, remains challenging due to difficulties in tokenizing and embedding complex data structures. While combining Large Language Models (LLMs) with large-scale pretraining shows potential, issues like computational cost, memory usage, and overfitting still pose significant obstacles.In this work, we present modifications to the Graphormer architecture aimed at addressing these limitations. Our contributions include a chemically intuitive graph reduction strategy, structural feature normalization, and a learnable graph readout. These improvements enhance efficiency and prediction accuracy while significantly reducing training time. Notably, our hierarchical pooling approach compresses graphs by reducing node count while preserving essential structural features, enabling more effective learning. Validation on molecular datasets demonstrates that our method reduces training time by an average of 98.35% and decreases testing error by approximately 47.97%. These results highlight the potential of our approach for large-scale pretraining and downstream molecular property prediction. Our work emphasizes the value of domain-aware architectural adaptations for applying graph transformers in chemical and biological contexts.