The Transformer Enhanced Feature Representation with CNNs

The self-attention-based Transformer has revolutionized artificial intelligence, achieving remarkable success across various applications beyond traditional convolutional neural networks (CNNs). While multiple studies have explored integrating self-attention mechanisms into CNNs to enhance their performance, these approaches often encounter insufficiency in effectively utilizing self-attention principles. This research proposes a novel approach to address this challenge by integrating the rich and valuable nonlinear information encasulated in the feature maps generated by CNNs with the self-attention mechanisms. Specifically, we leverage the self-attention encoder mechanism to further capture the global dependencies and contextual relations from the informative feature maps, thus enabling more robust and discriminative feature representations. Through experiments and evaluations with benchmark datasets on classification tasks, the findings indicate that our proposed approach not only boosts the performance of CNNs but also delivers a more efficient and scalable solution, achieving the accuracies of 95.65%, 78.48% and 95.65% on datasets CIFAR10/100 and fashion mnist, respectively. This work points to a potential path for future investigations and applications in integrating the transformer encoder with CNNs.