Multi-Scale Vision Transformer with Local-Global Attention

The extraction of multi-scale features from input images is crucial for computer vision tasks. However, some backbone networks based on visual transformers lack the ability to effectively extract multi-scale features or use relatively simplistic methods to do so. This paper proposes a novel multi-scale visual transformer, called Multi-Scale Extraction Transformer (MSET), designed to proficiently tackle a range of downstream visual tasks. MSET incorporates innovative concepts of Multi-Scale feature extraction Embedding (MS Embedding) and Local-Global self-attention (LGSA). MS Embedding uses a unique architecture convolutional layer method that constructs hierarchical residual-like connections, allowing for the extraction of multi-scale features at finer granularity levels. Meanwhile, LGSA integrates local and global dependencies within the self-attention module, reducing computational complexity while facilitating connections between the global and local features of the image. The importance of position information in the embeddings and the impact of amplitude on model performance are also considered, allowing MSET to handle position bias information of variable size in the input image and to ensure stability during training. Numerous experiments have demonstrated the significant advantages of MSET over most existing state-of-the-art backbone networks across various visual tasks, including image classification, object detection, instance segmentation, and semantic segmentation tasks.