GC-TopFormer: Global Context-Enhanced Token Pyramid Transformer for Semantic Segmentation on Mobile Devices

In the realm of computer vision, Vision Transformers (ViTs) have showcased remarkable efficacy, setting new performance benchmarks. However, substantial computational demands limit their utility in resource-constrained environments, particularly for sophisticated tasks like semantic segmentation on mobile devices. TopFormer, an architecture optimized for mobile devices, utilizes tokens from diverse scales to generate scale-informed semantic attributes, which are subsequently reinfused into the respective tokens to enhance the hierarchical features. In segmentation task, effectively modeling detail information is paramount, and a more comprehensive initial capture of detail information can get more precise results of object edges and finer details. GC-BLOCK fuses global context information into channel features to achieve more effective global context modeling during the early stages of segmentation, and is lightweight with a negligible increase in parameters. However, we find that TopFormer has limitations in the early feature extraction stage because of insufficient extraction of detailed information. To address this issue, this paper proposes a new approach: Global Context-Enhanced Token Pyramid Transformer for Semantic Segmentation on Mobile Devices (GC-TopFormer). Specifically, global context modeling is integrated between diverse scales of the token pyramid with global information being channel-wise integrated, which enables the proposed approach to effectively extract global information and perform better feature fusion. Extensive experiments on two challenging semantic segmentation datasets, ADE20K and Cityscapes, demonstrates that GC-TopFormer not only improves segmentation results and detail preservation but also maintains the lightweight nature.