A Token-Sparsity-Based Image Panoptic Segmentation Method for Natural Scenes

Panoptic segmentation is the task of assigning unique semantic labels and instance IDs to each pixel in the image, which can realize image understanding more comprehensively. However, most of the recent work is based on transformer mechanism to achieve powerful partitioning performance, but they always have huge computing consumption. We propose a token-sparsity-based image panoptic segmentation method, named the Token Sparse Transformer (TS-Former), for panoptic segmentation to achieve a balance between segmentation performance and computing consumption. Our work is based on Mask2Former, and the key improvement involves the design of a sparse pixel decoder for implementing sparse attention computation. To ensure that the most effective tokens are selected, a foreground selector-based scoring mechanism is used with ground truth labels for supervision. On the COCO dataset, our method can balance model performance and computational consumption better than most existing panoptic segmentation models. Compared to the baseline model, when $28.6 \%$ of tokens are used for sparse attention calculation, the panoptic segmentation performance(PQ) of the model is only reduced by less than $1 \%$, while the FLOPs value is reduced by $12 \%$.