Attention-Based Layer Fusion and Token Masking for Weakly Supervised Semantic Segmentation

Weakly Supervised Semantic Segmentation (WSSS) has received increasing attentions recently, which leverages only limited supervision to train the segmentation networks. Meanwhile, Transformer utilizes multi-head attention mechanism to overcome the limited receptive fields of convolution neural networks (CNN). As a result, a large amount of WSSS work has been built upon Transformer architecture. However, most of them overlook over-smoothing issue (i.e. the patch tokens in deep layers tend to be similar, indicating that the learned information is nearly identical), which will affect the accuracy of Class Activation Map (CAM) in representing the pixel level semantic information. In this paper, an Attention-based Layer Fusion scheme with Token Masking is proposed to address the above issue. The core modules include a Weighted Layer Attention Aggregation (WLA) module and a Random-Masked Class Token Refinement (RMTR) module. The former integrates the attention information of different layers in an adaptive way, which refines CAM to obtain a refined object contour. The latter introduces random masks to aggregate higher-level semantic information. Extensive experiments have been conducted on public datasets, in which our network manifests superior performances than other state-of-the-arts methods.