An End-To-End Class-Aware Transformer Framework For Weakly-Supervised Semantic Segmentation

Weakly-Supervised Semantic Segmentation (WSSS) is a challenging task that utilizes image-level labels to generate pixel-level pseudo labels. Current approaches predominantly rely on CNNs, which often fail to extract integral object regions. To remedy this limitation, several recent methods have adopted transformer architectures. However, we note that transformer-based methods continue to grapple with two critical issues: over-activation, where background pixels are erroneously classified as foreground, and mistake-activation, where pixels from one foreground class are incorrectly attributed to another. Both problems stem from the long-range dependencies intrinsic to the attention mechanism. In this paper, we propose an end-to-end class-aware transformer framework, denoted as CAT, to address the two issues above. Experiments on the PASCAL VOC 2012 and MS COCO 2014 datasets show that the proposed CAT can significantly address the above mentioned issues and achieve impressive results.