CC4S: Encouraging Certainty and Consistency in Scribble-Supervised Semantic Segmentation

Deep learning-based solutions have achieved impressive performance in semantic segmentation but often require large amounts of training data with fine-grained annotations. To alleviate such requisition, a variety of weakly supervised annotation strategies have been proposed, among which scribble supervision is emerging as a popular one due to its user-friendly annotation way. However, the sparsity and diversity of scribble annotations make it nontrivial to train a network to produce deterministic and consistent predictions directly. To address these issues, in this paper we propose holistic solutions involving the design of network structure, loss and training procedure, named CC4S to improve Certainty and Consistency for Scribble-Supervised Semantic Segmentation. Specifically, to reduce uncertainty, CC4S embeds a random walk module into the network structure to make neural representations uniformly distributed within similar semantic regions, which works together with a soft entropy loss function to force the network to produce deterministic predictions. To encourage consistency, CC4S adopts self-supervision training and imposes the consistency loss on the eigenspace of the probability transition matrix in the random walk module (we named neural eigenspace). Such self-supervision inherits the category-level discriminability from the neural eigenspace and meanwhile helps the network focus on producing consistent predictions for the salient parts and neglect semantically heterogeneous backgrounds. Finally, to further improve the performance, CC4S uses the network predictions as pseudo-labels and retrains the network with an extra color constraint regularizer on pseudo-labels to boost semantic consistency in color space. Rich experiments demonstrate the excellent performance of CC4S. In particular, under scribble supervision, CC4S achieves comparable performance to those from fully supervised methods. Comprehensive ablation experiments verify the effectiveness of the design choices in CC4S and its robustness under extreme supervision cases, i.e., when scribbles are shrunk proportionally or dropped randomly. The code for this work has been open-sourced at https://github.com/panzhiyi/CC4S