Coupling Two-Stream RGB-D Semantic Segmentation Network by Idempotent Mappings

In RGB-D semantic segmentation tasks, it has been shown that HHA embeddings effectively encode rich depth features and using HHA together with RGB images can improve segmentation performance. In this paper, we propose a novel method to effectively integrate RGB and HHA features. By replacing identity mappings in ResNet-based two-stream network with idempotent mappings, we can couple the originally separated two branches to mix features from two modalities, while still keep the good information flow nature of ResNet. Moreover, our method does not bring any additional network blocks or parameters, and only needs very small modification on basic two-stream networks. We conduct experiments on two challenging RGB-D semantic segmentation datasets NYUDv2 and SUN-RGBD. The experiment results show that our method can significantly improve segmentation performance and our method achieves the state-of-the-art on these two datasets.