Improving Multi-Scale Models with A Comparative Framework for Semantic Segmentation

State-of-the-art models of semantic segmentation, based on global convolutional network, ASPP, self-attention, and so on, focus on capturing context information through the fusion of multi-scale features and integrating local features with large kernels. However, these models have not been compared yet in parallel to interpret their relative efficacies. This makes it difficult to further combine or improve these models due to their complicated network structures. In this paper, a general multi-scale framework of semantic image segmentation was proposed to investigate and compare the network structures of the models in parallel. Three alternative modules were proposed to improve these methods, and the experiments show the proposed modules can give superior segmentation results and achieve outstanding performance on Pascal VOC2012 images segmentation datasets. Additionally, this framework was shown to be flexible for integrating the multi-scale features and operations from different levels. Experimental results show that the low-level operation can extract local details and the high-level operation the overall contour, so the output features from different levels complement each other to improve the performance effectively.