Using Multi-Scale Feature Predictions for FPN Architecture Based Real-Time Semantic Segmentation

Semantic segmentation is a challenging topic that requires the categorization of every pixel of input images. When each pixel output is computed by neighborhood values from the previous stage, the field of views is directly affected by feature extraction. A large receptive field captures many classes for a single shoot, so it leads to poor extraction. When a receptive field is small, it can not cover the global information of large objects. Feature pyramid network (FPN) includes multi-scale features to process information. Four features with different receptive views are suitable for extracting a long range of objective sizes. However, the feature aggregation of the FPN method is not optimal when it demands huge computation resources and reduces rich semantic information by concatenating all features together. In order to address the above limitations, this paper proposes multi-scale feature predictions based on the traditional FPN method to improve the model's performance and efficiency. The proposed decoder comprises two predictions to effectively process highlighted information from each feature. A pair of the feature pyramid is selected to generate one prediction. The context and spatial branches have different contributions to the progression of information-sharing. When the atrous spatial pyramid pooling is applied to the context path to obtain denser information, the spatial branch is basically processed by a convolution layer to maintain the coarse information. The model is trained and evaluated by different public datasets. We achieved outstanding results with 76.2 % mIoU and 65 FPS on Cityscapes, and 43.5 % mIoU on the Mapillary Vistas dataset.