MPFNet: Multiscale Prediction Network With Cross Fusion for Real-Time Semantic Segmentation

Semantic segmentation currently plays an important role in computer vision and is widely applied in both industry and human life. The self-driving car is one of the most trending applications, which assists humans in making informed decisions. The self-driving application has to interpret visual information from street scenes. However, how to effectively segment a long range of objective sizes is still a challenging problem. A feature pyramid network (FPN) builds up an architecture by processing four different features to contribute contextual and spatial information to the final map. Each feature can suitably process a specific range of objective sizes. Nevertheless, the final feature combination is not optimal when they raise the computation cost and reduce the semantic weights. We propose a multi-scale prediction network with cross-fusion in order to address the aforementioned drawbacks. The prediction module consists of three different predictions that allow the architecture to efficiently extract information of various sizes. Each prediction is generated from a pair of feature pyramids used to predict object classes. Furthermore, the cross-scale fusion is designed to enhance the weight aggregation of the final score map. The core component of the cross-fusion is the selective attention mechanism that determines uncertain weights of the lower prediction and then selects the complement from the adjacent prediction. By implementing this proposed scheme, we have achieved good results 78.3% mIoU and 45 FPS on Cityscapes and 45.9% mIoU on Mapillary Vistas datasets. Our method outperforms the baseline method with 7.0 mIoU improvement and a 27 FPS speedup on Cityscapes dataset. The experiment results demonstrate that the proposed model achieves a reasonable balance between performance and efficiency.