Importance-Aware Semantic Segmentation for Autonomous Vehicles

Semantic segmentation (SS) partitions an image into several coherent semantically meaningful parts and classifies each part into one of the pre-determined classes. In this paper, we argue that the existing SS methods cannot be reliably applied to autonomous driving system as they ignore the different importance levels of distinct classes for safe driving. For example, pedestrian, car, and bicyclist in the scene are much more important than sky and building when driving a car, so their segmentations should be as accurate as possible. To incorporate the importance information possessed by various object classes, this paper designs an “importance-aware loss” (IAL) that specifically emphasizes the critical objects for autonomous driving. The IAL operates under a hierarchical structure and the classes with different importance are located in different levels so that they are assigned distinct weights. Furthermore, we derive the forward and backward propagation rules for IAL and apply them to four typical deep neural networks for realizing SS in an intelligent driving system. The experiments on CamVid and Cityscapes data sets reveal that, by employing the proposed loss function, the existing deep learning models, including FCN, SegNet, ENet, and ERFNet, are able to consistently obtain the improved segmentation results on the pre-defined important classes for safe driving.