Deep Learning Based Architecture Reduction on Camera-Lidar Fusion for Autonomous Vehicles

Autonomous vehicles (AVs) are the dream of the present era and are close to become reality. In AVs, perception is a challenging task. It gives understanding of the driving environment. One type of such task is road detection, where the goal is to segment the road area into drivable and non-drivable using multi-modal sensors like cameras and lidars. For their ability on a road detection task, deep neural networks, with an encoder-decoder architecture, are chosen in this paper. Since deep learning models have large size and AVs have constrained computational power, model reduction is important. Therefore, architecture reduction of a convolutional neural network is proposed on a deep learning based multi-modal fusion model. This model is used as the baseline of our work, and camera and lidar are its modalities. The baseline model's weights that are used to fuse the camera processing pipeline with the lidar pipeline are analyzed. The analysis shows that the strength of fusion between the two modalities changes from layer to layer. Using this result and a support from generic encoder-decoder architecture, a reduced architecture is proposed. The latter is further processed by removing some layers of the baseline to produce a lite model. The reduced architectures are validated to show comparable performance with the baseline. Furthermore, both the reduced architectures outperform the baseline on a brightness adjusted camera image. These reduced architectures can be used from the perspective of embedded system, or they can be used to boost performance by appending additional algorithm. The training and validation are done on the KITTI dataset.