A Single-Stream Segmentation and Depth Prediction CNN for Autonomous Driving

Convolutional neural networks (CNN) have been used successfully in solving many challenging visual perception tasks facing mobile robots and self-driving cars. To facilitate deploying such models on embedded hardware onboard mobile robots that have limited resources, multitask learning approaches have become common. In typically used multitask learning, a shared encoder network extracts features from inputs whereas multiple task-specific decoders transform these features into their target output. However, properly combining different tasks’ losses into the final network loss such that each task is making progress learning is a major challenge in these approaches. In this article, we present an innovative approach to extend a typical single-task network with the capability of performing two tasks without multiple decoders, i.e., a single-stream two-task network. The two output tasks are semantic segmentation and monocular depth prediction which are essential tasks in visual perception for autonomous driving. The method is centered on solving semantic segmentation with a regression loss function rather than a classification one. With our approach, we seize multitask learning benefits of reduced overhead and enhanced generalization while alleviating the need to balance different loss functions. Experimental evaluations with baseline single tasks and a multitask network are presented.