Thermal Imaging on Smart Vehicles for Person and Road Detection: Can a Lazy Approach Work?

This paper proposes the addition of a thermal camera to an RGB system with the goal of improving person and road detection reliability in unfavorable weather and illumination conditions. Custom data is gathered on an experimental vehicle and used for development and testing. For person detection, we propose a novel multi-modal approach, where bounding boxes are initially obtained from RGB and thermal images using YOLOv3-tiny. We then identify high-intensity connected components in thermal images to compensate for missed detections. Detections from the two cameras and the two algorithms are finally weighed and combined into a confidence map. Using the proposed fusion method, recall and precision are improved compared to using RGB only, without the need to retrain the network. For thermal-based road segmentation, we achieve an average precision of 94.2% after re-training MultiNet's KittiSeg decoder on a small thermal dataset, while using pre-trained weights for MultiNet's VGG-based encoder. These results show that the addition of thermal cameras to perception systems of autonomous vehicles can bring substantial benefits with minimal labelling, implementation effort and training requirements.