Exploring Image Fusion Techniques for Off-Road Semantic Segmentation in Harsh Lighting Conditions. A Multispectral Imagery Analysis

In recent years, we have witnessed significant progress in the field of autonomous mobility. However, these advancements have been highly limited to urban environments. Autonomous mobility in off-road environments is more challenging because of diverse environments, illumination conditions, lack of distinct features, among others. This paper delves into the issue of semantic segmentation for off-road conditions, focusing on the fusion of RGB (Red-Green-Blue) & NIR (Near Infrared) under intense lighting conditions. Given the huge variability associated with off-road environments, the existing datasets fail to capture strong illumination variations in the environment and therefore, to address this, we present “RPTU-Forest dataset” with 285 RGB images along with their respective multi-spectral images including (RGB, NIR, GREEN, RED and Red Edge (REG)) channel images. The paper explores the different fusion approaches for multispectral semantic segmentation documented in literature such as concatenation, Variational autoencoders, dual branch and DenseFuse fusion and provides a thorough analysis of each approach for the task of semantic segmentation. Two state-of-the-art semantic segmentation networks (UNet & DeepLab V3) are additionally compared for our use-case and the best among them is selected. The paper concludes with a qualitative and quantitative analysis. This work represents a significant contribution to the ongoing research in off-road autonomous mobility. The code is publicly available at https://github.com/ShubhamAbhayDeshpande/RobustSemanticSegmentationWithSensorFusion