Image fusion by considering multimodal partial deep neural networks for self driving during winter

Recent advancements in autonomous vehicles have gathered significant attention from researchers and the general public alike. However, one of the most critical issues encountered by autonomous vehicles is their performance degradation due to poor detection in adverse weather conditions, such as rain, snow, fog, and hail. This paper aims to address the detection problem by the fusion of different image modalities at the network level. Specifically, we modularize the existing neural network designed for detection to adopt any imaging modality such as thermal and depth imaging. By separating the network in the initial layers, the network can easily be upgraded by adding or removing any imaging modality. Our experiments demonstrate that by adopting a parallel multimodal network, object detection performance is improved at par or above the current solutions in adverse conditions.