Dual-Stream Deep Neural Network with Learnable Fusion Weights for 3D Object Detection and Recognition

To enable effective obstacle avoidance in autonomous vehicles, 3D object detection and recognition play a crucial role. Currently, there are various models for perceiving 3D objects in the environment, utilizing multiple sensors such as RGB cameras, LiDAR, and radar. Each sensor has its advantages and disadvantages. The objective of this study is to effectively integrate the data from these three sensors and process them in an integrated manner. Thus, we propose a dual-stream Deep Neural Network (DNN) based on the modified VoxelNet, achieved through neural architecture search. The two streams consist of the integrated data from imager, LiDAR and radar, and the LiDAR data with depth completion. We propose an improved depth completion technique to generate a dense depth map, aiming to address the point cloud's boundary blurring issues introduced by depth inpainting. The fusion of the two streams occurs through the cost function, where learnable fusion weights are employed to achieve outstanding performance. In the validation set of the nuScenes dataset, we achieved remarkable results, with an average precision (AP) score of 61.9% for car classification and 47.5% for pedestrian classification, solely using frontview data with a 77-degree field of view. These scores surpass those reported in previous literature, highlighting the effectiveness of the proposed dual-stream DNN for 3D object detection and recognition.