3D Vehicle Detection in Highway Vibration Scenes using Multi-Modal Fusion Network with Point-Voxel Global-Align

In highway scenarios, LiDAR and cameras are prone to vibrations due to the rapid passage of vehicles through gantries and environmental factors. To address the decline in detection accuracy caused by this vibration, this study investigates the use of hard projection connections and adaptive soft attention-based connections for voxel- and point-level feature fusion. First, to maintain consistency in data augmentation between point clouds and images, a Cross Modal GT-AUG is constructed, employing a cross-modal feature alignment data augmentation method applied to both point clouds and images simultaneously. Next, to link LiDAR BEV with image features, the paper introduces a Cross Modal Multi-Head Attention mechanism, utilizing a LiDAR BEV-guided query initialization strategy to select object queries based on LiDAR BEV information. Finally, the Point-Voxel fusion with GAM is developed, employing the GAM structure for Point-Image and Voxel-Image feature aggregation. Extensive comparative experiments conducted on the nuScenes and nuScenes-C mixed datasets validate the model’s effectiveness. The results demonstrate that the proposed model successfully addresses the feature misalignment issue in scenes, while image data compensates for the semantic information corresponding to sparse point cloud locations, achieving mAP and NDS scores of 62.25% and 68.23% on the mixed validation set, respectively.