GATR: Transformer Based on Guided Aggregation Decoder for 3D multi-modal Detection

In the automatic driving 3D object detection task, the multi-modal fusion method can realize the complementarity of different modal information, especially the fusion algorithm of LiDAR point cloud and camera image has been widely used. Nowadays, most point cloud-image fusion methods employ external projection matrix to achieve data alignment. However, it's not reliable in practical environments, and whether the data can be accurately aligned will directly affect the final detection performance. To alleviate this problem, we propose a novel multi-modality fusion 3D object detection algorithm based on Transformer named GATR. In our method, we give up external projection matrices to achieve the alignment of two modalities but use the attention mechanism to adaptively fuse the features of the two modes. We propose the Fusion Encoder, which uses the detection results of the image as queries to match the range image of point cloud to obtain the fusion features. The Guided Aggregation Decoder is used to output prediction and classification. We also design methods such as spatial weight graphs and category embeddings in the decoder to accelerate model convergence. Experiments on the KITTI and nuScenes datasets demonstrate that our proposed algorithm achieves a competitive level compared to some state-of-the-art algorithms.