Learn From Voxels: Knowledge Distillation for Pillar-Based 3D Object Detection With LiDAR Point Clouds in Autonomous Driving

3D object detection has achieved great progress recently. However, the contradiction between high accuracy and rapid inference is a crucial issue, which is particularly evident in voxel-based networks and pillar-based networks. Voxel-based networks can achieve high accuracy, but the 3D sparse convolution backbone in voxel-based networks blocks real-time inference and model deployment. Pillar-based networks are deploymentfriendly and can achieve real-time inference, but they cannot perform as accurately as voxel-based networks. To reduce the gap in accuracy between these two types of networks as well as keep the inference speed of pillar-based networks, in this paper, we propose Learn from Voxel Knowledge Distillation (LVKD), an effective voxel-to-pillar knowledge distillation framework. In our LVKD, we design Sparse Convolution to Pillar Knowledge Distillation (SCP KD) to transfer the rich knowledge from the voxel-based teacher network to the student network. The SCP KD selects the crucial regions and transfers the rich information from the teacher network. In addition, to alleviate the representation differences between teacher and student networks and improve the performance of distillation, we propose the Voxel Occupancy Prediction module, a plug-and-play task that encourages the pillar-based network to predict the occupancy of each voxel to achieve the reconstruction of structure and spatial information. We conduct experiments on two popular public datasets (i.e., nuScenes and KITTI), and the results demonstrate the superiority of the proposed LVKD framework. In particular, our LVKD framework can enhance the performance of the pillarbased network by 3.1% in mean average precision and 2.6% in the nuScenes detection score.