MonoSIM: Simulating Learning Behaviors of Heterogeneous Point Cloud Object Detectors for Monocular 3D Object Detection

Monocular 3D object detection is a fundamental but very important task to many applications including autonomous driving, robotic grasping, and augmented reality. Existing leading methods tend to estimate the depth of the input image as auxiliary information for monocular 3D detection. However, this routine suffers from the inherent gap between depth estimation and object detection, which would accumulate the prediction error. In this paper, a novel training paradigm named MonoSIM is proposed. The insight behind introducing MonoSIM is that we propose to simulate the feature learning behaviors of two heterogeneous object detectors. During the training period, monocular detectors simulate the learning behaviors of the target point cloud based detector. Hence, during the inference period, the steps of feature extraction and prediction process would be similar to the point cloud based detector as possible. To achieve it, we propose one scene-level simulation module, one RoI-level simulation module, and one response-level simulation module, which are progressively used for the detector’s full feature learning and prediction pipeline. In this paradigm, we adopt the famous M3D-RPN, CaDDN, and GUPNet detectors, and conduct extensive experiments on KITTI and Waymo Open datasets. Results show that our method has achieved a minimum 1% performance improvement for different baselines on various datasets. When compared with current state-of-the-art methods, our method only imposes the almost negligible cost for training and no extra computational cost for inference. Our codes are publicly available at https://github.com/sunh18/MonoSIM.