Pseudo-LiDAR With Two-Dimensional Instance for Monocular Three-Dimensional Object Tracking

Establishing a framework capable of performing 3D object tracking is crucial for various applications, such as autonomous driving and robot navigation. Monocular cameras offer economic and flexible advantages over LiDAR sensors; however, monocular camera-based 3D object detection is fraught with uncertainty owing to difficulty in depth estimation, visual occlusion, and ambiguous object appearance. This uncertainty poses additional challenges to object tracking as well, making performing stable multiple-object tracking on monocular 3D detection results more complex and difficult. To address these challenges in monocular 3D multiple-object detection and tracking, we propose the innovative framework pseudo-LiDAR-MOT, which accurately infers complete 3D bounding box information from 2D image sequences captured by a monocular camera and efficiently correlates and tracks moving objects in the temporal dimension. This framework increases the object detection accuracy and tracking stability of objects in dynamic environments via the following three methods. Monocular depth prediction and 2D object detection networks are leveraged to estimate pixelwise depth and object boundaries. Subsequently, a pseudo-LiDAR approach is employed to generate a 3D point cloud for each object, where 3D point clouds are transformed into 2D representations. Furthermore, a novel 3D detection network based on image-based detection is introduced, simplifying the complexity of conventional 3D point-cloud-based detection while increasing the accuracy of monocular image-based 3D object detection. A specialized association network is designed to enhance object trajectory tracking across frames, thereby improving 3D tracking performance. The effectiveness of our approach was validated through extensive experiments conducted on the KITTI dataset. Our proposed framework exhibits excellent performance in 3D object detection and tracking, outperforming conventional methods. This study introduces innovative ideas for 3D object detection and tracking and establishes a basis for advancing autonomous driving systems.