Fusion Technology of Radar and RGB Camera Sensors for Object Detection and Tracking and its Embedded System Implementation

This paper proposes a Camera and Radar sensor fusion algorithm combining Radar and RGB camera for object detection. The proposed design detects the type of the object with images/videos inputs and tracks the object followed by using a radar object detection and recognition to provide the actual type and distance of the object from the radar. Utilizing cameras, the deep learning model is employed to identify the objects in the image by applying Unscented Kalman Filter (UKF) and Kalman filter to track the objects. After projecting the radar tracking points in images, the radar tracking points and the image tracking points are regarded as the input to the Track-to-Track system to generate more stable tracking points. Finally, Track-to-Track points are input to the next image tracking to stabilize the labeling of the objects in the image. The average accuracy of the proposed method is around 95%, with 15% higher compared to only using deep learning model. The proposed sensor fusion method is developed on a desktop computer and implemented on the Nvidia Xavier embedded system yielding about 10 FPS with 77GHz radar input and 640x360 image input.