External Joint Calibration of A Novel Multi-Modal Perception System

We design a new multi-modal perception fusion system for the whole scene, which can be divided into two parts. The first part is lidar, rgbamera1, rgbamera2. The second part is RGB camera 3, RGB camera 4, depth camera. The combination of these three modes provides a close range (0.3-10 m) of environmental perception, and provides the texture and distance information of the environment in the cone-shaped blind area of lidar. Our main work is to design the experimental equipment of the proposed multi-modal perceptual fusion system, and to push forward the optimization theory with the re projection error as the objective function and analyze the multi-modal fusion sensing system. The graph structure is established, and the relative pose transformation between various modes is analyzed. In the back-end optimization part, MCPE (Minimally connected pose estimation) and PSE(Pose and structure estimation)are used to analyze and compare the system. In the experimental part, we use matlab to calibrate the binocular camera by minimizing the error of re projection. The lidar, RGB camera and depth camera are calibrated with the help of the multi-modal calibration tool (binding on ROS) proposed. It provides a solid foundation for the future experiments in perception of self-driving, UAV perception, medical field and military field.