Monocular Visual Odometry Based on Depth and Optical Flow Using Deep Learning

Visual odometry (VO) is one of the essential techniques in mobile robots field; an accurate VO system is of great significance for mobile robot simultaneous localization and mapping. As for traditional monocular VO systems, they work by presuming the monocular scale is 1 (scale = 1), or relying on ground truth (GT) to estimate scale. As a result, the traditional monocular VO systems estimate the pose state with big drift or cannot work on the image sequence without GT. Although some classical monocular VO systems have been proposed, they still have imperfect performance or even unable to work in some extreme scene conditions, such as scene is monotony without obvious texture information or camera large-scale displacement motion. As for learning-based VO system, it is realized by training deep neural networks in supervised or self-supervised manner to end-to-end estimate the pose state; however, the accuracy of pose estimation entirely depends on the ability of networks. Although the ability of networks can be improved by increasing the number of training data sets and optimizing the network structure, it is inevitable to encounter problems such as insufficient generalization ability and insufficient accuracy on rotational pose estimation. In this article, a monocular VO system named DL_Hybrid is proposed, which takes full advantage of DL networks used in image processing and geometric localization theory based on hybrid pose estimation methods. The DL_Hybrid VO system can estimate a six-DoF pose one-frame-by-one-frame and recover camera trajectory, and it can extract accurate key points from per-frame even in extreme scene condition, and it has good performance even in the extreme moving condition, such as camera rotation-only action or static action, also it can work well in the condition of camera large-scale displacement motion. The real scale is also accurately estimated without depending on GT, and the pose estimation method is designed based on hybrid 2d–2d and 3d–2d localization theory to make the DL_Hyrid VO system to estimate translational and rotational information with accuracy and robustness. Experimental results show that the proposed DL_Hybrid VO system has a better performance than traditional and learning-based VO systems.