Self-supervised Learning of Occlusion Aware Flow Guided 3D Geometry Perception with Adaptive Cross Weighted Loss from Monocular Videos

Self-supervised deep learning-based 3D scene understanding methods can overcome the difficulty of acquiring the densely labeled ground-truth and have made a lot of advances. However, occlusions and moving objects are still some of the major limitations. In this paper, we explore the learnable occlusion aware optical flow guided self-supervised learning of depth and ego-motion with an adaptive cross weighted loss to address the above limitations. Firstly, we try to learn the self-discovered occlusion mask fused optical flow network by an occlusion aware photometric loss combined with temporally supplemental information and backward-forward consistency between adjacent views. And then, we design an adaptive cross-weighted loss of the geometric and photometric error between the cross tasks to distinguish the moving objects that violate the static scene assumption. Our method shows promising results on KITTI, Make3D and Cityscapes datasets on multiple tasks, and also shows good generalization ability under a variety of challenging scenarios.