3D Object Trajectory Reconstruction using Stereo Matching and Instance Flow based Multiple Object Tracking

This paper presents a method to reconstruct three-dimensional object motion trajectories in stereo video sequences. We apply stereo matching to each image pair of a stereo sequence to compute corresponding binocular disparities. By combining instance-aware semantic segmentation techniques and optical flow cues, we track two-dimensional object shapes on pixel level. This allows us to determine for each frame pair object specific disparities and corresponding object points. By applying Structure from Motion (SfM) we compute camera poses with respect to background structures. We embed the vehicle trajectories into the environment reconstruction by combining the object point cloud of each image pair with corresponding camera poses contained in the background SfM reconstruction. We show qualitative results on the KITTI and CityScapes dataset and compare our method quantitatively with previously published monocular approaches on synthetic data of vehicles in an urban environment. We achieve an average trajectory error of 0.11 meter.