Semantic Instance Segmentation in a 3D Traffic Scene Reconstruction task

We research into a 3D Traffic Scene Reconstruction (3DTSR) task. 3DTSR aims to reconstruct a 3D traffic scene from video footage captured from a car's dash-camera. The 3D traffic scene provides a new platform for various services to exploit, for example, self-driving cars, driving behavior analysis, and traffic accident analysis. In our approach, we resort to a passive sensing approach which detects objects and their positions based on visual information. Spatial positions of objects in a 2D scene are lifted into a 3D scene based on information from multi-sources: (i) semantic instance segmentation, (ii) spatial position and volume estimation through orthogonal images, and (iii) prior knowledge concerning shape and volume of objects. In this paper, we focus on semantic instance segmentation, the first phase of the proposed 3DTSR method. The semantic instance segmentation task is accomplished with the Mask R-CNN model pre-trained on COCO dataset. We report the performances of the semantic segmentation task from different Detectron2 models undergone the transfer learning process using information from various datasets. We show that it is feasible to obtain the shape and appearance of objects in the road scene using our proposed segmentation process.