Simultaneous motion segmentation and Structure from Motion

This paper presents a novel approach to simultaneously compute the motion segmentation and the 3D reconstruction of a set of 2D points extracted from an image sequence. Starting from an initial segmentation, our method proposes an iterative procedure that corrects the misclassified points while reconstructing the 3D scene, which is composed of objects that move independently. This optimization procedure is made by considering two well-known principles: firstly, in multi-body Structure from Motion the matrix describing the 3D shape is sparse, secondly, the segmented 2D points must give a valid 3D reconstruction given the rotational metric constraints. Our formulation results in a bilinear optimization where sparsity and metric constraints are enforced at each iteration of the algorithm. The final result is the corrected segmentation, the 3D structure of the moving objects and an orthographic camera matrix for each motion and each frame. Results are shown on synthetic sequences and a preliminary application on real sequences of the Hopkins 155 database is presented.