Spacetime Gaussian Grouping for 4D Object Segmentation

Novel-view synthesis of dynamic 3D scenes is an attractive but challenging problem. Many recent methods extend the 3D Gaussian Splatting model with temporal attributes in order to achieve high-quality and real-time rendered dynamic 3D scene representation. In order to have a more complete understanding of 4D scenes, we propose Spacetime Gaussian Grouping, which classifies and segments the objects spatially and temporally. The proposed method trains the Spacetime Gaussians (STG) model in conjunction with the multi-view video object segmentation masks corresponding to the input images. These masks are used to label the 3D Gaussians with instance identities. The identities of the Gaussians allow them to be grouped in 4D space, which has a positive impact on reconstruction. Furthermore, it allows the user to select and edit the objects in a spatio-temporal manner.