2s-TAS: Two-Stream Transformer for Multi-modal Human Action Segmentation

Human action segmentation in videos has increasingly shown its significant applications in various domains, but most of previous methods rely on single data modality that limits the richness of features and the robustness of segmentation. To address this limitation, we introduce a multi-modal human action segmentation approach that harnesses the synergistic power of diverse data types from videos, named Two Stream Transformer Action Segmentation network (2s-TAS). Our proposed 2s-TAS network consists of two independent segmentation streams (temporal and spatial), which respectively handle input data of optical flow and RGB image. The effective fusion of these two independent streams is achieved through a late fusion. Additionally, unlike comparison methods, 2s-TAS utilizes a new multi-modal feature fusion method for representing human action features, which proves to be more effective than previous feature extraction methods. Evaluation experiments on the 50Salads, GTEA, and Breakfast datasets demonstrate that our method outperforms the state-of-the-art approaches. The code will be made publicly available at https://github.com/celicvml/2s-TAS.