Transformer-Based Cascade U-shaped Network for Action Segmentation

Action segmentation requires predicting the action that occurs in each frame of the original video, and existing methods tend to focus on the global relationship of the sequence, ignoring the contextual information at different granularities. To address this problem, this paper proposes a Transformer-based cascaded U-network for action segmentation. The proposed method adopts a cascaded transformer structure, where the feature sequences between the encoder-decoder are connected in a U-shape, which fully combines the global context information as well as the local context information between neighboring frames. The extended temporal convolution as well as the local window attention mechanism are used to enhance the model’s ability to perceive long-range action interactions. The proposed method outperforms the current mainstream action segmentation methods on two challenging datasets 50 salads and GTEA.