A Supervised Approach With Transformer, CNN, Optical Flow, and Sliding Window for Temporal Video Action Segmentation

Temporal video action segmentation (TVAS) is a challenging problem in analyzing and understanding long-term video content. Action segmentation attracts significant interest from both researchers and practitioners to many applications such as video editing in social media, surveillance in security, robotics, daily activities in instructional videos, and sports analysis. This paper addresses the action segmentation problem in videos including procedural activities as a supervised approach. We proposed an end-to-end framework to segment video streams by applying the sliding window method and state-of-the-art deep models. First, proposed pipeline extracts spatiotemporal features from the input video. Second, with the supervised approach with the sliding window method, we applied domain-specific segmentation task which belongs to the text segmentation domain to video domain. Third, we performed action classification with Transformer and 3D ConvNet enabling us to combine consecutive frames and optical flow to predict each atomic action in the video. Inspired by the success of the sliding window approach in text segmentation, we assigned an atomic action class to each frame in the video sequence and implemented end-to-end temporal video action segmentation pipeline. We present state-of-the-art experimental results on two small-scale benchmarks with small datasets including 50Salads and magnetic wall chess. Our experimental results exhibit the efficiency of our methodology.