Robust Motion-Guided Frame Sampler With Interpretive Evaluation for Video Action Recognition

Due to the presence of redundancy and interference, frame sampling is a promising but challenging solution to mitigate the expensive computation of video action recognition. Although the motion prior has shown great potential for frame selection, existing motion-based strategies suffer from limitations in terms of robustness and interpretive evaluation. In this paper, we devise a robust frame sampling strategy called positive motion guided sampler (PMGSampler). It consists of two procedures, local motion capture and global motion statistics. At the local level, we propose two concepts about inter-frame motion amplitude and motion continuity, which helps to perceive the movement of subjects and identify abnormal events that may generate negative pseudo-motion information. Then, through a global analysis of the obtained local motions, the sampler becomes more sensitive to informative frames and robust to outliers. The proposed sampler can be applied to most existing models for improving recognition accuracy. We conduct extensive experiments on four widely-used benchmarks to demonstrate the superiority of our PMGSampler over other methods of the same type. In addition, to analyse how sampled frames influence action recognition, we present a visual interpretation method for video models, termed as spatio-temporal class activation map (STCAM). By introducing spatial and temporal branches, our STCAM is able to visualise the salience of spatio-temporal features. With the help of STCAM, we can further intuitively evaluate the performance of different sampling strategies.