Analysis of temporal alignment for Video Classification

Thanks to their sucess on image recognition, deep neural networks achieve best classification accuracy on videos. However, traditional methods or shallow architectures remain competitive and combinations of different network types are the usual chosen approach. A reason for this less important impact of deep methods for video recognition is the motion representation. The time has a stronger redundancy, and an important elasticity compared to the spatial dimensions. The temporal redundancy is evident, but the elasticity within an action class is well less considered. Several instances of the action still widely differ by their style and speed of execution. In this article, we analyze the temporal dimension by focusing on its singular dynamism, and we focus on the normalization of temporal elasticity on sequences to reduce speed variation within a class. We propose a framework to temporally align video instance in a classification task using the latest temporal warping method, Generalized Canonical Time Warping (GCTW). We evaluate our strategy on video datasets where the intra-class variations lie in temporal dimension rather than in spatial dimensions. Finally, we show the interest of accounting for temporal elasticity for a better video classification and we draw perspectives on more efficient ways to normalize simultaneously temporal and spatial intra-class variations.