Learning Similarity: Feature-Aligning Network for Few-shot Action Recognition

Deep learning structures have achieved impressive results in action recognition. However, most of deep models require extensive training on large scale datasets. Besides, Insufficient data can easily lead to overfitting. In this work, we propose a conceptually simple, flexible, and general approach for few-shot action recognition, where a model must learn to reliably classify an example having seen only few previous instances which belongs to the same class with it. Our method, called the Feature-Aligning Network (FAN), can be trained with a small amount of data. By applying "alignment" on representations from a pair of videos, we use a CNN to get an incorporating feature and learn a nonlinear similarity metric of it. In this way, FAN mainly focuses on capturing similarities and differences in the same type of feature maps between two feature map sets. We conduct standard few-shot classification experiments on UCF11, UCF101 and HMDB51 datasets, showing the ability of our model that it can quickly learn from few examples. Moreover, FAN is also applicable to action similarity labeling task, which is not only competitive, but also far simpler and more efficient than other approaches.