Enriching Optical Flow with Appearance Information for Action Recognition

Optical flow is a widely used data source for learning motion information, but the complete loss of appearance information limits its ability for action recognition. Therefore we think of enriching optical flow frames with supplementary appearance information to form a new motion data source denoted as Appearance-Supplemented Optical Flow (ASOF). Specifically, we propose a data embedding layer and a stagewise training method to mitigate the scale-wise and density-wise data distribution divergence between the RGB and optical flow frames respectively. We conduct experiments on three benchmark datasets: UCF101 [1], HMDB51 [2] and SomethingSomething-V1 [3]. The results show that our methods can prominently improve optical flow stream recognition accuracy, and further improve the performances of two-stream (score fusion with the RGB stream) with only a little storage increase.