Background Interference-Based Temporal Enhancement for Action Recognition

Temporal modeling is key for action recognition. In the process of modeling temporal information, the speed of character's movement and interference of background are two main factors. Motion Excitation(ME) module based on two-frame differencing is more suited for scenes with less motion and slower speeds but it is vulnerable to background interference. Background Interference Weakening(BIW) module based on three-frame differencing excels in scenarios with rapid motion changes and faster speeds but it tends to lose too much information about the target's movement. To break through the limitations of existing approaches, this paper proposes the following improvements: (i) We propose Background Interference-based Temporal Enhancement (BITE) module, which designed to flexibly adapt to different scenarios. A self-learning threshold parameter is designed to calculate background interference in the BITE module. (ii) When the background interference is strong, our method gives higher weight to the BIW when fusing the ME and the BIW, thus enhancing the motion information under the premise of reducing the background interference. (iii) When the background interference is weak, the model gives higher weight to ME when fusing ME and BIW. In this way, our method can better enhance the motion information without losing important information. Our method achieves superior recognition accuracy across multiple datasets such as Kinetics400, UCF101, HMDB51, thus confirming its robustness and adaptability.