Patch-Based Privacy Attention for Weakly-Supervised Privacy-Preserving Action Recognition

Privacy-preserving action recognition aims to prevent privacy leakage by learning to anonymize video frames for action recognition. However, apart from video-level action labels, supervised methods require costly frame-level privacy labels. To achieve privacy-preserving action recognition in the absence of privacy labels, weakly-supervised privacy-preserving action recognition proposes to merely utilize action labels for learning without using any privacy labels during training, and therefore the challenge lies in removing the privacy information without privacy annotations. Inspired by the fact that private information such as the identity of the participant is not significant to action recognition, our main idea is to utilize the attention mechanism to automatically discover the action-sensitive information in frames and remove the other information to prevent privacy leakage without using the privacy labels. Our method contains a novel patch-based privacy attention module and an action recognition module. The patch-based privacy attention module splits raw frames into patches and exploits self-attention among patches to adaptively discover action-sensitive but privacy-less information. Our patch-based privacy attention module mines the action-sensitive information from both individual frames and adjacent frames to generate anonymized frames. A distance correlation loss is introduced to enforce that the generated anonymized frames are distinguished from the original frames and contain less private information. In addition, the action recognition module learns to recognize actions based on anonymized frames. Extensive experiments demonstrate that our model can effectively alleviate privacy leakage and maintain the performance of action recognition without using privacy labels.