P2SL: Private-Shared Subspaces Learning for Affective Video Content Analysis

Most existing methods on affective video content analysis are dedicated to first obtain visual and audio representations, and then fuse them into a compact representation. However, these fusion techniques are often challenged by the modality gaps between heterogeneous modalities. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework named Private-Shared Subspaces Learning(P2SL), which projects each modality into two distinct subspaces. The first subspace is shared by all modalities, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is private to each modality and captures their characteristic features. Specifically, we introduce inter-modal similarity loss and intra-modal loss to obtain the robust features. In order to capture cues in different grains, global and local features are extracted from both visual and audio signals. Experiments show that our proposed approach outperforms the state-of-the-art methods and demonstrate the effectiveness of our approach.