Two-Stage Multi-Instance Multi-Label Learning Model for Video Social Relationship Recognition

Recognizing social relationships among characters from videos facilitates multimedia content analysis and helps viewers understand the storyline quickly. However, treating the information at each moment in the video equally will affect the extraction of information related to social relationships. Most existing methods only consider the social relationships between any two characters in the video, ignoring the potential correlations of social relationships among multiple characters. To address these issues, we propose a Two-Stage Multi-Instance Multi-Label learning model (TSMIML) for video social relationship recognition. In the first stage, we focus on obtaining effective social relationship features and mining which moments' information reflects the social relationships between characters better by generating multiple instances. In the second stage, we focus on discovering which social relationships exist in the video, exploring potential correlations among social relationships to promote social relationship recognition in the video. Compared with the state-of-the-arts methods, extensive validation on the large-scale dataset has demonstrated the effectiveness of our proposed model.