Emotional Intensity-Aware Learning for Facial Expression Manipulation

The advancement of deep learning has significantly improved the performance of generative models, enabling realistic facial image creation and natural translation of specific facial attributes. Among various tasks, facial expression manipulation (FEM) aims to naturally synthesize facial expressions that meet user requirements. While FEM research demonstrates good performance in image-to-image translation, there are still limitations in representing facial expressions across all frames within a sequence for video-to-video translation. Inspired by the sparsity of emotion-expressive frames in a sequence, we propose Mutual Information-based Emotional Intensity-aware Learning (MIEL), which enhances learning by reconstructing sequences using key frames based on point-wise mutual information (PMI). Our method selects key frames considering valence-arousal (VA) values for emotional intensity. The qualitative results show that MIEL can express emotions more clearly than existing techniques in all six emotion classes, indicating that the mutual information-based key frame selection scheme has enhanced the effectiveness of emotion representation learning and can be represented at the pixel level.