Facial Micro-Motion-Aware Mixup for Micro-Expression Recognition

Data-driven learning models have demonstrated strong benefits in capturing subtle facial movements for micro-expression recognition (MER), but are limited by the available data. Generative models can generate a variety of new data, but are typically computationally prohibitive compared to efficient Mixup-like methods. In this paper, we propose a novel Facial Micro-Motion-Aware Mixup approach for MER, namely MEMix. Our MEMix constructs a micro-motion-aware mask to select the most salient facial motions and generate a new sample with a mixed motion feature. This mixed motion feature can effectively expand the data distribution, leading to smoother decision boundaries for MER models. To demonstrate the good generality of MEMix, we integrate it with three advanced vision transformer-based models. The results show that the three integrated models consistently achieve performance improvements ranging from 4.07% to 7.32% in accuracy and from 6.54% to 9.18% in F1-score. Besides, to further explore the ability of MEMix, we propose a two-stream network called MixMeFormer, which unlocks the potential of the transformer by simply integrating mixed motion features with facial semantics for MER. Extensive experiments demonstrate that our MixMeFormer outperforms other state-of-the-art methods on three well-known micro-expression datasets.