Multimodal Facial Action unit Detection with Physiological Signals

Most of the existing facial action unit detection models seek to improve detection accuracy by utilizing multiple visual modalities, including 3D geometry, thermal, and depth images. However, the potential usage of heterogeneous physiological modalities (e.g., heart rate and blood pressure) for AU detection is not considered in current works. Meanwhile, it’s challenging to fully utilize the hidden emotion-correlated physiological signals. In this paper, we propose deep networks to extract temporal features from periodic and non-periodic time-series signals and design an informativeness-based feature fusion module to handle the signal noise. Then, we utilize spatial-temporal visual representations to infer the physiological embeddings, allowing absent physiological data during testing. Experiments show that our multimodal framework achieves state-of-the-art performances on two widely used datasets: MMSE and BP4D.