Motion Semantic Enhancement and Autonomous Information Mining for Static-Dynamic Visual Emotion Recognition in Human–Robot Interaction

The advancement of static-dynamic visual emotion recognition (DVER) plays a pivotal role in the evolution of intelligent and empathetic machines. Currently, the progress of static visual emotion recognition (SVER) faces challenges, primarily the need to balance model efficiency with robust recognition performance. Moreover, obstacles such as noisy data and limited labeled datasets restrict models from effectively learning the appearance features and dynamic dependencies intrinsic to DVER. In the realm of SVER, facial pixel, and semantic representation are derived using a lightweight surface and landmark features embedding network, followed by neuron-energy-based feature fusion-filtering to enhance the fusion of semantic representations positively. For DVER, a Global Anchor-Dependent Noisy Emotional Data Filtering (GADNEF) method is employed for noisy label learning (NLL), facilitating clip-wise filtering of ambiguous data via iterative computations of frame-wise attention statistics across batches. Furthermore, a self-supervised learning (SSL) paradigm based on a unified appearance and motion-guided masked autoencoder (UAMGMAE) is implemented, enabling large-scale knowledge transfer tailored for video-based facial expression analysis. Our approach has achieved accuracies of 97.07%, 98.90%, 61.39%, and 92.28% on SVER datasets including KDEF, RaFD, SFEW, and RAF-DB, respectively, while the weighted average recall (WAR) and unweighted average recall (UAR) on DVER datasets such as DFEW and MAFW were 75.12%, 64.37%, 55.26%, and 42.53%, respectively. A preliminary application experiment has also been conducted to validate the practical applicability of our method within human–robot interaction (HRI) scenarios.