A Viseme-Based Approach to Emotion-Consistent Facial Expression Generation in Humanoid Head Robots: Synchronizing Lip Shape, Speech, and Emotional Expression

This paper presents a viseme-based approach for generating coordinated facial expressions and lip movements in humanoid robots. The method integrates facial expression control with synchronized lip movements to address the issue of dynamic lip movements affecting emotional expression during speech. It ensures precise synchronization between lip movements and speech, while incorporating dynamic changes in other facial features, such as eyebrows and eyes, thereby enhancing the accuracy of emotional expression. The approach uses eight basic visemes to align lip movements with speech across ten emotional states(neutral, serious, happy, bored, angry, disgusted, fearful, sad, tense, and surprised). Additionally, psychological theories are applied to refine the control of eyebrow and eye movements, improving the naturalness and emotional accuracy of the robot’s expressions. Cognitive evaluation experiments demonstrate that the overall accuracy of facial expression is 94%, while the synchronization accuracy rate between emotional lip movements and speech is 74%. In comparison to speech-driven lip synchronization, this method exhibits higher emotional expression consistency and a greater variety of expressions. Future research should focus on improving dynamic emotion expression and synchronization in speech and facial movements for more natural human-robot interaction.