Privacy-Preserving Facial Emotion Classification With Visual Micro-Doppler Signatures for Hearing Aid Applications

Facial expressions are a crucial aspect of nonverbal communication and often reflect underlying emotional states. Researchers often use facial emotion detection (FED) as a tool to gain insights into cognitive processes, emotional states, and cognitive load. The conventional camera-based methods to sense human emotions are privacy intrusive, lack adaptability, and are sensitive to variability. These technologies have limited generalization and may not adapt well to variations in ambient lighting, facial landmark localization, facial occlusions, and emotion intensity. Radio frequency (RF) sensing offers promising avenues for improvement with contactless, noninvasive, privacy-preserving, and reliable radar-based measurements. The proposed framework uses deep-learning techniques to classify facial micro-Doppler signatures generated from an ultra-wideband (UWB) radar. The method relies on continuous multilevel feature learning from radar time-frequency Doppler measurements. The spatiotemporal facial features are extracted from the radar data to train deep-learning (DL) models. The proposed system achieves a high multiclass classification accuracy of 77% on the continuously streamed data covering basic emotions of anger, disgust, fear, happiness, neutrality, and sadness. The system can transform next-generation multimodal hearing aids with emotion-aware listening effort and cognitive load detection. This can be particularly useful in translating the emotion-assisted cognitive effort for real-time speech enhancement and personalized auditory experience.