Attention-Based Multi-Modal Multi-View Fusion Approach for Driver Facial Expression Recognition

As Advanced Driver Assistance Systems (ADAS) become increasingly intelligent, facial expression recognition (FER) has become a significant requirement for the purpose of monitoring a driver’s emotional state as well as fatigue level. An automobile system with FER is very useful in improving transportation safety by recognizing the driver’s state to provide timely alerts and potentially reduce the likelihood of accidents. While deep neural networks (DNN) based systems have achieved high accuracy in FER in recent years based on data collected under good laboratory environments, recognizing real-world facial expressions remains challenging due to variations in lighting and head pose especially prevalent in the driving scenario. In this paper, we propose an attention-based multi-modal and multi-view fusion FER model that can accurately recognize facial expressions regardless of lighting conditions or head poses, using image data of various modalities including RGB, Near-infrared (NIR), and Depth Maps from different viewpoints. The model is developed on a novel facial expression dataset we collected that includes multiple modalities captured from multiple viewpoints, with varying lighting conditions and head poses. Our multi-modal and multi-view fusion approach shows superior performance compared with models that use data from a single modality/view. The model achieves an accuracy of over 95% when recognizing drivers’ facial expressions in real-world scenarios, even in poor lighting conditions and different head poses.