A Dual-Pathway Driver Emotion Classification Network Using Multitask Learning Strategy: A Joint Verification

Negative emotion (e.g., anger and fear) may influence normal driver behavior, resulting in serious traffic accidents. Thus, developing an automatic driver emotion classification method is necessary and urgent. Most of the existing methods are performed in realistic indoor environment and always lack effective utilization of heterogeneous information, resulting in low accuracy and reliability. In this article, a novel dual-pathway driver emotion classification network using the multitask learning strategy is proposed. To illustrate the design of the proposed driver emotion classification network, three modules are constructed: 1) visual-facial data processing module; 2) driving behavioral data processing module; and 3) fusion output module. Meanwhile, considering the influence of emotional states on driving behavior, a comprehensive analysis is conducted to distinguish the positive, neutral, and negative influence on driving behavior. Furthermore, a joint verification in both realistic indoor environment (i.e., laboratory simulation on the PPB-Emo dataset) and real-world outdoor scenario is performed. The experimental results illustrate that the proposed network exhibits superior performance in terms of classification accuracy and response time, achieving good balance between classification accuracy and running speed in Internet of things scenarios.