Intelligent Recognition of Fatigue and Sleepiness Based on InceptionV3-LSTM via Multi-Feature Fusion

Fatigue is a common state of mankind characterized by a reduction in the level of consciousness and alertness. Therefore, the recognition of fatigue and sleepiness has become indispensable in many alertness-dependent situations, such as when driving vehicles on public roads, performing demanding tasks in the workplace, or monitoring intensive care unit patients. This study proposes a method based on novel multi-feature fusion to detect fatigue and sleepiness by using traditional image processing and heart rate variability (HRV). The proposed method performs initial feature extraction using InceptionV3 (a convolutional neural network (CNN)), following which the second decision is made by a long short-term memory network (LSTM) using the features collected by InceptionV3 to process the sequence of video data for recognition. The LSTM provides coherent and precise sequence recognition that avoids static distortions. Then, the final decision is made by the blood volume pulse vector (PBV) method after the features are fused. Because fatigue recognition is usually employed to monitor driver fatigue, we verified the feasibility of our method by testing its ability to successfully recognize driver fatigue. Following the experiments, we compared the different steps in the proposed method with those in existing methods. We selected four other methods to perform the comparison tests and used the same videos for training networks. In comparison with state-of-the-art methods, our method in its entirety achieved an average increase of 5% in terms of both accuracy and stability.