Activity and Stress Estimation Based on OpenPose and Electrocardiogram for User-Focused Level-4-Vehicles

Increasing vehicle automation changes the role of humans in the car, which imposes new requirements on the design of in-vehicle software and hardware for flexible interior concepts. An option to meet these requirements is the development of user-focused automation based on combined user and context monitoring in real time. The system behavior may be dynamically adapted by adjusting the driving style or the interior lighting. Here, we present a hierarchical approach on the basis of semantically motivated low-level features for activity and stress recognition based on OpenPose and electrocardiogram data. A driving simulator study with 29 participants was conducted to determine the potential of the approach. Participants had to accomplish different tasks: manual driving (MD); mobile office work with varying task load levels (high task load: MO-HT, low task load: MO-LT); and relaxing (REL) during automated driving. The validation revealed that our model is able to correctly distinguish between different activities using only a set of primitive features (average precision: driving: 76% and mobile office work: 93%, relaxing: 86%). Furthermore, we evaluated a person-independent and a person-specific approach for stress detection and found that both strategies show similar trends in accordance with our predictions (person-independent: stress detected in MO-HT: 22%, MO-LT: 18%, MD: 18%, REL: 15%; person-specific: stress detected in MO-HT: 79%, MO-LT: 72%, MD: 65%, and REL: 50%). These results demonstrate the efficacy of using a lightweight semantic approach for activity recognition and stress detection as basis for user-focused vehicle automation.