Real-time Criticality Evaluation Through Vision-based Human-centric Emotion, Activity and Object Interactions

In the ever-evolving landscape of computer vision (CV), human-computer interaction has emerged as a ground-breaking method. In dynamic scenarios where instant analysis is important, utilizing vision-based technology to assess video scenes becomes crucial for quick decision-making. The proposed real-time scene evaluation (RSE) framework underscores its significance with a specific focus on human-centric interactions by analyzing face emotions, human activities, and objects. We have employed fine-tuned transfer learning models such as Mobile-Net for face emotion recognition (FER) and Xception for human activity recognition (HAR) tasks, respectively. The Mobile-Net achieved an accuracy of 63% with a ROC score of 88% for FER on the FER-2013 dataset, and the Xception obtained an accuracy of 89% with a 99% ROC score for HAR models using the Stanford-40 dataset. The RSE approach also employed an object detection model (ODM) using YOLO-V3 to address the object diversities in the scene. Additionally, it provides insight for immediate contextual information extraction from real-time video analysis by evaluating the criticality of the scene based on the importance of emotions, activities, objects, and their interaction combinations. The proposed vision-based human-centric RSE method considers the sensitivity, diversity, interaction, and trends in the degree of criticality subject to the occurrence of suspicious events. Our proposed strategy aims to advance industrial CV technology by facilitating real-time scene detection mechanisms.