Privacy-Preserving and Robust Multimodal Monitoring System Using LWIR and mmWave for Abnormal Behavior Recognition in Hospital Wards

In this paper, we propose a multimodal monitoring sensing system of LWIR camera and mmWave radar for detecting abnormal health behaviors in hospital wards. LWIR cameras can recognize posture with high accuracy while preserving privacy with low resolution. However, it cannot be used in an invisible environment, such as closing curtains in a ward environment. To solve this problem, we intend to solve it using mmWave radar that can be sensed through a thin cloth. We design and implement a multimodal system that uses two sensors for privacy and environmental robustness. For proper use of both sensors, we propose a multimodal black-out network called COMBONet, which switches inputs based on confidence, considering background features and circumstances. Our implementation and experiments used people not used for training as the evaluation. It shows that it can be used comprehensively in various environments.