SoHAM: A Sound-Based Human Activity Monitoring Framework for Home Service Robots

Monitoring daily activities is essential for home service robots to take care of the older adults who live alone in their homes. In this article, we proposed a sound-based human activity monitoring (SoHAM) framework by recognizing sound events in a home environment. First, the method of context-aware sound event recognition (CoSER) is developed, which uses contextual information to disambiguate sound events. The locational context of sound events is estimated by fusing the data from the distributed passive infrared (PIR) sensors deployed in the home. A two-level dynamic Bayesian network (DBN) is used to model the intratemporal and intertemporal constraints between the context and the sound events. Second, dynamic sliding time window-based human action recognition (DTW-HaR) is developed to estimate active sound event segments with their labels and durations, then infer actions and their durations. Finally, a conditional random field (CRF) model is proposed to predict human activities based on the recognized action, location, and time. We conducted experiments in our robot-integrated smart home (RiSH) testbed to evaluate the proposed framework. The obtained results show the effectiveness and accuracy of CoSER, action recognition, and human activity monitoring.
Note to Practitioners
—This article is motivated by the goal to develop companion robots that can assist older adults living alone. Among many capabilities, monitoring human daily activities is an essential one for such robots. Though computer vision or wearable sensors-based methods have been developed by other researchers, they are not practical due to the privacy concern and intrusiveness. Sound-based daily activity recognition can address these concerns and offer a viable solution. In this regard, our proposed method adopts microphones on the robot and a small set of motion sensors distributed in the home. The proposed theoretical framework was tested in a small-scale mock-up apartment with promising results. Before such companion robots can be deployed to real homes for elderly care, there is a need to improve the robustness of the algorithms. More thorough tests in various realistic home environments should be conducted to fully evaluate the performance of the robots. In addition, privacy concern related to audio capture should be further mitigated.