A Dual-Modal Human Activity Recognition System Based on Deep Learning and Data Fusion

As the primary sensing device in human activity recognition (HAR), non-intrusive sensors have been widely used in various applications such as gaming, sporting, health care monitoring, and others. However, the existing work mainly utilizes single modality of non-intrusive sensors, which limits the capability of information acquisition, and thus limits the performance of sensor-based HAR. This paper presents a dual-modal, non-intrusive, deep learning enhanced sensing system to improve the performance of sensor-based HAR. This proposed system consists of inertial sensor and Kinect sensor. Each of them is able to acquire activity information from different perspectives, and the combination enlarges data and feature diversities. The local features and temporal dependencies are further extracted by multi-layer convolutional neural network and long short-term memory models. Furthermore, data fusion is implemented at the decision level with different schemes to improve the performance. Experimental results demonstrate the effectiveness and improvement of the proposed system on HAR over other single-modal systems.