Complex Human Activity Recognition Based on Spatial LSTM and Deep Residual Convolutional Network Using Wearable Motion Sensors

Human activity recognition (HAR) is an important task for health monitoring and lifelog applications. HAR employs wearable sensors and can be a low-cost and effective solution. With the increasing demand for HAR in recent years, greater attention must be paid to complex and high-level semantic behaviors. To this end, we suggest describing complex human activity in terms of subaction time combination and subaction frequency decomposition. We propose a deep learning framework named “convolution and spatial long-short-term memory (CSLSTM)” for complex HAR. CSLSTM consists of a downsample module and a spatial distribution encoding module. After processing raw sensor data (accelerometer, gyroscope, and magnetometer) into time–frequency spectrograms, the downsample module uses the residual convolution block to downsample and extract local texture features of frequency responses from the spectrogram. The spatial distribution encoding module, an LSTM network in the 2-D time–frequency domain, mines the time–frequency distribution of frequency responses. In this framework, LSTM is no longer used to process time series data, but to process 2-D time–frequency spectrogram data. We evaluate the performance of CSLSTM on two widely used datasets: ExtraSensory and PAMAP2. Experimental results demonstrate that the
$F1$
-score of CSLSTM reaches 79.45% and 99.22% on the ExtraSensory and PAMAP2 datasets, respectively.