Spectro-Temporal Modeling for Human Activity Recognition Using a Radar Sensor Network

Radar-based human activity recognition is attracting a wide range of interest from both industry and academia because of its through-wall ability, privacy-preserving capability, and device-free detection. Currently, most radar-based systems consider signal analysis and feature extraction in the frequency domain or the temporal domain independently without fusing them together. In this article, in order to model both frequency properties and temporal profiles of human activity, we proposed a spectro-temporal network (STnet) that integrates a temporal convolutional network (TCN) and a convolutional neural network (CNN). It can extract temporal patterns and micro-Doppler features from radar signals for human activity recognition. In the experiments, two radar sensors and one base station were used to build a low-power wireless radar sensor network. Fifteen activities were investigated in a real kitchen scenario by using this radar sensor network. Frequency spectrograms were obtained after signal processing using a short-time Fourier transform (STFT). They were further segmented using a short sliding window (2.5 s), which enables a very small latency. The proposed STnet achieved 99.64% overall accuracy (OA) in testing, which is superior to the other three networks that we implemented in this work. Our work also can be used as a generic solution to other sensor-based (wearable sensors, WiFi channel state information (CSI), etc.) activity recognition.