Handle Dense Labeling in Human Activity Recognition Using Self Attention and BiLSTM

Dense labeling, which annotates an activity label for each data sample in the segment, is a common approach to handle the problem of multi-class windows in wearable sensor-based human activity recognition. Recent success in image-based semantic segmentation offers opportunities to solve this problem by using well-known fully convolutional neural networks. However, the long-term dependencies of human activity are often ignored in these works, thus, lead to unstable prediction results. In this study, we address this problem by proposing a hybrid deep learning system that can effectively extract the context information from the sequential sensor data and densely predict the activity for each data sample. The system is constructed from two main components: 1) a transformer encoder with multi-head self-attention modules that capture the relationship between data samples and extract the salient features from long sequential data, 2) a bidirectional long short-term memory (BiLSTM) maintains the long-term temporal information in human activity. Our experiments on the UCI HAPT public dataset indicate that the proposed hybrid model achieves an accuracy of 93.41%, which is 2% higher compared to other state-of-the-art image-based dense labeling HAR models.