HARFormer: A Masked Self-Supervised Transformer-Based Model for Human Activity Recognition With Predicting Somatosensory Tokens

In the realm of pervasive smart media technology, human activity recognition (HAR) stands as a critical functionality, particularly for applications in smart homes, virtual reality, and mobile devices. These data can be collected through accelerometers and gyroscopes embedded in smart devices such as smartphones and smartwatches. Traditional HAR algorithms are often highly specialized, being developed for specific datasets and tasks. These methods typically rely on large amounts of high-quality labeled data to achieve optimal performance, but such labeled data is notoriously difficult and expensive to obtain in practice. In contrast, the daily use of smart devices generates a vast amount of unlabeled data as people engage in various activities. If the unlabeled data is properly and effectively utilized, that could significantly advance the development of HAR. In this work, we propose a Transformer-based framework that first converts continuous human activity data into discrete tokens, which shortens the sequence length while preserving local information. We then construct a somatosensory lexicon, where the model facilitates self-supervised learning by predicting masked somatosensory tokens as a proxy task. After learning useful features through unlabeled proxy tasks, the model requires only a small amount of labeled data for fine-tuning to achieve performance comparable to supervised learning. Experiments on three public datasets (UCI HAR, MHEALTH, and UniMiB SHAR) show that our approach can achieve performance comparable to fully labeled supervised learning, even with minimal labeling. This reduces the need for extensive labeling while maintaining high accuracy, providing a more efficient and scalable solution for HAR in the age of widespread smart technology.