BiCLR: Radar–Camera-Based Cross-Modal Bi-Contrastive Learning for Human Motion Recognition

Radar-based human motion recognition is gaining attention due to its inherent resistance to lighting conditions, especially in healthcare and safety applications concerning personal privacy. In this article, we propose a novel cross-modal bi-contrastive learning model named “BiCLR,” which utilizes a Transformer-based network for temporal modeling to conduct instance discrimination in both single- and cross-modal modalities in a self-supervised learning manner. To enhance data density, a new radar data format, the “radar combination map (RCM),” is presented to seamlessly integrate range-Doppler map (RDM), range-azimuth map (RAM), and range-elevation map (REM) into a single map. The objective of this article is to address the inherent sparsity of radar data through cross-modality and newly introduced RCM, offering a transferable framework for various kinds of downstream tasks, advancing understanding through radar-based recognition. After a comprehensive evaluation, the pretrained encoder demonstrates effectiveness in a new human motion recognition task using only radar data, despite being trained on a significantly smaller dataset. The experimental results clearly demonstrate BiCLR’s capability to utilize cross-modal and contrastive learning methods, as well as the improved performance in downstream tasks.