Radar-Camera-Based Cross-Modal Bi-Contrastive Learning for Human Motion Recognition

Radar-based human motion recognition (HMR) is gaining increasing attention, primarily due to its robust performance in various lighting conditions, especially in healthcare and safety applications, with a specific emphasis on personal privacy. This paper introduces a novel cross-modal bi-contrastive learning method, named BiCLR. Utilizing a Transformer-based network [1] for temporal modeling, BiCLR excels in discriminating instances across both single-modal and cross-modal settings through self-supervised learning. Additionally, the “Radar Combination Map (RCM)” is proposed to provide a comprehensive representation by seamlessly integrating the Range-Doppler Map (RDM), Range-Azimuth Map (RAM), and Range-Elevation Map (REM) into a unified map. The primary objective of this work is to address the inherent sparsity in radar data through cross-modality and the newly introduced RCM, presenting a transferable framework applicable to various downstream tasks. This contributes to a deeper understanding of radar-based recognition systems. Despite being trained on a smaller dataset, the pre-trained encoder demonstrates remarkable effectiveness in leveraging cross-modal and contrastive learning methods and the newly introduced radar data format in a HMR task using solely radar data, as substantiated by a thorough evaluation.