FMCW Radar Range Profile and Micro-Doppler Signature Fusion for Improved Traffic Signaling Motion Classification

Human activity recognition plays a crucial role in Advanced Driver-Assisted Systems (ADAS). A significant challenge in achieving automotive autonomy lies in the difficulty faced by self-driving cars in navigating roads without operational traffic lights. In such scenarios, human intervention often involves directing vehicles through signaling with appropriate signs or gestures. This poses a considerable challenge for autonomous vehicles to interpret these gestures effectively. This study focuses on leveraging a dataset of traffic signaling motions obtained through millimeter-wave (mmWave) radar, a technology commonly used in the US traffic system. In this paper, we developed multimodal convolutional neural networks (CNN), considering both data fusion and feature fusion of radar range profiles and micro-doppler
$(\mu-\mathbf{D})$
signatures, and compared the results with Unimodal CNNs on range profile and
$\mu-\mathbf{D}$
signatures. The findings indicate that the fusion-based CNNs outperform unimodal CNNs based on individual radar range profiles and
$\mu- \mathbf{D}$
signatures by about 10% and 6.5% respectively. Specifically, the accuracy achieved through multimodal CNNs reached around 92% and 96% for data-level and feature-level fusion respectively, showcasing the effectiveness of combining information from both modalities in enhancing human gesture recognition in traffic-directing scenarios.