RADFormers for Detection and Classification of Hand Gestures

Touchless interfaces and controls have many applications and gained popularity, especially since the onset of the COVID pandemic. Radar sensors enable nonverbal touchless gesture interaction without privacy concerns. Radar can also facilitate low-cost implementation and outperform other sensors in variable lighting and/or occlusion conditions. In this article, we propose a novel deep neural network (DNN) transformer-based hand gesture detection and classification solution using raw radar data (with minimal pre-processing). The solution enables efficient real-time, flexible, and scalable usage. To this end, we develop: 1) an algorithm-guided methodology for radar data annotation to provide accurate and reliable ground truth and 2) a deformable transformer-based algorithm for detection and classification using 3-D point cloud. The input point cloud is created from radar range-doppler (RD) and range-angle information from the complete scene without using a custom constant false alarm rate (CFAR) algorithm. We demonstrated the mean average precision (mAP) of approximately 60% showcasing an improvement in the performance of over 20% over the baseline method, over a set of complex hand shapes with this solution. We also demonstrated the solution converged at a faster rate compared to the baseline which is critical for latency-sensitive gesture usage.