MPTFormer: Toward Robust Arm Gesture Pose Tracking Using Dual-View Radar System

Locating human joints using low-cost commercial millimeter-wave (mmWave) radars is an emerging technology. Compared to single-view radar, dual-view radar fusion is more efficient for robust arm gesture pose tracking since it can capture a wider area of human reflection and radial velocity in both directions. One challenging task for dual-view radar fusion is to effectively utilize the reflection points’ spatial and velocity information and focus perceptual cues related to joint position and velocity for accurate joint tracking. In this article, we propose a millimeter-wave pose tracking transformer (MPTFormer), a method for arm gesture pose tracking using both front- and side-view mmWave radars. Specifically, MPTFormer consists of a multidomain fusion transformer (MFFormer) and a joint tracking transformer (JTFormer). In the MFFormer, joint position and velocity estimation tasks are set, and the range–azimuth map (RAM) and range–Doppler map (RDM) of the dual-view radar are set as token embeddings, which interacted with joint position and velocity tokens for learning perceptual cues related to joint position and velocity as well as joint constraint relationships. Meanwhile, the nonlocal dependencies between dual-view radar and its multidomain features are mined to obtain good global feature representation. In the JTFormer, a hybrid learning architecture of the temporal transformer and long short-term memory (LSTM) is designed, which tracks the joint position according to the global and local features of the joint in the temporal dimension. Finally, in the pose-tracking task containing 12 different arm gestures, the proposed method can achieve 25.2 mm mean per joint position error (MPJPE) and 19.1 mm Procrustes analysis-MPJPE (P-MPJPE).