Tracking of Wrist and Hand Kinematics with Ultra Low Power Wearable A-mode Ultrasound

Ultrasound-based Hand Gesture Recognition has gained significant attention in recent years. While static gesture recognition has been extensively explored, only a few works have tackled the task of movement regression for real-time tracking, despite its importance for the development of natural and smooth interaction strategies. In this paper, we demonstrate the regression of 3 hand-wrist Degrees of Freedom (DoFs) using a lightweight, A-mode-based, truly wearable US armband featuring four transducers and WULPUS, an ultra-low-power acquisition device. We collect US data, synchronized with an optical motion capture system to establish a ground truth, from 5 subjects. We achieve state-of-the-art performance with an average root-mean-squared-error (RMSE) of 7.32◦ ± 1.97◦ and mean-absolute-error (MAE) of 5.31◦ ± 1.42◦. Additionally, we demonstrate, for the first time, robustness with respect to transducer repositioning between acquisition sessions, achieving an average RMSE value of 11.11◦ ± 4.14◦ and a MAE of 8.46◦ ± 3.58◦. Finally, we deploy our pipeline on a real-time low-power microcontroller, showcasing the first instance of multi-DoF regression based on A-mode US data on an embedded device, with a power consumption lower than 30mW and end-to-end latency of ≈ 80 ms.