TinyssimoRadar: In-Ear Hand Gesture Recognition with Ultra-Low Power mmWave Radars

Smart Internet of Things (IoT) devices are on the rise in popularity, with innovative use cases and applications emerging every year. Including intelligence in these novel systems presents the challenge of integrating interaction and communication in scenarios where traditional interfaces are not viable. Hand Gesture Recognition (HGR) has been proposed as an intuitive Human-Machine Interface, potentially suitable for controlling several classes of devices in the context of the Internet of Things. This paper proposes a low-power in-ear HGR system based on mm-wave radars, efficient spatial and temporal Convolutional Neural Networks and an energy-optimized hardware design. The design is suitable for battery-operated devices, with stringent size and energy constraints, enabling user interaction with wearable devices, but also suitable for home appliances and industrial applications. The proposed machine learning model is characterized thoroughly for robustness and generalization capabilities, achieving 94.9% (single subject) and 86.1% (Leave-One-Out Cross-validation) accuracy on a set of 11+1 gestures with a model size of only 36KiB and inference latency of 32.4ms on a 64MHz Cortex-M33 microcontroller, making it compatible with real-time applications. The system is demonstrated in a fully integrated, miniaturized in-ear device with a full-system average power consumption of 18.4mW, a more than 6x improvement on the current state of the art.