Eyes on You: TinyML-Powered On-Device Face Tracking for Low-Cost, Low-Power, Secure MCU Environments

This paper presents a low-cost, low-power face-tracking system utilizing TinyML, designed for real-time face detection and tracking on an ESP32-CAM board. The system employs two servo motors for pan and tilt adjustments, ensuring that detected faces remain within the camera’s view. Operating entirely on-device, the system prioritizes privacy by eliminating the need for network access. It utilizes a multitask learning model that both classifies the presence of a human face with 96% accuracy and predicts the corresponding bounding box coordinates, achieving an Intersection over Union (IoU) of 0.65 for precise face tracking. Optimized for low-power microcontroller units (MCUs), the model has hardware requirements of 103 KB RAM, 323 KB Flash, and 7.5 million MAC operations, enabling inference times of 850 ms on the ESP32-CAM and 150 ms on the Teensy 4.1. To address the lack of publicly available datasets for single-face tracking in resource-constrained settings, a custom dataset derived from the COCO-Faces and Visual Wake Words datasets was created and published on Zenodo, accessible at https://doi.org/10.5281/zenodo.14474899. To the best of our knowledge, this is the first work demonstrating real-time face tracking in MCU environments, with a detection range of up to 2.5 meters, offering a low-resource footprint and high accuracy. This contribution supports advancements in low-power face-tracking applications for IoT security, interactive displays, and human-computer interaction.