3D Facial Tracking and User Authentication Through Lightweight Single-Ear Biosensors

Facial landmark tracking and 3D reconstruction have gained considerable attention due to their numerous applications such as human-computer interactions, facial expression analysis, and emotion recognition, etc. Traditional approaches require users to be confined to a particular location and face a camera under constrained recording conditions, which prevents them from being deployed in many application scenarios involving human motions. In this paper, we propose the first single-earpiece lightweight biosensing system, BioFace-3D, that can unobtrusively, continuously, and reliably sense the entire facial movements, track 2D facial landmarks, and further render 3D facial animations. Our single-earpiece biosensing system takes advantage of the cross-modal transfer learning model to transfer the knowledge embodied in a high-grade visual facial landmark detection model to the low-grade biosignal domain. After training, our BioFace-3D can directly perform continuous 3D facial reconstruction from the biosignals, without any visual input. Additionally, by utilizing biosensors, we also showcase the potential for capturing both behavioral aspects, such as facial gestures, and distinctive individual physiological traits, establishing a comprehensive two-factor authentication/identification framework. Extensive experiments involving 16 participants demonstrate that BioFace-3D can accurately track 53 major facial landmarks with only 1.85 mm average error and 3.38% normalized mean error, which is comparable with most state-of-the-art camera-based solutions. Experiments also show that the system can authenticate users with high accuracy (e.g., over 99.8% within two trials for three gestures in series), low false positive rate (e.g., less 0.24%), and is robust to various types of attacks.