Sensor-Based Occluded Face-Part Reconstruction: Eye Tracking and Facial Expressions

Tracking facial expressions while wearing a head-mounted display is challenging as relevant features are occluded by the device. For example, camera-based sensors are limited in their ability to access the entire upper facial region due to occlusion and distortion of this specific region by the headset. To overcome these limitations, this research explores the potential of using biosignals from facial muscles to reconstruct eye movements and facial expressions, particularly in the context of virtual reality headsets that occlude the upper facial region. Two different approaches for eye tracking are explored, while machine learning is used to track facial expressions. For this purpose data is collected for different facial expressions (seven basic emotions plus neutral, and grimace). As a traditional approach, a classification model is trained to predict one of these facial expressions. In this work, a novel approach is proposed to train a regression model to predict the coefficient values of ARKit-compatible blendshape. Both these models are then used to reconstruct the facial expressions by setting the blendshape values of the 3D avatar generated using Ready Player Me. Our approach demonstrates how to successfully track eye movements while simultaneously predicting and reconstructing facial expressions in real-time on partially occluded faces.