A Wearable Real-Time 2D/3D Eye-Gaze Interface to Realize Robot Assistance for Quadriplegics

For people with severe sensory-based motor disorders or musculoskeletal disorders, robotic assistance offers a promising solution to improve their daily living standards. In this paper, we present a wearable real-time 2-Dimensional/3-Dimensional (2D/3D) eye-gaze control interface for people with quadriplegia to enable robot-assisted locomotion, sensing, and manipulation. Compared to other modalities in human-robot interaction, gaze point information in Cartesian space has the advantage of being directly feasible for robotic control. To achieve accurate 3D gaze point estimation, we propose a method that leverages a commercially available 2D wearable eye tracker and an off-the-shelf stereo camera. Unlike traditional stereoscopic depth computation or 3D eye reconstruction approaches, which often rely on user-specific eye model calibration, our method reformulates the 2D-to-3D mapping as an online Newton-Raphson search problem that does not depend on individual eye model parameters for gaze depth estimation, allowing the system to operate effectively across varied environments and depth ranges. This results in a solution that is easy to implement, robust to individual variability, and computationally efficient for accurate 3D gaze point estimation. At the same time, 2D gaze is utilized to interact with a screen displaying robotic sensing of specific environments that are not directly visible to the user, thereby enabling extended sensing through robotic assistance. The feasibility of the proposed method is verified through quantitative evaluations of the 3D gaze point estimation. On average, the 3D gaze point estimation yields a mean Euclidean distance error of approximately 1.88 cm across the 0.5–4.0 m distance (corresponding to a 0.9 % percentage error), outperforming comparable methods. A proof-of-concept study further demonstrates successful robot-assisted locomotion, sensing, and manipulation using the proposed gaze interface. The implementation of the core method is available at https://github.com/SavickTso/ros2-3d-gaze-mapping.git.