Automated Mouth State Recognition for Robotic Feeding Assistance

Feeding poses a significant challenge for patients with limited upper limb mobility. A robotic assistant equipped with the capability to detect a patient’s mouth and deposit food can effectively support the user during feeding. Since each patient exhibits unique physical characteristics, it is essential to develop an automated system capable of accurately identifying the mouth’s location. This allows the definition of a precise three-dimensional target point where a robotic manipulator can deliver food. Although various techniques are available for facial feature detection, some demonstrate notable advantages in specific applications. Considering the constraints of limited processing capacity, we propose the use of Facemesh to identify the patient’s point of interest, specifically the mouth. This technique enables the determination of the precise location for food delivery. To ensure the robot can reach both the patient’s mouth and the food container, an inverse kinematics-based controller is implemented. The system’s performance is evaluated, demonstrating the effectiveness of integrating the patient into the control loop for seamless operation.