Vision-Based Swallowing Phase Recognition for Adaptive Robotic Feeding Assistance: A Closed-Loop Control Framework

Assistive feeding robots face a critical challenge in balancing task efficiency with user-centric adaptation, as existing systems often neglect the dynamic synchronization between robotic actions and human physiological responses. This paper proposes a vision-based swallowing detection framework integrated with closed-loop robotic control to enable adaptive feeding timing for individuals with motor impairments. Our system employs a multi-modal sensing approach: (1) YOLOv5-based food detection with depth perception for precise utensil positioning, (2) MediaPipe Face Mesh for real-time mouth state tracking, and (3) SORT-enhanced marker tracking to capture thyroid cartilage motion during swallowing phases. By correlating laryngeal elevation patterns with the swallowing reflex, the robot further refines its algorithm for understanding human feeding intentions, enabling better recognition of the eater’s needs and optimization of feeding timing. Experimental results demonstrated a 167.6% efficiency improvement after implementing swallowing detection. This enhancement significantly improves the robot’s performance in assistive feeding tasks. By bridging the critical gap between robotic automation and human biomechanical feedback, this work provides a clinically viable solution for personalized care.