Hand-Object Interaction Detection based on Visual Attention for Independent Rehabilitation Support

Hand rehabilitation in post-stroke patients with visual impairment is currently not supported by the availability of a cyber-physical-social system (CPSS) that can monitor physical development during daily activities. This paper discusses how to extract hand activity information on objects based on visual attention in the task-specific reach-to-grasp cycle. We used perception-based egocentric vision to observe hand-object interactions (HOI) in grasping tasks. Our approach combines object detection with hand skeletal model estimation and visual attention to validate HOI detection. We choose a multilayer Gated Recurrent Unit (GRU) based on Recurrent Neural Networks (RNN) architecture to classify the four main activities when the hand interacts with an object (wonder-reach-grasp-release). We evaluated the algorithm quantitatively on the new dataset we introduced for cup grasping activity. This method can validate the HOI detection with 97.0
^%
precision with less training time for small data. Further research will use these results to increase self-efficacy for independent hand-eye coordination rehabilitation support in community-centric systems. The code and dataset are available at https://github.com/anom-tmu/hoi-attention/.