Gaze orientation to evaluate object interaction in human functional assessment

The paper goal is to identify what and where each person, in a RGB image, is looking at, and correctly handle the case where the gaze target is out-of-frame. Human-Gaze-Target (HGT) projects mainly focus on “Where the person is looking at” on video-surveillance RGB images, but fail to exploit “What the person is looking at” in such context. On this document, we present a video-processing workflow that takes that step, fusing an accurate HGT algorithm with an object detection one, and enhancing the resulting system for the application of interest. Concretely, we introduce an algorithm called “Looking at whose function is transforming the HGT output into a shape comparable to that of the object detection standard algorithms. We also propose the inclusion of a preprocessing layer into the object detection algorithm, in order to increase its confidence on detecting small items, typically used in functional assessment evaluation tasks (such as silverware), that is the application of interest of this proposal. Finally, we generate and use a novel dataset for validating the global workflow proposed, that has been recorded in a specially adapted kitchen for the assessment task, the EYEFUL dataset, obtaining promising quantitative and qualitative results.