A 3D Point-of-Intention Estimation Method Using Multimodal Fusion of Hand Pointing, Eye Gaze and Depth Sensing for Collaborative Robots

Hand pointing psychologically expressing intention has been fused with eye gaze to assist in detecting the point of intention (POI). Ideally, a POI detection approach using a pair of hand-pointing and eye-gaze rays may achieve precise POI positioning in 3-dimensional (3D) space; however, an object sometimes occludes the detection process in real applications. This paper proposes a method of 3D POI estimation using multimodal fusion of hand pointing, eye gaze and depth sensing for collaborative robots. In this method, depth sensors are used to assist in the fusion of the original hand-pointing and eye-gaze rays to determine the POI based on a volume of interest (VOI) to find 3D POI. At real production sites where humans work with collaborative robots, workpieces may sometimes occlude the depth sensors, resulting in blind views. This problem is solved by increasing the number of depth sensors and arranging them with overlapping view ranges to cover otherwise blind views due to occlusion. The number of depth sensors is determined based on the largest possible obstacle size determined by users in advance. Experiments performed by 10 participants confirm the effectiveness of POI estimation in the presence of obstacles; specifically, with our proposed method, we measured 3D POIs at 50 cm, 70 cm, 75 cm, 80 cm, 85 cm, 90 cm, and 95 cm with average distance errors of 0.82%, 0.68%, 0.68%, 0.72%, 0.76%, 0.87%, and 0.81%, respectively, compared with the conventional POI estimation method based on the fusion of hand pointing and eye gaze.