Detecting 3D Hand Pointing Direction from RGB-D Data in Wide-Ranging HRI Scenarios

This paper addresses the detection of 3D hand pointing direction from RGB-D data by a mobile robot. Considering ubiquitous forms of pointing gestures, the 3D pointing direction is assumed to be inferable from hand data only. First, a novel sequential network-based learning model is developed for the simultaneous detection of hands and humans in RGB data. Differing from previous work, its performance is shown to be both accurate and fast. Following, a new geometric method for estimating the 3D pointing direction from depth data of the detected hand is presented along with a mathematical analysis of sensor noise sensitivity. Two new data sets for pointing gesture classification and continuous 3D pointing direction with varying proximity, arm pose and background are presented. As there are no such data sets to the best of our knowledge, both will be publicly available. Differing from previous work, the robot is able to estimate the 3D hand direction both accurately and fast regardless of hand proximity, background variability or the detectability of specific human parts - as demonstrated by end-to-end experimental results.