Probabilistic Multi-modal People Tracker and Monocular Pointing Pose Estimator for Visual Instruction of Mobile Robot Assistants

In this paper, we present two important aspects of our human-robot communication interface which is being developed in the context of our long-term research framework PERSES dealing with the development of highly interactive mobile robotic assistants. First, we introduce a multi-modal people detection and tracking system, a fundamental prerequisite for the observation of a human interaction partner and his nonverbal instructions given by pointing poses, gestures, head pose and eye gaze. Based on this detection and tracking system, we present a hierarchical neural architecture that is capable of estimating a target point at the floor given a pointing pose, thus enabling a user to command his mobile robot to a specific target position in his local surroundings by means of pointing. In this context, we were especially interested in determining whether it is possible to accomplish such a target point estimator using only monocular images of low-cost cameras. Both the tracker and the target point estimator were implemented and experimentally investigated on our mobile robotic assistant HOROS. The achieved recognition results presented finally demonstrate that it is in fact possible to realize a user-independent pointing pose estimation using monocular images only, but further efforts are necessary to improve the robustness of this approach for everyday application.