Developing an Engagement-Aware System for the Detection of Unfocused Interaction

We introduce a perception system for social robots that is able to detect a person’s engagement in an interaction from nonverbal cues independently of principal user activity. This was achieved by the introduction of a set of proxemics, body posture and attention features relevant for human-human interaction. The features were extracted from RGB-D image data of a single Kinect and utilized to train two separate machine learning models. Multiple system configurations and feature combinations were tested, and their impact on the detection of user engagement evaluated. Combining all features, our perception system reaches an F1-score of 81% when estimating an observed person’s interaction intent through binary classification. Regression of a user’s level of availability deviates from the given ground truth values by 13.27% on average. Finally, a prototype was implemented which is able to simultaneously run both previous estimates in real-time using a shared feature vector. In the following, the proposed system shall be used to design robots whose behavior shows their awareness of user engagement.