Data-Driven, 3-D Classification of Person-Object Relationships and Semantic Context Clustering for Robotics and AI Applications

We introduce a framework for detection and classification of spatio-temporal person-object interactions. Our method clusters similar semantic contexts from interactions detected from RGB-D data. 2-D object detection (YOLO) is run on RGB data from a Kinect v2 sensor on a mobile robot navigating an office and observing persons and desk spaces. Person and object detections are converted into 3-D point cloud time series via RGB-Depth co-registration and successive Euclidean and k-means spatial clustering. 3-D person and object point cloud streams are used to create time-series occupancy maps and person-object co-localization maps. From these maps, spatiotemporal correlations between persons and distinct objects are computed. Correlation patterns are clustered using k-means to obtain distinct human-object interactions, i.e. segment semantic context over time. We evaluated the performance of our approach to detect person-object correlations and cluster semantic context by recording 90 30-second RGB-D data episodes, with three persons handling representative objects (books, cups, bottles). Experimental results show that our framework is able to consistently assign semantic context to the same cluster in > 79% of cases (scene frames). Semantic contexts in visual scenes can be distinguished without the need to provide prior information, allowing mobile agents to learn and explore in new environments.