SKP: Semantic 3D Keypoint Detection for Category-Level Robotic Manipulation

As human-assisting robots are starting to play an important role in the food service and hospitality field, service vision for robot manipulation on intra-category objects, which have variations in shape, size, and appearance, is an inevitable and valuable task. Previous works typically rely on estimating the pose of the objects, such methods are not sufficient to generalize well to category-level. Hence, we propose Semantic Keypoint Detection Network (SKP) for category-level robotic manipulation tasks, that 1) uses semantic keypoint representation to expand the perception module to category level and drive the entire grasping pipeline to handle shape variations and dynamic environment, 2) leverages the underlying geometric constraints via jointly optimize instance-aware features with the self-selection mechanism, 3) utilizes multi-scale multi-layer features for 3D keypoint position regression and keypoint association and 4) train with synthetic data only with domain randomization. Extensive experiments on synthetic and real-world datasets show that our work improves accuracy for category-level keypoint detection and demonstrates the generalizability and practicality in robotics grasping tasks.