Efficient Alignment of Unconditioned Action Prior for Language-Conditioned Pick and Place in Clutter

We study the task of language-conditioned pick and place in clutter, where a robot should grasp a target object in open clutter and move it to a specified place. Some approaches learn end-to-end policies with features from vision foundation models, requiring large datasets. Others combine foundation models in a zero-shot setting, suffering from cascading errors. In addition, they primarily leverage vision and language foundation models, focusing less on action priors. In this paper, we aim to develop an effective policy by integrating foundation priors from vision, language, and action. We propose A2, an action prior alignment method that aligns unconditioned action priors with 3D vision-language priors by learning one attention layer. The alignment formulation enables our policy to train with less data and preserve zero-shot generalization capabilities. We show that a shared policy for both pick and place actions enhances the performance for each task, and introduce a policy adaptation scheme to accommodate the multi-modal nature of actions. Extensive experiments in simulation and the real-world show that our policy achieves higher task success rates with fewer steps for both pick and place tasks in clutter, effectively generalizing to unseen objects and language instructions. Videos and codes are available at https://xukechun.github.io/papers/A2 Note to Practitioners—This research is motivated by the challenge of generalizable policy learning of language-conditioned pick and place in clutter. Solving such a challenge could significantly improve the robot’s level of automation and intelligence in household and industrial pick and place tasks. Existing methods struggle with large data requirements, poor generalization to unseen scenarios, and cascading errors across individual components. To overcome these limitations, we propose to integrate priors from vision, language, and action foundation models by learning-based alignment. Our policy aligns action priors with 3D vision-language priors by learning one attention layer, requiring less data and preserving zero-shot generalization capabilities from foundation models. Experiments show that our method can improve both task success rate and generalization for pick and place tasks in simulation and the real world. In future work, we will incorporate more action foundation models to extend our approach of action prior alignment to a wider range of tasks, offering a promising direction for general manipulation.