Learn-Gen-Plan: Bridging the Gap Between Vision Language Models and Real-World Long-Horizon Dexterous Manipulations

Long-horizon dexterous tasks have been a long-standing problem in robotic manipulation. Previous studies have developed task and motion planning, imitation learning, and reinforcement learning methods for long-horizon manipulations. However, these methods are hard to achieve efficient planning for new tasks. Empowered with the Vision Language Model (VLM), recent studies significantly improve the generalization of robot systems. However, these works are only verified in simple pick-and-place tasks due to limited skills. To this end, we propose the Learn-Gen-Plan (LGP), which combines the VLM and learning-based primitives to endow robots with the ability to efficiently plan and complete various long-horizon dexterous tasks. LGP contains two key phases: skill generation and task planning. In skill generation, the Skill Generator is proposed to utilize the learned key primitives and hand-crafted trivial primitives to generate adaptive robot skills. In task planning, the Multimodal Planner generates the robot plan based on image observation, generated skills, and text prompts. We set up a series of dexterous tasks (e.g., cable routing, peg-in-hole assembly) in a real-world lighting circuit wiring scenario to evaluate LGP. The experimental results show that LGP efficiently generates robot plans with learned skills, controlling the robot to complete various multi-step cable wiring tasks. Note to Practitioners—The motivation of this paper is to tackle the challenge of multi-step robotic manipulation tasks, where robots must utilize various skills (e.g., grasping, inserting) to manipulate different objects (e.g., pegs, wires) multiple times to achieve the task goal. Existing approaches either generate robot plans inefficiently for new tasks or support only limited skills (e.g., pick-and-place), leading to poor performance in real-world scenarios. We present a novel approach LGP, which leverages a Vision Language Model (VLM, e.g., ChatGPT-4o) and data-driven robot skills to enable robots to plan and adaptively execute multi-step tasks efficiently. LGP has two key stages: skill generation and task planning. In skill generation, we develop a ChatGPT-based generator that utilizes data-driven key and hand-crafted trivial motions to create adaptive robot skills. In task planning, a ChatGPT-based Multimodal Planner generates the robot’s plan using image observations, generated skills, and human instructions. Real-world cable wiring experiments demonstrate the feasibility of this approach in robotic assembly. Our approach is particularly beneficial for reducing the deployment costs of robotic systems in small and medium-sized enterprises where production tasks frequently change. Future research will focus on developing on-device models to improve control frequency and productivity in more complex tasks.