Lang2Pose: Language-Based Visual Servoing and Pose Control for Autonomous Pick-and-Place in Real and Simulated Environments

We present Lang2Pose, a modular robot control system that interprets natural language commands to perform either end-effector control or pick-and-place actions. Built upon ROS 2, our framework integrates a large language model (LLM), a perception module leveraging FoundationPose for 6D object pose estimation, and motion planning components using Lula IK for simulation and MoveIt 2 for real-world execution. The system transforms user commands into structured actions, estimates object poses using RGB-D input and segmentation masks, and calculates grasp configurations based on object geometry and orientation. We validate the approach in both simulated and physical environments using a Kinova Gen3 arm and a Robotiq 2F-85 gripper. Simulated trials were conducted in Isaac Sim with high-fidelity physics tuning, while real-world experiments employed fine-tuned YOLO/YOLO-seg models and Realsense RGB-D data. Lang2Pose successfully enables intuitive language-guided manipulation and demonstrates robust performance under partial occlusion. These results highlight the system’s potential as a flexible interface for natural human-robot interaction in both virtual and physical domains.