Seg-CURL: Segmented Contrastive Unsupervised Reinforcement Learning for Sim-to-Real in Visual Robotic Manipulation

Training image-based reinforcement learning (RL) agents are sample-inefficient, limiting their effectiveness in real-world manipulation tasks. Sim2Real, which involves training in simulations and transferring to the real world, effectively reduces the dependence on real data. However, the performance of the transferred agent degrades due to the visual difference between the two environments. This research presents a low-cost segmentation-driven unsupervised RL framework (Seg-CURL) to solve the Sim2Real problem. We transform the input RGB views to the proposed semantic segmentation-based canonical domain. Our method incorporates two levels of Sim2Real: task-level Sim2Real, which transfers the RL agent to the real world, and observation-level Sim2Real, which transfers the simulated U-nets to segment real-world scenes. Specifically, we first train contrastive unsupervised RL(CURL) with segmented images in the simulation environment. Next, we employ two U-Nets to segment robotic hand-view and side-view images during real robot control. These U-Net are pre-trained with synthetic RGB and segmentation masks in the simulation environment and fine-tuned with only 20 real images. We evaluate the robustness of the proposed framework in both simulation and real environments. Seg-CURL is robust to the texture, lighting, shadow, and camera position gap. Finally, our algorithm is tested on a real Baxter robot with a dark hand-view in the cube lifting task with a success rate of 16/20 in zero-shot transfer.