Digital Twin for Robotic Vision: A Blender-Based Approach

Digital twins are widely used alongside deep learning techniques for robotic vision, offering a safe and cost-effective means for model training and data collection. However, most existing approaches utilize digital twins solely for offline synthetic data generation, which is typically sufficient for training object detection models but may fall short for more complex tasks such as navigation, path planning, obstacle avoidance, and grasping. Additionally, few methods take real-world camera parameters into account, leading to misalignment between synthetic images and real-world camera views. A key challenge in training deep learning models with synthetic data is sim-to-real transfer. Ensuring precise alignment between synthetic and real images not only enhances the realism of synthetic data but also enables automatic annotation of real-world images, further improving sim-to-real performance. In this paper, we present a Blenderbased digital twin for deep learning-driven robotic vision. Our approach supports real-time updates to the virtual environment and incorporates real-world camera parameters, facilitating the creation of datasets that enhance sim-to-real transfer for deep learning models.