Development of Visual and Tactile based Human Behavior Imitation Learning Platform

A novel approach to robot control, inspired by human behavior, has been introduced with the aim of effectively navigating complex and dynamically changing environments while handling a wide variety of objects. To achieve this objective, researchers have introduced the DexMV (Dexterous Manipulation from human Videos) platform [1], which harnesses human demonstration videos within a simulation environment to rapidly amass extensive datasets for learning purposes. Building upon this foundation, this study takes the DexMV platform a step further by enhancing its capabilities to incorporate tactile data obtained from human grasping actions. Moreover, significant efforts have been dedicated to bridging the gap between simulation and real-world implementation inherent in applying imitation learning outcomes to actual manipulation tasks. This paper presents a comprehensive overview of the evolution of the DexMVT (Dexterous Manipulation from human Video and Tactile sensing) platform.