Imitation Learning of Robotic Arm with Hierarchical Training Based on Human Videos

Human videos encapsulate a diverse spectrum of operational behaviors, serving as invaluable pedagogical resources for imitation learning. In this paper, we introduce an algorithmic framework aimed at emulating expert operational skills gleaned from human videos. The proposed algorithm computes the positional movements of individual steps within a robotic arm by quantifying the disparities between successive frames of the videos, which enables the emulation of skills exhibited in the demonstrations. In detail, we begin by utilizing image translation to seamlessly transition human demonstrations into the realm of the robotic arm, mitigating morphological differences across the multi-joint mechanical structure. Subsequently, we incorporate a keypoint detection network to extract structured information from the transformed images, generating a latent representation that encapsulates motion details for the learning process of the policy network. Experimental results affirm the efficacy of our approach in replicating operational skills.