Learning Tone: Towards Robotic Xylophone Mastery

Audio information plays an important role in various robotic manipulation tasks, such as pouring and music performance, as the produced audio can serve as an informative indicator for evaluating actions. However, it is rarely explored in reinforcement learning methods. Due to the unique nature of audio information, it is challenging to simulate in a simulator or use it as direct feedback. Therefore, in this paper, we propose a reinforcement learning method based on audio feedback, aiming to train a dexterous hand to play the xylophone in the real world. By optimizing the dexterous hand’s actions using the produced audio, we can make the characteristics of the audio-—such as amplitude, waveform shape, and timing—similar to human performance.