Generating Realistic Sound with Prosthetic Hand: A Reinforcement Learning Approach

In this study, we tackle the complex task of enabling prosthetic hands to accurately reproduce sounds, a crucial aspect for distinguishing between different materials through auditory feedback. Sound identification, such as discerning a drywall tap from that on a brick wall, significantly enhances the functionality and user experience of prosthetic devices. However, achieving this level of auditory feedback in prosthetic hands poses considerable challenges. We utilize reinforcement learning (RL) techniques to train prosthetic hands in emulating human-like sound characteristics, focusing on key auditory signals like amplitude and onset timing. Our approach integrates a detailed analysis of these sound attributes to direct the prosthetic hand’s movements for the sound generation that mimics natural human actions. We developed a tailored reward function incorporating amplitude, onset strength, and timing criteria to ensure the prosthetic hand’s movements align closely with the intended human-like sound output.