AI and AR Based Interface for Piano Training

Multimedia teaching patterns for piano performance have gained great momentum benefiting from the rapid development of virtual reality (VR)/augmented reality (AR) technologies in recent years. Considerable efforts are made to achieve high-fidelity and fluency 4D visualization for the actions of hands. However, it is still challenging to effectively generate demonstration animation for large batches of music, especially for complex music requiring professional playing skills. We propose an AR-based individual tutorial system for piano training which is competent in automatically generating hand action animations and displaying animations with real pianos using head-mounted displays. Given a piece of music as input, our system first predicts the correspondence between target keys and fingers using a Hidden Markov model. Based on the prediction results, we employ musical prior knowledge to establish a generation mechanism for hand actions that aims to coordinate arm and finger motions taking naturalness and convenience into consideration.