Improvisation Ensemble Support Systems for Music Beginners Based on Body Motion Tracking

Melody recognition consists of three cognitive elements: pitch contour, rhythm, and tonality. Pitch contour and rhythm can be relatively easily represented by body motion. In comparison, tonality is difficult to understand and to represent for music beginners. In this paper, we develop two systems that support improvisational ensembles for music beginners on the basis of body motion tracking. The first system is implemented by using a 3D motion capture camera. The second is implemented by using sensors in a smartphone. Users of our systems can participate in improvisational ensembles by making hand movements that correspond to the pitch contour and rhythm without considering tonality because the generated pitch is automatically adjusted to a consonant pitch with the chords of a background tune. To deal with the delay in gesture recognition by using the 3D motion capture camera, we improved a method for recognizing gestures. The experimental results show that the delay of our method was improved over that of the conventional one. Furthermore, we implemented a method for motion tracking on the basis of smartphone sensors. The experimental results show the difficulties of motion tracking with smartphone sensors. Moreover, we discuss perspectives on social reuse of improvisational melody data shared it as open data.