Marker-less piano fingering recognition using sequential depth images

Piano fingering is one of the important skills for piano performance, especially for beginners. Consequently, technology for recognizing a player's fingering is required in order to develop an automated piano lesson system. The term “piano fingering” refers to which fingers are used for pressing piano keys. In this paper, we propose a method for recognizing piano fingering by analyzing motion of multiple fingers of a piano player through the use of depth images continuously acquired with a depth sensor. Our method makes it possible to develop a practical system that does not require the use of any special markers such as color labels on fingers. First, a dictionary data set for various fingering patterns is registered. Each data element consists of a depth image, the name of a pressed key, correct information for its fingering, and the wrist positions of the player in an image. Next, a fingering pattern for unknown depth images is identified by matching acquired images to those in the dictionary data set. To reduce the search space size, the wrist position detected from an input image and a note name signal obtained from a MIDI keyboard are effectively used. The Nearest Neighbor search algorithm is utilized to search for solutions. Experimental results obtained using actual piano pieces for beginners demonstrate that the system achieves an 91.6% recognition rate and that its processing time is less than 120 msec per note.