Air Piano Interaction via Flex Sensors and Computer Vision on Edge Devices

This paper presents the development of a portable and interactive musical instrument using edge devices and various sensors. The goal is to create a versatile device that allows users to play different notes and melodies in a user-friendly manner with a graphical user interface. Two methodologies have been presented, one with flex sensors and the other using computer vision on edge devices. The first method employs an Arduino UNO board, flex sensors, an ultrasonic sensor, and an infrared sensor. Through the integration of these components, this paper enables real-time recognition of the user’s hand orientation and generates corresponding musical output. The Arduino board is programmed to capture sensor data, including flex sensor movements, ultrasonic sensor distance measurements, and IR sensor signals. The second method uses computer vision instead of flex sensors to capture user input on edge devices. Both methods create an interactive musical instrument that promotes creativity and learning. The average accuracy achieved by the flex sensor method is 83.15% with a latency of 505 milliseconds and the accuracy of the computer vision method is 95.7% and its latency in Jetson Nano is 134 milliseconds. The resulting device holds importance as an educational tool for musicians.