Strumming in the Metaverse: A Deep-Learning-Enabled Virtual Air Guitar System in VR With Enhanced Chord Recognition and Simulated Pedal Effects

Virtual reality (VR) is increasingly capable and inexpensive, and VR devices have become indispensable in many domains, such as gaming, videoconferencing, education, and healthcare. VR has also been applied to music performance and learning. Virtual instruments, such as virtual pianos and drums, free users from the need to own physical forms of these (often bulky and expensive) instruments. VR devices enable users to enjoy music anytime and anywhere without constraints. Virtual concerts, including spatial audio simulations and reconstructions of historical performances, are becoming increasingly common. Previous studies have primarily examined virtual guitars in non-VR environments and air guitar chord recognition. However, systematic research on virtual air guitar systems in VR remains scarce. Virtual guitar games that are available on the market cannot recognize hand gestures accurately and thus cannot accurately identify the strumming patterns and chords played by the player. To overcome this problem, we propose a VR-based virtual air guitar system that can recognize 30 chords and various strumming techniques through deep learning and visual feedback. Employing a black-box approach, we combine WaveNet and FiLM to simulate electric guitar pedal effects with a knob difference loss mechanism, which simulates the turning of knobs on a guitar effects pedal, for enhanced accuracy.