Guitar Parts Extraction and Keypoint Detection Based on Synthetic Annotation

The recognition of specific instrument components and key points from images and videos is a significant challenge in various music technology applications. To solve this problem, this study proposes the use of deep neural network models trained with synthetic datasets. Currently, convolutional neural networks (CNNs) rely heavily on manually annotated training data for instrument feature estimation, which is time-consuming and impractical for detailed instrument features. To address this, we propose a synthetic dataset approach based on guitar features, utilizing guitar CAD models and static background images for realistic rendering. The synthetic datasets allowed us to train object detection and semantic segmentation models that achieved results comparable to those trained on real datasets. Moreover, we proposed a guitar keypoint detection network and trained it using synthetic datasets, achieving a mAP@0.5 score of 0.935 on the real validation set. These findings offer new ideas and methods for addressing data annotation challenges in instrument pose estimation, supporting applications such as guitar fingerings estimation and automatic sheet music transcription in instrument information retrieval.