A Subjective Evaluation Dataset for GuitarSet

In this paper, we create and evaluate a dataset of subjective evaluation for acoustic guitar performances. The study uses audio clips divided into four-measure segments from the GuitarSet. As evaluation criteria, we adopt ASR (Acoustic Satisfaction Rating), which assesses four items on a five-point scale: discomfort of the sound, variation in picking intensity, onset deviation from the metronome, and offset deviation from the metronome. We conducted an online subjective evaluation experiment with 84 crowd workers to calculate the Mean Opinion Score (MOS). After data screening, the ICC3K exceeded 0.50, demonstrating the dataset’s reliability. For validation, we tested the dataset’s suitability for a deep learning approach by using a Transformer model that takes four-measure guitar performance audio as input and outputs MOS. In the evaluation experiment, the model exhibited low SROCC and PLCC values. These results indicate that human perceptual evaluation of guitar performances is highly complex and challenging to model.