Loading [a11y]/accessibility-menu.js
MusCaps: Generating Captions for Music Audio | IEEE Conference Publication | IEEE Xplore

MusCaps: Generating Captions for Music Audio


Abstract:

Content-based music information retrieval has seen rapid progress with the adoption of deep learning. Current approaches to high-level music description typically make us...Show More

Abstract:

Content-based music information retrieval has seen rapid progress with the adoption of deep learning. Current approaches to high-level music description typically make use of classification models, such as in auto-tagging or genre and mood classification. In this work, we propose to address music description via audio captioning, defined as the task of generating a natural language description of music audio content in a human-like manner. To this end, we present the first music audio captioning model, MusCaps, consisting of an encoder-decoder with temporal attention. Our method combines convolutional and recurrent neural network architectures to jointly process audio-text inputs through a multimodal encoder and leverages pretraining on audio data to obtain representations that effectively capture and summarise musical features in the input. Evaluation of the generated captions through automatic metrics shows that our method outperforms a baseline designed for non-music audio captioning. Through an ablation study, we unveil that this performance boost can be mainly attributed to pre-training of the audio encoder, while other design choices - modality fusion, decoding strategy and the use of attention - contribute only marginally. Our model represents a shift away from classification-based music description and combines tasks requiring both auditory and linguistic understanding to bridge the semantic gap in music information retrieval11Code available at https://github.com/ilaria-manco/muscaps.
Date of Conference: 18-22 July 2021
Date Added to IEEE Xplore: 20 September 2021
ISBN Information:

ISSN Information:

Conference Location: Shenzhen, China

Funding Agency:


Contact IEEE to Subscribe

References

References is not available for this document.