Birds of a Feather: Learning to Retrieve Dance Poses From Music Via Ground-Truth Annotation Lifting

Learning to retrieve dance poses from music, a cross-modal retrieval task, has gained prominence in assisting choreographers in creating dances that harmonize with music. The recent predominant approach is to map music into 3D pose and shape space, and then match it with dance poses [1]. However, we found that the mainstream choreography dataset lacks discriminative power in terms of 3D pose and shape annotations across different dance genres, hindering the model’s ability to learn effective mappings, which in turn reduces retrieval performance. To address the issue, we propose LiftNet, a deep-net model that uses dance genres as guidance to lift 3D pose and shape annotations, making them more discriminative and easier for the downstream retrieval model to learn. Experimental results demonstrate that using the lifted annotations from our LiftNet as new learning targets substantially enhances the performance of all existing cross-modal music-to-dance pose retrieval models.