Improving Classification of Marine Mammal Vocalizations Using Vision Transformers and Phase-Related Features

In this study, we investigate the relative performance of Vision Transformers (ViTs) compared to convolution-based neural networks in categorizing vocalizations from a medium-sized marine mammal dataset. Additionally, we evaluate whether phase information derived from Fourier decomposition can serve as a complementary source of useful information to magnitude for classification tasks. Our study focuses on bioacoustics, utilizing the publicly available Watkins Marine Mammal Sound Database, which contains sound clips identified as originating from 32 marine mammal species. In this framework, we first trained convolution-based networks (ResNet-101, MobileNetV3) and Transformer-based networks (ViT B-16, Swin Transformer V2) on log-magnitude spectrograms (baseline models). In a second set of experiments, we incorporated the derivative of unwrapped phase from the Fourier representation into the magnitude spectrograms. Our results show that ($a$) Shifted Window (Swin) Transformers outperform MobileNets and achieve performance similar to ResNets while maintaining lower computational complexity and (b) the inclusion of phase derivatives into spectrograms leads to (i) consistently improved performance metrics across all biosignal categories for Swin Transformers and (ii) enhanced classification ability for both convolution-based and self-attention-based networks, particularly for the narrow-band frequency modulated (FM) whistles emitted by delphinids,