A Comparative Analysis of Transformer Models for Bird Song Recognition Using Mel-Spectrograms

This study explores the application of Vision Transformer (ViT), Data-efficient Image Transformer (DeiT), and Swin Transformer models for bird species classification using mel-spectrogram representations of bird songs. The research utilized the BirdClef 2024 dataset, consisting of 24,641 audio recordings across 182 bird species. The dataset was processed into mel-spectrograms and split into training, validation, and test sets to evaluate model performance. Each model was trained over 75 epochs with early stopping mechanisms in place. The results indicated strong training accuracy for all three models; however, validation accuracy plateaued between 78% and 84%, revealing overfitting issues. To address these challenges, various optimizations were implemented, including advanced learning rate scheduling, targeted dropout rates, weight decay adjustments, and data augmentation techniques. Despite these modifications, some overfitting persisted, particularly in the DeiT model, which showed decreased test accuracy after optimization. These findings highlight the effectiveness of transformer-based models for audio-based bird species classification, while also underscoring the need for further regularization techniques and fine-tuning. Future work should focus on exploring additional transformer architectures, refining regularization methods, and utilizing larger datasets to enhance model generalization and accuracy in complex classification tasks.