Vision Transformer based Audio Classification using Patch-level Feature Fusion

Recently, transformer based model has been widely employed for audio classification due to its capability of modelling long feature dependencies in audio representations such as spectrograms and Mel-Frequency Cepstral Coefficients (MFCC). However, the transformer model focuses on learning attention coefficients between features of each patch and using it only is incapable of identifying the informative patches in the audio representations and obtain the corresponding patch information of an audio event. In this paper, in addition to fuse different features of each patch using a vision transformer (ViT) encoder, a patch-level feature fusion scheme is proposed to fuse the feature information learned from another ViT encoder. In essence, an audio representation is fed into a ViT to generate a feature map such that the merit of the network pre-trained on a large amount of data can be exploited. Rather than outputting the audio class directly, the feature map is reconstructed to a number of patches, and a transformer is then applied to learn the patch-level information. The performance of the proposed approach using the network weights from the pre-trained model of ImageNet and AudioSet is extensively studied. Several tasks including environment sound classification (Dataset ESC-50), speech command recognition (Dataset SCV2), and crowd-sourced emotional multimodal actors (Dataset CREMA-D) are considered. The results show that our propose method outperforms existing ViT based methods on all these tasks.