Insights of Neural Representations in Multi-Banded and Multi-Channel Convolutional Transformers for End-to-End ASR

End-to-End automatic speech recognition (ASR) models aim to learn generalised representations of speech. Popular approaches for End-to-End solutions have involved utilising extremely large amounts of data and large models to im-prove recognition performance. However, it is not clear if these models are generalising the training data or memorising the data. This paper combines the power of a mixture of experts (MoE) models, which is referred to as multi-band, multi-channel, with a popular model for ASR, the CNN-transformer, to capture longer-term dependencies without increasing the computational complexity of training. The goal is to investigate how the transformer models adapt to these different input representations of the same data. No external language models were used to remove the impact of external language models during inference. Although the proposed multi-band transformer shows performance gain, the main finding of this paper is to show the adaptive memo-risation nature of transformers and the neural representations of transformer embedding. Using the statistical correlation index SVCCA, comparative discussion of the neural repre-sentations of the proposed model and transformer approach is provided, with key insights into the distinct learned structures.