End-to-end speech recognition from raw speech: Multi time-frequency resolution CNN architecture for efficient representation learning

We propose a multi time-frequency (t-f) resolution CNN architecture for end-to-end speech recognition from raw speech waveform. We address issues related to the nature of front-end convolutional kernels and the kind of multi t-f spectrographic feature maps formed and the back-end convolutional processing of the feature maps within two tasks, namely, frame-wise phoneme classification and an encoder-decoder (with CTC-Attention loss) based continuous phoneme decoding. Our multi t-f resolution CNN (MT-CNN) architecture works with unconstrained learnt kernels and back-end 2D-convolutional layers to process the multi t-f spectrographic feature maps for these tasks. We contrast this architecture with two other variants in recent work - a multi-scale feature based system and the SincNet (which uses parameterized convolutional kernels constrained in the form of Sinc functions with learnable bandwidths). We show a consistent performance gain of the proposed multi t-f architecture over these two variants - a 3-8% accuracy (absolute gain) in the frame-wise classification task and 3% PER (absolute gain) in the continuous phoneme decoding task. These two performance gains together establish the effectiveness of the proposed architecture in using the multi t-f unconstrained variable length 1-D convolutional kernels, 2-D multi t-f spectrographic feature maps and the back-end 2-D convolution layers.