Multi-task joint-learning of deep neural networks for robust speech recognition

Although deep neural networks (DNNs) have achieved great success in automatic speech recognition (ASR), significant performance degradation still exists in noisy environments. In this paper, a novel multi-task joint-learning framework is proposed to address the noise robustness for speech recognition. The architecture integrates two different DNNs, including the regressive denoising DNN and the discriminative recognition DNN, into a complete multi-task structure and all the parameters can be optimized in a real joint-learning mode just from the beginning in model training. In addition, the basic multi-task structure is further explored and reorganized into a more general framework which can get substantial gains. Furthermore, noise adaptive training can also be easily incorporated within this architecture to achieve further performance improvement. Experiments on the Aurora4 task showed that the proposed approach can achieve a WER below 10% without using adaptation or sequence training, a very large and significant (more than 20% relative) improvement over a strong DNN-HMM baseline.