Speech Enhancement Based on Masking Approach Considering Speech Quality and Acoustic Confidence for Noisy Speech Recognition

In recent years, voice-operated applications have been widely accepted by the public, while the background noise is still a challenging issue for automatic speech recognition (ASR). This paper proposes a mask-based speech enhancement front-end approach taking into account speech quality score as well as acoustic confidence for speech enhancement to reduce the word error rate (WER) for noisy speech recognition. First, the features of speakers, phones, and noises are extracted and considered in the loss function to improve the speech quality. In addition to speech quality, this study also considers the phone confidence from the Kaldi-based ASR into the loss function for training the mask generation model to improve speech quality as well as improve noisy speech recognition performance. Compared with the baseline model, the proposed model not only improved STOI by 2.14% and PESQ by 7.22%, but also reduced 12.13% in WER.