Acoustic Model Combination Incorporated With Mask-Based Multi-Channel Source Separation for Automatic Speech Recognition

In this paper, we propose an acoustic model combination (AMC) technique for reducing a mismatch between training and testing conditions of an automatic speech recognition (ASR) system in a multi-channel noisy environment. In our previous work, we proposed a hidden Markov model (HMM)-based mask estimation method for multi-channel source separation using two microphones, where HMMs were adopted for mask estimation in order to incorporate an observation that the mask information should be correlated over contiguous analysis frames. However, it was observed that a certain degree of noise still remained in the separated speech source especially under low signal-to-noise ratio (SNR) conditions. This was because the estimated mask was not ideal, which resulted in limiting the improvement of ASR performance. To mitigate this problem, the remaining noise can be further compensated in the acoustic model domain under a framework of parallel model combination (PMC). In particular, a noise model and a weighting factor for the proposed AMC can be estimated from the remaining noise and the average of the relative magnitude of the mask, respectively. It is shown from the experiments that an ASR system employing the proposed AMC technique achieves a relative average word error rate (WER) reduction of 56.91%, when compared to a system using the mask-based source separation alone. In addition, compared to a conventional PMC implemented with a log-normal approximation, the proposed AMC relatively reduces WER by 43.64%.