Improving accented Mandarin speech recognition by using recurrent neural network based language model adaptation

In this paper, we propose adapt the recurrent neural network (RNN) based language model to improve the performance of multi-accent Mandarin speech recognition. N-gram based language model has already been applied to speech recognition system, but it is hard to describe the long span information in a sentence and arises a serious phenomenon of data sparse. Instead, RNN based language model can overcome these two shortcomings, but it will take a long time to decode directly. Taking these into consideration, this paper combines these two types of language model (LM) together and adapts the RNN based language model to rescore lattices for different accented Mandarin speech. The architecture of the adapted RNN LM is accent-specific top layers and shared hidden layer. The accent-specific top layers are used to adapt different accents and the shared hidden layer stores history information, which can be seen as memory layer. Experiments on the RASC863 corpus show that the proposed method can improve the performance of accented Mandarin speech recognition over the baseline system.