Progressive Neural Network-based Knowledge Transfer in Acoustic Models

This paper presents a novel deep neural network architecture for transfer learning in acoustic models. A well-known approach for transfer leaning is using target domain data to fine-tune a pre-trained model with source model. The model is trained so as to raise its performance in the target domain. However, this approach may not fully utilize the knowledge of the pre-trained model because the pre-trained knowledge is forgotten when the target domain is updated. To solve this problem, we propose a new architecture based on progressive neural networks (PNN) that can transfer knowledge; it does not forget and can well utilize pre-trained knowledge. In addition, we introduce an enhanced PNN that uses feature augmentation to better leverage pre-trained knowledge. The proposed architecture is challenged in experiments on three different recorded Japanese speech recognition tasks (one source and two target domain tasks). In a comparison with various transfer learning approaches, our proposal achieves the lowest error rate in the target tasks.