Adaptation of Artificial Neural Networks Avoiding Catastrophic Forgetting

In connectionist learning, one relevant problem is "catastrophic forgetting" that may occur when a network, trained with a large set of patterns, has to learn new input patterns, or has to be adapted to a different environment. The risk of catastrophic forgetting is particularly high when a network is adapted with new data that do not adequately represent the knowledge included in the original training data. Two original solutions are proposed to reduce the risk that the network focuses on new data only, loosing its generalization capability. The first one, conservative training, is a variant to the target assignment policy, while the second approach, support vector rehearsal, selects from the training set the patterns that lay near the borders of the classes not included in the adaptation set. These patterns are used as sentinels that try to keep unchanged the original boundaries of these classes. Moreover, we investigated the extension of the classical approach consisting in applying linear transformations not only to the input features, but also to the outputs of the internal layers. The motivation is that the outputs of an internal layer represent a projection of the input pattern into a space where it should be easier to learn the classification or transformation expected at the output of the network. We illustrate the problems using an artificial test-bed, and apply our techniques to a set of adaptation tasks in the domain of automatic speech recognition (ASR) based on artificial neural networks. Supervised ASR adaptation experiments with several corpora and for different adaptation types are described. We report on the adaptation potential of different techniques, and on the generalization capability of the adapted networks. The results show that the combination of the proposed approaches mitigates the catastrophic forgetting effects, and always outperforms the use of the classical transformations in the feature space.