Reduction of catastrophic forgetting for multilayer neural networks trained by no-prop algorithm

Neural networks encounter sever fatal forgetting or fatal interference when information is learned sequentially. This is called catastrophic forgetting. One way to reduce catastrophic forgetting is pseudorehearsal, in which pseudopatterns are learned with training patterns. This method has shown excellent performance in multilayered neural networks trained by the backpropagation algorithm. In recent years, a new learning method of multilayered neural networks called the no-propagation algorithm has been proposed. This algorithm is far easier to implement because only the weights of output layer neurons are trained, and convergence is much faster than backpropagation. Nevertheless, its performance is essentially equivalent to that of backpropagation. Our previous research has shown that pseudorehearsal is effective even for networks trained by no-propagation, whereas it is less effective than networks trained by backpropagation. In this paper, we modify the no-propagation algorithm for pseudorehearsal so that it can much reduce catastrophic forgetting. Computer simulation results show that networks trained by the proposed algorithm can much improve the reduction of catastrophic forgetting in comparison with the conventional no-propagation algorithm and becomes comparable to those trained by backpropagation.