Transfer Learning With Less Negative Transfer for Multi-Agent Reinforcement Learning: Application and Evaluation in Base Station Control

Deep multi-agent reinforcement learning (DMARL) can derive the optimal control rules for systems in which multiple agents interact with each other and has been used in various fields. Since DMARL requires a long learning time to learn the optimal rules, it is important to improve the learning efficiency to apply DMARL to real-world problems. Transfer learning is a common approach to reduce learning times of deep learning models. However, in transfer learning for deep reinforcement learning including DMARL, negative transfer, a phenomenon in which the learning efficiency worsens even though transfer learning is applied, may occur when the degree of similarity between tasks is low. We propose a stable transfer learning method for reducing the learning time of DMARL by preventing negative transfer. For the transfer learning in which parameters of deep learning models are reused, the proposed method eliminates the two causes of negative transfer: (i) dead neurons, i.e., neurons in a neural network that fail to activate, and (ii) biased action probabilities of transferred agents. By applying a new parameter reusing method to avoid dead neurons in addition to the conventional method to promote agent exploration, it is possible to apply transfer learning without negative transfer. We apply the proposed method to two multi-agent control tasks including base station sleeping control and show that the method can stably reduce the learning time regardless of the type of agent model and model architecture.