Multi-Agent DQN with Sample-Efficient Updates for Large Inter-Slice Orchestration Problems

Data-driven network slicing has been recently explored as a major driver for beyond 5G networks. Nevertheless, we are still a long way before such solutions are practically applicable in real problems. Reinforcement learning based solutions, addressing the problem of dynamically placing virtual network function chains on top of a physical topology, have to deal with astronomically high action spaces (especially in in multi-VNF, multi-domain, and multi-slice setups). Moreover, their training is not particularly data-efficient, which can pose shortcomings, given the scarce(r) availability of cellular network related data. Multi-agent DQN can reduce the action space complexity by many orders of magnitude compared to standard DQN. Nevertheless, these algorithms are data-hungry and convergence can still be slow. To this end, in this work we introduce two additional mechanisms on top of (multi-agent) DQN to speed up training. These mechanisms intelligently decide how to store to, and how to pick from the experience replay buffer, in order to achieve more efficient parameter updates (faster learning). The convergence speed gains of the proposed scheme are validated using real traffic data.