Constrained Probabilistic Routing with RouterRL: A General Packet-Level Network Simulation Framework

Routing optimization remains challenging in traffic engineering as increasing traffic in dynamic networks often leads to congestion and resource wastage. To address it, probabilistic routing strategies based on deep reinforcement learning (DRL) with finer-grained routing control and autonomous learning capabilities are considered promising solutions. However, current DRL-based probabilistic routing approaches are limited by the lack of constraints for routing safety and cross-comparisons with other routing strategies. This paper proposes a constrained multi-agent DRL-based probabilistic routing approach, which employs a novel actor-critic architecture called JointGAT based on graph attention networks and is trained using clipped proximal policy optimization. Besides, this paper proposes a novel probabilistic routing protocol for constraints to ensure routing safety. For validation and comparison, an open-source general packet-level network simulation framework called RouterRL is designed and implemented based on the knowledge-defined networking architecture, supporting cross-comparisons of different routing strategies. We compare the performance of the proposed approach with various benchmark approaches using different routing strategies. Extensive simulation results across various topologies and loads demonstrate that the proposed approach significantly outperforms benchmark approaches regarding network quality of service and robustness.