Q-Delegation: VNF Load Balancing Through Reinforcement Learning-Based Packet Delegation

While Network Function Virtualization (NFV) and Service Function Chaining (SFC) offer unprecedented capabilities to network operators, there are still performance concerns that need to be addressed, especially with the advent of network slicing requiring automated and efficient resource allocation. In this context, sharing the available networking and computational resources is an appealing option to minimize under-utilization, but this can lead to over-subscription of the available resources under heavy traffic loads. To overcome this, distributed load balancing mechanisms, such as packet delegation, can be utilized to move the decision-making further from centralized orchestrators and closer to the Virtual Network Functions (VNF) themselves, allowing them to quickly redirect incoming traffic to other servers when they are overloaded. However, the effectiveness of packet delegation can vary considerably depending on the selection of the target VNF that receives the delegated packets. As such, in this paper, the Q-Delegation algorithm is proposed, which treats this as a non-stationary multi-agent Reinforcement Learning (RL) problem, providing an adaptive way of selecting appropriate delegation targets based on the VNF's current queueing state through Q- Learning. Two variations of the Q-Delegation algorithm are proposed, either allowing an RL agent to act once a VNF's queue has exceeded a static threshold or enabling it to freely learn how to reroute packets to other VNFs regardless of their queue size. Both variations are experimentally evaluated and compared to other baseline algorithms under diverse topologies and traffic rates in a simulation setting, indicating that in most cases both variations of Q- Delegation can be effective at delegating packets and outperforming the baselines, though their effectiveness can vary based on how evenly the traffic is initially distributed.