Deep Reinforcement Learning-Based Joint Scheduling and Routing for Time-Sensitive Networks

In the deployment of 5 G and the development of the future 6 G, a set of use cases are generated that require URLLC (Ultra-Reliable Low Latency communications) service-type communications with very low latency and extremely high reliability. At the same time, verticals such as Industry 4.0 require Key Performance Indicators (KPIs) that can only be offered by deterministic networks. Time Sensitive Networking (TSN) is a set of IEEE 802.1 standards that aim to provide highly reliable, low-latency deterministic communications over Ethernet. The IEEE 802.1Qcc standard defines three architectural models for TSN networks. One of them is fully centralized, which is based on an SDN architecture where the control plane is distributed between two entities: the Centralized Network Configuration (CNC) and the Centralized User Configuration (CUC). These systems’ increasing complexity and scalability led to the introduction of Machine Learning (ML) tools that will allow them to move towards zero-touch. Currently, solutions are proposed using Integer Linear Programming (ILP), whose computational complexity implies a significant lack of scalability, which worsens as the network size or traffic heterogeneity increases. This paper describes a traffic scheduling and routing algorithm based on deep reinforcement learning (DRL), which is located in the CNC, receiving requests from CUC users and configuring and maintaining the Gate Control lists of the Ethernet switches. The proposed mechanism jointly solves a deterministic routing of data flows with scheduling. This proposal also optimizes the load balancing of the links and the occupation of the time slots of each link.