Towards DRL-based Routing and Spectrum Assignment in Optical Networks: Lessons to be Learned from Markov Decision Processes

Monte Carlo and temporal-difference control, Q-Learning and Deep Reinforcement Learning (DRL) are well-established methods to solve problems modelled as Markov Decision Processes (MDP). Within these type of problems, the connection admission control (CAC) and routing and spectrum assignment (RSA) problem is of particular interest in optical networks. Recent research has applied DRL to solve the RSA problem. This work, however, does not completely disclose the strengths and weaknesses of DRL-based RSA since published results mainly present performance comparisons against k-shortest path methods, which are known to be outperformed by exact and heuristic optimization strategies. Furthermore, these methods overlook the state representations and the definition of deterministic policy from the MDP network model. To fill this gap, in this paper we show that by modelling resource allocation as an MDP process, it is possible to unveil properties that can be regarded in the design of approximate solution methods such as DRL. We formulate both the reward maximization problem for CAC & RSA and the definition of the concept of deterministic policy. By applying a policy iteration algorithm, we illustrate that optimum decisions on CAC & RSA depend on the traffic load and on the bandwidth of connections. We also provide a set of guidelines that can be applied for the design of adaptive DRL-based CAC & RSA.