An Automated VNF Manager based on Parameterized Action MDP and Reinforcement Learning

Managing and orchestrating the behaviour of virtual network functions (VNFs) remains a major challenge due to their heterogeneity and the ever-increasing resource demands of the served flows. In this paper, we propose a novel VNF manager (VNFM) architecture to automate the process of selecting appropriate VNF management actions (e.g., migration and vertical and horizontal scaling) with their corresponding configuration parameters (e.g., migration location or amount of resources needed for scaling). More precisely, we first propose a novel Markov decision process with parameterized actions to accurately describe each VNF and its permissible lifecycle management (LCM) operations. The use of parameterized actions allows us to rigorously represent the functionalities of the VNFM in order perform various operations on the VNFs. Next, we propose a two-stage reinforcement learning (RL) scheme that alternates between learning optimal LCM actions and updating their parameters selection policy. In contrast to existing schemes, the proposed work uniquely provides a holistic management platform that unifies individual efforts targeting single LCM functions such as VNF placement and scaling. Performance evaluation results demonstrate the efficiency of the proposed VNFM in maintaining the required performance level of the VNF while optimizing its resource configurations.