Delay-Aware NFV Resource Allocation with Deep Reinforcement Learning

Network Function Virtualization (NFV) can support flexible services provisioning in form of Service Function Chains (SFCs) consisting of ordered Virtual Network Functions (VNFs). The end-to-end (E2E) delay of flow traversing SFC has been an important indicator, especially for delay-sensitive E2E services, but there does not exist an analytical model that can accurately evaluate it. Moreover, the complicated network and stochastic request arrival are hard to predict and model. Therefore, quantitative delay model and dynamic NFV resource allocation method are needed. In this paper, an adaptive allocation method is designed to meet E2E delay requirements. Firstly, we devise an NFV resource allocation framework based on deep reinforcement learning (DRL) that can adapt to network changes by interacting with the network. Then a dynamic queuing model is established to determine average E2E packet delay. Based on the delay, we define the network utility function and propose a minimizing delay (MD) problem. According to the continuity of the problem, we use unsupervised reinforcement and auxiliary learning (UNREAL) to obtain the optimal allocation policy. At last, extensive simulation results show that UNREAL-MD has better performance compared to state-of-the-art methods in terms of delay, throughput and network utility.