A Novel Adaptive Resource Allocation Model Based on SMDP and Reinforcement Learning Algorithm in Vehicular Cloud System

In this paper, we propose a novel adaptive cloud resource allocation model based on Semi-Markov Decision Process (SMDP) and Reinforcement Learning (RL) algorithm in vehicular cloud system. The issue of adaptive resource allocation for vehicular request is formed as an SMDP in order to gain the dynamics of vehicular requests' arrival and departure. An optimized decision is made to guarantee the Quality of Service (QoS) of the vehicular cloud system and the Quality of Experience (QoE) of the vehicular users as well as to maximize the total system reward of the vehicular cloud system in consideration of the balance between the vehicular cloud resource expense and the system income. Furthermore, to capture the mobility feature of the vehicular cloud system, we also apply a neural-network-based RL algorithm to resolve our proposed SMDP-based adaptive cloud resource allocation model. Firstly, we use a Planning algorithm to get the action values under certain state-action pairs, which are the initial samples to train the neural network. Then the RL is used to update the parameters of the neural network, train the neural network and adaptively improve the decision strategy. Subsequently, an adaptive vehicular cloud resource allocation scheme which can approach the optimal strategy is obtained without the knowledge of the distribution function of vehicular requests' arrival and departure during the RL process. The simulation results show that our proposed adaptive cloud resource allocation model for vehicular cloud system can reduce the probability of delay in processing requests and achieve high system rewards in comparison with the regularly used greedy resource allocation method. The performance of the RL solution approaches that of traditional value iteration solution for our proposed adaptive cloud resource allocation model.