Intelligent Dynamic Spectrum Access Using Deep Reinforcement Learning for VANETs

In vehicular ad hoc networks (VANETs), vehicles can communicate with other vehicles or devices through vehicle-to-X communication. However, with the rise of the Internet of Things, it is a real challenge to make the limited spectrum resource meet the increasing communication demands, i.e., the channel contention problem. To solve this problem, this paper proposes a strategy combining multi-hop forwarding through vehicles and dynamic spectrum access. Firstly, a group-based multi-hop broadcast protocol, G-hop, is proposed. G-hop classifies vehicles with similar characteristics such as moving speed and communication distance into same groups using the depth-first-search algorithm. Messages are forwarded within a group in priority and then across groups, which limits both the range and the number of relay vehicles, i.e., channel contenders. Further, we adopt deep reinforcement learning techniques to achieve dynamic spectrum access. We design a Global Optimization algorithm based on Experience Accumulation (GOEA) using deep reinforcement learning. In GOEA, a network structure combined with recurrent neural network and deep Q-network is proposed for learning the time-varying process, and then a reward method is applied to optimize the global utility. Vehicles that need to transmit messages select channels dynamically following the guidance of GOEA. The experimental results demonstrate that the G-hop protocol reduces the packet loss rate from 0.8 to about 0.1. Meanwhile, compared with Slotted Aloha and DQN, our GOEA algorithm reduces the collision probability and channel idle probability by 60%. Moreover, as the number of vehicles increases, the transmission success rate can be improved by 20%.