Deep Reinforcement Learning Assisted Beam Tracking and Data Transmission for 5G V2X Networks

Beam tracking is a core issue in 5G vehicle-to-everything (V2X) networks. Specifically, higher beamforming gain is required to compensate for the path loss at higher frequencies, e.g., 5G FR2, to realize high data rate vehicle-to-infrastructure (V2I) communications. However, shorter time slots at higher frequencies, high velocity of vehicles, and unpredictable localization errors make this problem more challenging. Under these circumstances, wider beams can lead to higher beam tracking accuracy. Bear in mind that wider beams mean lower beamforming gain, which cannot compensate for high path loss at high frequencies and would further influence the data rate of V2I communications. Thus, there exists a trade-off between tracking accuracy and data rate in V2I communications. Furthermore, this problem needs to be solved within an extremely short time slot according to the high transmission frequency. To solve this problem, we propose a reinforcement learning (RL) assisted, high-resolution codebook-based beam tracking method. By comparing several different RL frameworks, we found that the twin delayed deep deterministic policy gradient (TD3) framework can help the roadside infrastructure (RSI) determine a proper beam pattern within a short duration. Moreover, according to the Hurst exponent analysis, recurrent neural networks (RNNs) are selected to improve the performance of the RL framework. The simulation results show that the proposed method performs well in tracking accuracy, data rate, and temporal efficiency.