Multitimescale Control and Communications With Deep Reinforcement Learning—Part II: Control-Aware Radio Resource Allocation

In Part I of this two-part paper (Multitimescale Control and Communications with deep reinforcement learning (DRL)—Part I: Communication-Aware Vehicle Control), we decomposed the multitimescale control and communications (MTCCs) problem in cellular vehicle-to-everything (C-V2X) system into a communication-aware DRL-based platoon control (PC) subproblem and a control-aware DRL-based radio resource allocation (RRA) subproblem. We focused on the PC subproblem and proposed the MTCC-PC algorithm to learn an optimal PC policy given an RRA policy. In this article (Part II), we first focus on the RRA subproblem in MTCC assuming a PC policy is given, and propose the MTCC-RRA algorithm to learn the RRA policy. Specifically, we incorporate the PC advantage function in the RRA reward function, which quantifies the amount of PC performance degradation caused by observation delay. Moreover, we augment the state space of RRA with PC action history for a more well-informed RRA policy. In addition, we utilize reward shaping and reward backpropagation prioritized experience replay (RBPER) techniques to efficiently tackle the multiagent and sparse reward problems, respectively. Finally, a sample- and computational-efficient training approach is proposed to jointly learn the PC and RRA policies in an iterative process. In order to verify the effectiveness of the proposed MTCC algorithm, we performed experiments using real driving data for the leading vehicle, where the performance of MTCC is compared with those of the baseline DRL algorithms.