Reinforcement Learning for Resource Allocation in Time-Sensitive System under Outdated CSI

In this paper, we investigate the timeliness performance of a time-sensitive system via Age of Information (AoI). Due to the timeliness requirement, the transmission date is carried by finite blocklength codes (FBL) and the decoding error probability can not be negligible. To achieve the requirement of reliability, we consider Hybrid automatic repeat request with incremental redundancy (HARQ-IR) (re)transmission protocol. Nonetheless, while (re)transmission can enhance reliability, it can also decrease timeliness, especially in the scenario where packet dates arrive randomly. Then, we formulate the transmit power allocation problem to minimize the average AoI, i.e., improve the timeliness performance of the considered system. Under the outdated channel state information (CSI) which leads to a mismatch between the power allocation and the channel, a twin delayed deep deterministic policy gradient (TD3) algorithm is proposed with the multi-step bootstrapping (MSB) to search the power allocation policy. Via simulations, we evaluate the performance of average AoI and demonstrate that our proposed algorithm outperforms existing methods in terms of stability and convergence speed which conclude a set of guidelines for system designs.