Average AoI Minimization in an HARQ-based Status Update System under Random Arrivals

We consider a status update system consisting of one source, one butter-aided transmitter, and one receiver. The source randomly generates status update packets and the transmitter sends the packets to the receiver over an unreliable channel using a hybrid automatic repeat request (HARQ) protocol. The system holds two packets: one packet in the butter, which stores the last generated packet, and one packet currently under service in the transmitter. At each time slot, the transmitter decides whether to stay idle, transmit the last generated packet, or retransmit the packet currently under service. We aim to find the optimal actions at each slot to minimize the average age of information (AoI) of the source under a constraint on the average number of transmissions. We model the problem as a constrained Markov decision process (CMDP) problem and solve it for the known and unknown learning environment as follows. First, we use the Lagrangian approach to transform the CMDP problem to an MDP problem which is solved with the relative value iteration (RVI) for the known environment and with deep Q-learning (DQL) algorithm for the unknown environment. Second, we use the Lyapunov method to transform the CMDP problem to an MDP problem which is solved with DQL algorithm for the unknown environment. Simulation results assess the effectiveness of the proposed approaches.