A heuristic evaluation PDS algorithm for energy-efficient delay constrained scheduling over wireless communication

Hard energy constraints impose huge challenges to wireless communication under the constraint of communication delay. In this paper, we consider the problem of energy-efficient point-to-point transmission scheduling of delay sensitive multimedia data over a fading channel. We first formulate the transmission scheduling problem as a Markov Decision Process (MDP) and systematically unravel it by the optimal solution. We then propose a Heuristic Evaluation Post-decision State (HE-PDS) learning algorithm for the problem of energy-efficient scheduling with respect to the communication delay. This algorithm is based on reformulating the value iteration equation by introducing a virtual state called Post-decision State (PDS). The advantages of the proposed algorithm are that: (i) it exploits only part of the information about the system so that less information needs to be learned than when using conventional reinforcement learning algorithms; (ii) it uses the heuristic function and evaluation function to reduce unnecessary action exploration in the whole search space, which severely limits the adaptation speed and runtime performance of conventional reinforcement learning algorithms; (iii) it chooses actions with favorable purpose and efficiency so as to optimize reward function and enhance convergence speed. We compare our algorithm with the traditional Q learning algorithm and PDS learning algorithm in terms of the energy consumption and transmission delay. The simulation results illustrate the performance of the proposed algorithm under various scenarios achieves a better trade-off between energy consumption and delay overhead. Moreover, it can converge in a reasonable number of slots for it to be practically useful with satisfaction of delay constraint.