Online learning of power allocation policies in energy harvesting communications

We consider the problem of power allocation over a time-varying channel with an unknown distribution in energy harvesting communication systems. In this problem, the transmitter needs to choose its transmit power based on the amount of stored energy in its battery with the goal of maximizing the average rate obtained over time. We model this problem as a Markov decision process (MDP) with the transmitter as the agent, the battery status as the state, the transmit power as the action and the rate obtained as the reward. The average reward maximization problem over the MDP can be solved by a linear program (LP) that uses the transition probabilities for the state-action pairs and their mean rewards to choose a power allocation policy. Since the rewards associated the state-action pairs are unknown, we propose an online learning algorithm called UCLP that learns these rewards and adapts its policy with time. The UCLP algorithm solves the LP at each time-step to choose its policy using the upper confidence bounds on the rewards. We prove that the reward loss or regret incurred by UCLP is upper bounded by a constant.