User association and power allocation for UAV-assisted networks: A distributed reinforcement learning approach

Unmanned aerial vehicles (UAVs) can be employed as aerial base stations (BSs) due to their high mobility and flexible deployment. This paper focuses on a UAV-assisted wireless network, where users can be scheduled to get access to either an aerial BS or a terrestrial BS for uplink transmission. In contrast to state-of-the-art designs focusing on the instantaneous cost of the network, this paper aims at minimizing the long-term average transmit power consumed by the users by dynamically optimizing user association and power allocation in each time slot. Such a joint user association scheduling and power allocation problem can be formulated as a Markov decision process (MDP). Unfortunately, solving such an MDP problem with the conventional relative value iteration (RVI) can suffer from the curses of dimensionality, in the presence of a large number of users. As a countermeasure, we propose a distributed RVI algorithm to reduce the dimension of the MDP problem, such that the original problem can be decoupled into multiple solvable small-scale MDP problems. Simulation results reveal that the proposed algorithm can yield lower long-term average transmit power consumption than both the conventional RVI algorithm and a baseline algorithm with myopic policies.