Deep Q Learning With Fourier Feature Mapping For Mobile Relay Beamforming Networks

The paper addresses motion control of mobile relays implementing cooperative beamforming in a time- and space- varying channel environment. The relays move in a time-slotted fashion and movement is confined within a 2D rectangular plane, discretized on a fine grid. In each time slot, the relays optimally beamform to maximize the Signal-to-Interference+Noise Ratio (SINR) at the destination, subject to power constraints, and determine their optimal next slot positions to which they move by the end of the slot. Prior works have assumed the availability of statistical channel models, which were used to predictively compute the optimal next slot relay positions. In this paper, we propose a novel, model-free, deep Q learning approach to govern relay motion policies, which drops all assumptions on channel model statistics and allows relays to learn solely from experience. Due to the randomness of the channels, the Q function is highly varying with respect to state and action. To facilitate the learning of Q, we propose to apply Fourier mapping of the state with a Gaussian matrix. Via simulations, we show that this approach leads to significant improvement in convergence and SINR performance, as compared to using the state directly.