Dynamic Antenna Control for HAPS Using Mean Field Reinforcement Learning in Multi-Cell Configuration

In this paper, we propose a Mean Field reinforcement learning (MFRL) method for dynamic antenna control in High-Altitude-Platform-Station (HAPS) communication system with Multi-Cell Configuration. HAPS works at stratospheric altitudes of about 20 km to provide an ultra-wide coverage area. However, the wind pressure caused HAPS movement leads to the degradation of users' throughput. Considering the multi-antenna arrays in the HAPS, to find the optimal antenna parameters of all antenna arrays for reducing the number of low-throughput users, we formulate the antenna control problem into stochastic game equilibrium. Usually, solving the stochastic to find the equilibrium needs very high computation complexity to calculate the transition probability for getting the
$\mathcal{Q}$
-value under a certain state and action. Therefore, we use the reinforcement learning (RL) named Deep
$\mathcal{Q}$
-Network (DQN) to learn the transition probability and predict the Q-value according to the reward fed backed from the environment. Besides, we employ the Mean field Game theory in conjunction with RL during the training phase of DQN to reduce the complexity of the interactions among agents. To evaluate the proposed method, we compare the proposed method with a genetic algorithm (GA) named Particle Swarm Optimization (PSO),
$\mathcal{Q}$
-learning, Fuzzy
$\mathcal{Q}$
-learning, and conventional DQN under four realistic user distribution scenarios. The simulation results show that the proposed method achieves comparable throughput performance with a high convergence rate.