Distributed DRL Based Beamforming Design for RIS-Assisted Multi-Cell Systems

Reconfigurable intelligent surface (RIS) has the potential to significantly enhance the performance of communication systems by dynamically adjusting wireless environment. In this paper, we aim to jointly optimize RIS matrices and beam-forming at base stations (BSs) in RIS-assisted multi-cell systems. Considering the high complexity of traditional algorithms, we adopt deep reinforcement learning algorithms (DRL) for the difficult joint design task. Moreover, to overcome the performance drawbacks of centralized DRL algorithms with large state-action spaces, we propose distributed DRL algorithms based on the distributed distributional deterministic policy gradient (D4PG) method and the federated learning (FL) framework. Experimental results demonstrate that distributed DRL algorithms can considerably improve learning efficiency while reducing complexity and communication overhead. Moreover, it can be applied to various systems with satisfactory performance and has better robustness compared to the centralized DRL algorithms.