Efficient Initial Access with Deep Reinforcement Learning Based Beam Sweeping in Wireless Cellular Communication Systems

Initial access (IA) is a procedure of establishing an initial connection between the base station (BS) and the user. In the fifth generation (5G) millimeter wave (mmWave) communication system, the IA procedure includes beam management, which determines the beam pair for random access and data transmission by beam sweeping. The existing beam sweeping method in the 3-rd generation partnership project (3GPP) standard mainly uses a predefined uniform beamforming codebook and sweeps the beams progressively, which is time-consuming and highly inflexible. Note that the non-uniform and quasi-stationary environment and user cluster distribution information can be exploited for the BS to optimize the beam sweeping patterns. Therefore, this paper proposes a novel reinforcement learning (RL) framework for IA to efficiently acquire beam sweeping patterns. Specifically, a dimension-reduced beamforming codebook is designed to solve the problem of large search space and a comprehensive RL environment is constructed for the BS to capture the properties of environment layout and user distributions. Simulation results verify the remarkable performance of our proposed schemes in terms of beam sweeping efficiency.