Reinforcement Learning Based Online Algorithm for Near-Field Time-Varying IRS Phase Shift Optimization: System Evolution Perspective

This paper proposes a reinforcement learning (RL) based intelligent reflecting surface (IRS) incremental control algorithm for a mmWave time-varying multi-user multiple-input single-output (MU-MISO) system. The research focuses on addressing the key challenge of near-field IRS design, which involves time-varying channels due to users’ mobility. In practice, the optimization becomes more challenging when the components of the concatenated channel are unknown. From a higher perspective, we leverage electromagnetic information theory and manifold theory to provide a unified description of the IRS-assisted MU-MISO system. We regard the communication system as a nonlinear dynamic system on reproducing kernel Hilbert space (RKHS), upon which the approximate evolution operator is defined as observables for system evolution. The IRS phase shift optimization problem is modeled as a nonlinear system eigenvalue maximization problem. Utilizing the geometric properties of the unitary evolution operator, we define a metric space where the geodesic-based distance function satisfies the Lipschitz condition, enabling efficient exploitation of channel similarities. We transform the complex non-convex optimization problem into a low-dimensional linear contextual bandit problem. The performance of the proposed GLinUCB algorithm is evaluated through numerical simulations in various scenarios, showing its effectiveness in achieving high sum rates with fast convergence speed.