Lyapunov Optimization-Based Latency-Bounded Allocation Using Deep Deterministic Policy Gradient for 11ax Spatial Reuse

With the growing demand for wireless local area network (WLAN) applications that require low latency, orthogonal frequency-division multiple access (OFDMA) has been adopted for uplink and downlink transmissions in the IEEE 802.11ax standard to improve the spectrum efficiency and reduce the latency. In IEEE 802.11ax WLANs, OFDMA resource allocation that guarantees latency, called latency-bounded resource allocation, is more challenging than that in cellular networks because severe unmanaged interference from overlapping basic service sets is enhanced due to the concurrent-transmission mechanism newly employed in IEEE 802.11ax. To improve the downlink OFDMA resource allocation with the unmanaged interference caused by IEEE 802.11ax concurrent transmissions, we propose Lyapunov optimization-based latency-bounded allocation with reinforcement learning (RL). We focus on the transmission-queue size for each station (STA) at the access point that determines the STA latency. Using Lyapunov optimization, we formulate the resource-allocation problem with the queue-size constraints in a form that can be solved using RL (i.e., a Markov decision process) and prove the upper bound of the queue size. Our simulation results demonstrated that the proposed method, which uses an RL algorithm with a deep deterministic policy gradient, satisfied the queue-size constraints. This means that the proposed method met the latency requirements, while some baseline methods failed to meet them. Furthermore, the proposed method achieved a higher fairness index than the baseline methods.