Distributed Multi-Agent Reinforcement Learning for Heterogeneous NOMA-ALOHA Systems

With ever-growing machine type users in the 6G wireless ecosystems, uncontrolled multiple access control (MAC) is vital to alleviate random collision and fading in their transmission. In this paper, 2-steps random access method is applied for a learning-aided non-orthogonal random access (NORA) system. Specifically, each user independently selects a slot and a power level for uplink packet transmission without any information about other users’ selection and channel state information (CSI); and the base station (BS) performs successive interference cancellation (SIC) to decode packets from multiple users with the use of power differences on the same slot. To design a model-free multiple access under growing complexity and CSI uncertainty, the joint slot and power level selecting problem is modelled as a Markov decision process (MDP) where actions are slot-power pairs. Multi-state Q-Learning algorithms and a confidence-aided Q-Learning method are tailored for the NORA system to solve the MDP under heterogeneous environments. Simulation results show that the three proposed algorithms help the distributed users to find their strategies for slot and power level selections, improving system throughput and fairness simultaneously. The proposed algorithms are particularly shown to make superior performance compared to the benchmarks in high congestion traffics scenarios. This is crucial for achieving massive connectivity in 6G ecosystems, which requires intelligent random access designs to accommodate the growing number of machine type users in diverse conditions.