Hierarchical Multi-Agent Deep Reinforcement Learning for Backscatter-aided Data Offloading

In this paper, we consider a hybrid computation offloading scheme that allows edge users to offload workloads to the edge servers by using active RF communications and backscatter communications. We aim to maximize the overall energy efficiency by jointly optimizing the beamforming of access point (AP) and the users’ offloading decisions. Considering a dynamic environment, we propose a hierarchical multi-agent deep reinforcement learning (H-MADRL) framework to solve this problem. The high-level agent resides in the AP and optimizes the beamforming strategy, while the low-level user agents learn and adapt individuals’ offloading strategies. To further improve the learning efficiency, we propose a novel optimization-driven learning algorithm that allows the AP to estimate the low-level users’ actions by solving an approximate problem efficiently. Then, the action estimation can be shared with all users and drive them to update individuals’ actions independently. Simulation results reveal that our algorithm can improve the reward performance by 50%. The learning efficiency and reliability are also enhanced comparing to the conventional model-free learning methods.