Offline Reinforcement Learning for Asynchronous Task Offloading in Mobile Edge Computing

Edge servers, which are located in close proximity to mobile users, have become key components for providing augmented computation and bandwidth. As the resources of edge servers are limited and shared, it is critical for the decentralized mobile users to determine the amount of offloaded workload, to avoid competition or waste of the public resources at the edge servers. Reinforcement learning (RL) methods, which are sequential and model-free, have been widely considered as a promising approach. However, directly deploying RL in edge computing remains elusive, since arbitrary exploration in real online environments often leads to poor user experience. To avoid the costly interactions, in this paper, we propose an offline RL framework which can be optimized by using a static offline dataset only. In essence, our method first trains a supervised offline model to simulate the edge computing environment dynamics, and then optimize the offloading policy in the offline environment with cost-free interactions. As the offloading requests are mostly asynchronous, we adopt a mean-field approach that treats all neighboring users as a single agent. The problem can then be simplified and reduced to a game between only two players. Moreover, we limit the length of the offline model rollout to ensure the simulated trajectories are accurate, so that the trained offloading policies can be generalized to unseen online environments. Theoretical analyses are conducted to validate the accuracy and convergence of our algorithm. In the experiments, we first train the offline simulation environment with a real historical data set, and then optimize the offloading policy in this environment model. The results show that our algorithm can converge very fast during training. In the execution, the algorithm still achieves high performance in the online environment.