Digital Twin Assisted Deep Reinforcement Learning for Computation Offloading in AAV Systems

The emerging digital twin (DT) technology involves creating a digital clone of the physical world through techniques such as computer simulation and machine learning. This facilitates the management and decision-making of the physical world. In this correspondence, we propose a DT-assisted deep reinforcement learning (DRL) method to optimize the decision of computation offloading in an autonomous aerial vehicles (AAV)-based communication system. Our focus is on minimizing task processing delay by jointly optimizing task offloading and the AAV's flight path. This problem can be effectively addressed using the DRL method. However, it necessitates frequent interactions with the environment, which may lead to increased resource consumption and risks associated with some “bad” actions. To mitigate these challenges, our proposed DT comprises a predictive model to anticipate the reward and next stage of the physical environment. Additionally, a generative model is introduced to enhance sample efficiency, with these two models realized by a fully connected neural network and a variational auto-encoder, respectively. Subsequently, we generate a hybrid experience replay buffer to facilitate DRL training, resulting in faster convergence, improved performance, and fewer environmental interactions. Numerical results demonstrate that our proposed method achieves superior delay performance and training efficiency compared to state-of-the-art methods.