Delay-Sensitive Task Offloading With Edge Caching Through Martingale-Based Deep Reinforcement Learning

In the forthcoming era of 6G networks, delay-sensitive applications for Internet of Things (IoT) are poised to become the prevailing services with ultra-reliable and low-latency (URLLC) requirements. Unlike traditional video caching, IoT-based edge caching faces unique challenges due to diverse data types, update frequencies, and computational needs, requiring integrated storage and computational resource management. To support the more stringent requirements for these innovative applications, mobile edge computing (MEC) is introduced to enhance the service reliability of delay-sensitive applications in the 6G era. However, task offloading, as an indispensable procedure in MEC, would encounter many challenges, such as network jitter and resource insufficiency, possibly leading to unpredictable queuing delays and other negative issues. To ensure reliable services in a dynamical MEC environment, the caching-enabled MEC network has emerged as a novel architecture, placing computing and storage resources in the edge network. In this paper, we investigate the caching-enabled MEC to support reliable task offloading for delay-sensitive applications, with a focus on IoT scenarios. In our system model, we formulate the task process as a two-hop tandem queuing system with limited capacity, including task transmission and computation queues. The Martingale theory is leveraged to analyze the delay violation probability in this system, demonstrating how the offloading and caching decisions affect the end-to-end (E2E) delay. Besides, task offloading and resource allocation policies are integrated to reduce high system costs, including energy consumption and cache resource rental costs. Based on the delay analysis of martingale theory, we propose an advanced deep reinforcement learning (DRL) algorithm called Dynamic Request Aware Soft Actor-Critic (DRA-SAC) algorithm to achieve minimal system costs by obtaining the optimal task offloading and resource allocation policies, including caching and computation resources. We conduct some illustrative studies to evaluate the proposed scheme. The algorithm we have put forward outperforms benchmark algorithms regarding both cache hit ratio and system cost.