Ratio-Based Offloading Optimization for Edge and Vehicular-Fog Federated Systems: A Multi-Agent TD3 Approach

Edge and vehicular-fog architectures enable a two-tier Multi-access Edge Computing (MEC) system, where computing resources are positioned behind base stations and vehicular-fog nodes are placed closer to user equipment (UE), resulting in lower latency compared to cloud alternatives. Vehicular-fog is a computing resource formed by a group of either stationary or mobile vehicles. In densely populated areas like congested intersections or festivals, UEs can generate concentrated hotspot traffic towards nearby MEC servers, potentially straining the access network MEC (AN-MEC) site behind the base station. To address this, a portion of the traffic can be offloaded, either vertically to nearby vehicular-fog nodes or horizontally to neighboring AN-MEC sites. The control plane rapidly determines traffic offloading locations and ratios within seconds, with the goal of minimizing average system latency. Our work proposes a reinforcement learning (RL)-based multi-agent TD3, which is built on top of the Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithm to determine optimal offloading ratios. Evaluation results underscore the remarkable decision-making speed of the multi-agent TD3-based approach, which surpasses the single-agent TD3 and simulated annealing (SA) methods by one and five orders of magnitude, respectively. Notably, the average latency of the multi-agent TD3 is better than that of the single-agent TD3, with only a marginal increase of 2 to 6 ms when compared to SA.