Zygos: A Reward Correction Mechanism for Reinforcement Learning-based Congestion Control

Network feedback, representing the impact of congestion control actions on the network, is crucial for evaluating the advantages of the actions taken. Previous reinforcement learning (RL)-based congestion control algorithms use average performances over fixed periods to measure network feedback, which fails to accurately capture the impact of an action and leads to performance degradation. In this paper, we propose Zygos, which accurately estimates network feedback. This accurate feedback can benefit other RL-based congestion control algorithms. Zygos contains a distribution-based reward correction mechanism that leverages a RL model to generate relevance distributions for the sequence rewards of each state-action pair, and then aggregates the rewards by weighted average. Zygos also adopts metagradient RL to capture network feedback offset patterns, thereby updating the relevance generation model during the training of the congestion control algorithm. Experiments show that the RL congestion control method using Zygos achieves an average 20–30% improvement in throughput and 20% decrease in delay compared to the original method, highlighting substantial enhancements in RL-based congestion control algorithms.