Seamless data delivery for distributed AI workloads via dynamic queue scheduling and FPGA-based implementation in converged optical-electrical networks

The deployment of universal AI training jobs in high-performance computing centers poses significant challenges to network architectures. Networks designed for artificial intelligence (AI) have emerged as a prevalent trend in datacenter networks. Traditional electrical packet-switching (EPS) networks face capacity limits due to the slowdown of Moore’s law and struggle to accommodate the specific traffic patterns of distributed deep learning (DDL). In contrast, optical circuit-switching (OCS) technology provides high-bandwidth, dedicated optical paths. The converged optical/electrical datacenter network (COE-DCN) has emerged as a promising solution for AI-DCN. However, optical circuit reconfiguration and multiplexing often introduce delays, disrupting traffic flow. Prior work in COE-DCN designs schedules the network, neglecting the negative influence during the optical path configuration. This paper addresses these challenges by analyzing traditional optical path provisioning methods and traffic-overlapping scenarios in COE-DCNs. We propose a bisection-assisted control mechanism to collaborate with optical and electrical networks, ensuring continuous data transmission. Our approach integrates queue-aware scheduling to dynamically allocate resources across EPS and OCS, minimizing transition latency and optimizing traffic flow. To validate the proposed scheme, we implement a field-programmable gate array (FPGA)-based hardware platform, which achieves sub-microsecond packet-level transition latency and demonstrates efficient queue management. Experimental results confirm significant improvements in job acceleration for overlapping DDL traffic scenarios, highlighting the effectiveness of our FPGA-based, queue-optimized design.