Coding-based In-Network Aggregation for Distributed Deep Learning

Distributed Deep Learning (DDL) has become essential for accelerating model training to manage the growing size of models and datasets. However, the communication overhead required for synchronizing model weights across parallel workers would be a performance bottleneck of DDL training. Recently, in-network aggregation (INA) has been proposed to harness the computational capabilities of programmable switches to merge weights before forwarding, thereby reducing bandwidth consumption during synchronization. However, ensuring the reliability of INA incurs significant retransmission costs and coordination overhead. To fully optimize the benefits of INA, we introduce Coding-based INA (CINA), a system that integrates network coding with INA to facilitate efficient loss recovery. Moreover, we propose a priority-aware weight batching (grouping) scheme that leverages model sparsity to enhance bandwidth utilization in our coding-based INA. Emulation results demonstrate that our design effectively reduces feedback overhead by up to 94.87% and improves overall throughput by up to 2.64×. Compared to priority-oblivious batching, our priority-aware weight batching reduces the mean square error of model weights by around 15.27%.