AD-KFAC: Asynchronous Decentralized Distributed K-FAC with Dynamic Load Balancing and Fault Tolerance

Second-order optimization methods, such as Kronecker-Factored Approximate Curvature (K-FAC), offer superior convergence properties compared to first-order methods but have predominantly been explored within synchronous communication frameworks. However, synchronous methods are inherently limited in heterogeneous or unstable distributed environments due to their sensitivity to node failures, latency, and communication delays. In this paper, we propose an Asynchronous Decentralized Distributed K-FAC (AD-KFAC) framework, effectively extending second-order optimization into asynchronous decentralized settings. Our approach integrates asynchronous K-FAC computations, peer-to-peer Remote Procedure Call (RPC)-based communication, and dynamic load balancing mechanisms leveraging the Raft consensus algorithm, significantly enhancing robustness and scalability. Experimental results conducted on a 16-node cluster using ResNet-34 on CIFAR-10 demonstrate that AD-KFAC consistently outperforms both synchronous K-FAC baselines and existing first-order asynchronous decentralized methods, particularly under scenarios involving latency and node failures. These results highlight AD-KFAC’s potential as a robust and scalable solution for distributed deep learning tasks in realistic, unstable network conditions.