Emerging Trends in Multi-Accelerator and Distributed System for ML: Devices, Architectures, Tools and Applications

As the complexity and diversity of machine/deep learning models is increasing at a rapid pace, multi-accelerator and distributed systems are becoming a critical component of the machine learning (ML) stack. Besides efficient compute engines and communication mechanisms, these systems also require intelligent strategies for mapping workloads to accelerators and memory management to achieve high performance and energy efficiency, while meeting the demands for high-performance ML/AI systems. This article presents an overview of the emerging trends in multi-accelerator and distributed systems designed for handling complex AI-powered application workloads.