AsymGroup: Asymmetric Grouping and Communication Optimization for 2D Tensor Parallelism in LLM Inference

Recent advances in Large Language Models (LLMs), such as GPT and LLaMA, have demonstrated remarkable capabilities across a wide array of natural language processing tasks. Despite these successes, efficient inference at scale remains challenging, particularly in heterogeneous computing environments characterized by variations in GPU counts, and computational capacities across nodes. Conventional tensor parallelism approaches typically assume homogeneous hardware, resulting in significant performance degradation under asymmetric conditions. To address this challenge, we propose AsymGroup, which is an asymmetric 2D tensor parallelism framework that dynamically constructs groups of uneven sizes based on node- and GPU-level metrics. AsymGroup proportionally allocates both computational and communication workloads according to individual device capabilities. In addition, we introduce a formal communication cost model to accurately quantify bottlenecks and a group optimization algorithm to systematically determine efficient group configurations. Experimental evaluations demonstrate that AsymGroup reduces inference latency by up to 26.3% and communication overhead by up to 33.4% compared to state-of-the-art frameworks, while also maintaining competitive performance under symmetric conditions.