A Low-Memory Community Detection Algorithm With Hybrid Sparse Structure and Structural Information for Large-Scale Networks

Community detection plays an essential role in the domains of social, bioinformatics, and e-commerce. The innovative structural information theory (SInfo, introduced by Li et al.) has achieved excellent performance for network analysis. Nevertheless, similar to traditional network algorithms, the SInfo algorithm will exhaust all system memory resources when processing large-scale networks based on adjacency data structures. Moreover, due to the irregular and sequential characteristics, the SInfo algorithm is challenging to parallelize. In this article, we propose a hybrid sparse data structure and design a low-memory community detection algorithm based on structural information theory (HSSInfo), which shrinks memory requirements and achieves high parallelism. Specifically, we first propose a general quantization method to quantify the information change in community transformation, with which community inner and outer connection complexity can be semantically interpreted. Second, we design a general hybrid sparse structure to store network data among CPU and GPU, which can reduce memory resource consumption for community detection on large-scale networks. Finally, we develop an HSSInfo algorithm based on the quantization method and hybrid sparse structure, in which parallelism intersection and community fusion algorithms are employed to improve parallel scalability. We execute the comparison experiments for HSSInfo and eleven baseline algorithms on nine real-world datasets. Empirically, HSSInfo can achieve orders of magnitude speedups and extend structural information theory to large-scale datasets of billions of edges. Meanwhile, compared with various community detection algorithms, HSSInfo can achieve higher or similar accuracy with up to 19x memory shrinkage ratio.