RAPO: An Automated Performance Optimization Tool for Redis Clusters in Distributed Storage Metadata Management

Distributed storage systems are well-equipped to address the demands of massive data storage and rapid access in the era of big data. However, as the scale of these systems continues to expand, challenges related to data localization, performance optimization, and secure access have become increasingly prominent, making effective metadata management critically important. Redis, a memory-based NoSQL (Not only SQL) database, has emerged as an ideal choice for metadata management in distributed storage systems due to its high performance, flexible data structures, excellent scalability, and robust data persistence mechanisms. This study employs Redis as the caching layer for metadata services in distributed storage systems and develops the Redis-cluster Automatic Performance Optimizer (RAPO) to enhance performance. By implementing load balancing among main nodes and read-write separation between main and secondary nodes, the tool aims to improve the metadata service performance of Redis clusters during system maintenance and in high-concurrency scenarios. To address load balancing among main nodes, the research defines the Load Balance Index (LBI) as a metric for evaluating the degree of load distribution within the cluster. Greedy and random iterative search algorithms are designed to migrate hash slots from selected nodes, thereby balancing the node loads and enhancing overall cluster performance. By optimizing with the greedy and random iterative search algorithms, the cluster’s LBI was reduced by an average of 29.36% and 25.20% respectively during random read operations and by 24.98% and 24.03% respectively during imbalanced write operations. A load threshold is defined to achieve read-write separation between main and secondary nodes. Random allocation, round-robin, weighted round-robin, and smooth weighted round-robin are employed to optimize the read-write separation. Experimental results demonstrate that the weighted round-robin and smooth weighted round-robin strategies reduce read response times by an average of 27.53% and 30.75%, respectively.