Arsc: Automated Reconciliation for State Correctness of Cluster Management

Modern cluster management platforms, such as Kubernetes, implement declarative interfaces based on state-centric principles and extend cluster capabilities through operators. Once the desired system state is declared, operators automatically reconcile the actual state to align with the desired state. Ensuring state correctness after operations is critical for system safety and stability, as incorrect or unhealthy states can lead to serious consequences. However, there is currently no systematic process in place to guarantee state correctness. The vast state space presents significant challenges in detecting and correcting abnormal states. To tackle this issue, we propose Arsc, an automated technique for test generation, detection, and correction aimed at maintaining state correctness. Arsc establishes the definition of state correctness, offers a state detection method, and presents three strategies for state correction. It was evaluated on five popular open-source operators, generating 11,581 tests and detecting 208 incorrect states, 71 unhealthy states, and 28 crashes, all of which were effectively corrected.