Rank-Two Correction and Fine-Tuning for Adaptive Byzantine Recovery in Federated Learning

In this paper, we propose ABR-FL, an adaptive recovery and selective backup algorithm designed to manage models compromised in federated learning, while minimizing computational costs for clients and memory costs for the server. Our algorithm integrates multiple stages, including selective backup mechanisms, recovery point selection, initial recovery stage fine-tuning, approximation training using historical gradients, and post-approximation fine-tuning. Initially, the server evaluates client updates using a Taylor expansion, backing up only those updates that align well with the global model and excluding potentially malicious updates. To optimize memory usage and enhance computational efficiency, the server evaluates updates over the most recent k iterations, groups clients, and assesses updates from each cluster. Subsequently, the server performs a sensitivity analysis on historical backup models and stores models with sensitivity below a certain threshold in a starting point pool for potential recovery initiation. The recovery stage involves fine-tuning the model to comprehend the models learning trajectory and ensure that the approximation training process on the server aligns with this trajectory. The server employs second-order gradient information (Hessian approximations) and a rank-two correction matrix to calculate the recovered gradient from the historical gradient. Post-approximation fine-tuning addresses errors stemming from omitted higher-order terms by making small, precise adjustments to the model parameters. Theoretically, we establish that the global model recovered by ABR-FL closely approximates that of a model trained from scratch under certain assumptions. Finally, we build a federated learning system and conduct extensive experiments to demonstrate the effectiveness of our algorithm in terms of model accuracy.