ElasticFed: Collaborative Large-Small Transformer Training for Federated Continual Learning at Edge

Executing transformer-based applications on edge devices encounters challenging scenarios of continually learning new tasks. Federated continual learning (FCL) is a prevalent framework that supports model training using local data across edge devices. However, state-of-the-art FCL techniques either cause high computation and communication costs on large transformer models, or train small/compressed models, limiting both the learning capacity/accuracy on clients' local data and the global knowledge exchange among them. In this paper, we propose ElasticFed, a neuron-grained scaling approach for large-small transformer collaborative training in edge-based FCL. ElasticFed's key design features are (i) a proxy mechanism in local training, which constructs a compact model consisting of the original transformer's most important neurons to the current task, thus collaboratively training both models with small overheads; and (ii) a neuron-grained global aggregator, which se-lectively aggregates different clients' knowledge belonging to the most important neurons, thus maximizing the positive knowledge transfer with small communication costs. The comparison results against state-of-the-art techniques show that ElasticFed improves accuracy by 59.69% under the same training time. Compared to the techniques on original transformers, ElasticFed reduces training time and communication costs by 1.5x and 2.9x with small accuracy losses of 0.88%