Beyond Federated Learning for IoT: Efficient Split Learning With Caching and Model Customization

Distributed training of Deep Learning (DL) models on resource-constrained devices has gained significant interest. Training data-driven DL models collaboratively using federated learning (FL) and split learning (SL) have become the most popular ways to do this task. We aim to optimize these techniques by reducing device computation during parallel model training and reducing high-communication costs due to the frequent exchange of models, data, and gradients. This article proposes efficient SL (ESL), a novel approach addressing these challenges through three key ideas: 1) a key-value store for caching and sharing intermediate activations across clients, significantly reducing redundant computations and communication during the training phase; 2) customization of state-of-the-art neural networks for SL context; and 3) personalized training allowing clients to learn individual models tailored to their specific data distributions. Existing communication-efficient FL/SL methods trade accuracy to reduce communication. In contrast, ESL achieves significant communication reduction while maintaining high accuracy. Extensive experimentation on real-world federated benchmarks for image classification and 3-D segmentation demonstrates significant improvements over baseline FL techniques: ESL achieves a reduction in computation by $1623 \times $ for image classification and $23.9 \times $ for 3-D segmentation on resource-constrained devices. Additionally, it reduces communication traffic, during training, between clients and the server by $3.92 \times $ for image classification and $1.3 \times $ for 3-D segmentation while improving average accuracy by 35% and 31%, respectively. Furthermore, when compared to the baseline SL approaches, ESL reduces communication traffic during training by $100 \times $ and improves accuracy by an average of 34.8%.