Tiny Federated Wireless Foundation Models for Resource-Constrained Devices

Deploying large-scale foundation models (FMs) in resource-constrained devices presents critical challenges due to their substantial computational and memory requirements. This is particularly relevant for multitask wireless sensing FMs running on sensors. To overcome these limitations, we propose a tiny federated wireless FM (WFM) framework that combines spectrogram-guided structured block-wise pruning with federated learning (FL) for efficient on-device deployment. Our approach prunes nonessential encoder blocks in vision transformers (ViTs) by leveraging the masked spectrogram modeling (MSM) pretraining loss as an importance indicator, ensuring only the most structurally significant components are retained. This enables federated adaptation with frozen backbones and lightweight, task-specific heads, minimizing both computational burden and communication overhead. The pruning strategy preserves the integrity of spectrogram reconstruction, while federated fine-tuning supports decentralized learning across clients with heterogeneous data distributions. Experimental results on human activity sensing and radio signal identification tasks confirm the efficacy of our approach. Specifically, the pruned ViT-based WFMs achieve up to 93% multiply-accumulate operations (MACs) reduction, 85% lower CPU inference time, and 49% reduction in communication overhead, all while maintaining high task accuracy. Our method demonstrates strong generalization and robustness across varying pruning ratios and data heterogeneity levels, while substantially reducing communication overhead, making it highly suitable for real-world industrial IoT deployments.