Distributed Foundation Models for Multi-Modal Learning in 6G Wireless Networks

Benefiting from the ability to process and integrate data from various modalities, multi-modal foundation models (FMs) facilitate potential applications across a range of fields, including computer vision (CV), natural language processing (NLP), and diverse multi-modal applications such as imagetext retrieval. Currently, FMs are deployed on computing clusters for training and inference to meet their considerable computational demands. In the foreseeable future, the parameter size of FMs is expected to evolve further, posing challenges to both computation resources and energy supply. Fortunately, leveraging the next-generation wireless networks (6G) to aggregate substantial computation resources and multi-modal data from myriad wireless devices holds promise for handling the aforementioned challenges. In this work, we delve into state-of-the-art artificial intelligence (AI) techniques, specifically focusing on pipeline parallelism, data parallelism, and multi-modal learning, with the aim of supporting the sustainable development of distributed multi-modal FMs in the 6G era. In the context of pipeline parallelism, compressing activations and gradients while intelligently allocating communication resources can overcome communication bottlenecks caused by unstable wireless links. For data parallelism, federated learning (FL) with over-the-air computation (AirComp) seamlessly integrates communication and computation, significantly expediting gradient aggregation. Furthermore, by following the recent success of large language models (LLMs) and incorporating multi-modal learning into FMs, we can seamlessly integrate NLP and CV, along with the broader AI community, establishing the cornerstone for the intrinsic AI within 6G wireless networks.