MOUSSE: A Multimodality-Oriented Unified Semantic Communication System by Contrastive Learning

The sixth generation (6G) communication posed higher requirements for the communication system regarding accurate semantic transmission. The existing studies about semantic communication principally concentrate on tackling task-oriented problems, rather than directly design modality-oriented system which is more generic and adaptive to different tasks. To improve the flexibility and robustness of communication system, we propose a Multimodality-Oriented Unified Semantic Communication SystEm (MOUSSE) based on contrastive learning. MOUSSE is designed to firstly orient modality then matches different modalities combination up to various tasks. Existing task-oriented philosophy primarily considers tasks whilst restricting modal versatility. MOUSSE could also intake and output various tasks with multiple modalities while transmit them in a concise and unified representation. Specifically, the system consists of structure-symmetric twin encoder-decoder for modality unification, which cascades joint source-channel coding (JSCC) module and contrastive learning based alignment module. Finally, the experiments verify the validity of proposed MOUSSE by quantitative results from different modalities with their respective tasks. The reliability and robustness are also improved from the point of entire communication system view.