On the Benefits of Cross-Modal Communications: from Source and Channel Coding Perspectives

Multi-modal services that integrate signals such as audio, video, and haptic are poised to dominate the 5G and beyond era. Due to the presence of inter-modal redundancy and interference, it is highly challenging to simultaneously meet the demands for real-time, reliability, and high capacity in multi-modal services. Cross-modal communication, taking full use of inter-modal correlations, has been viewed as a promising solution. This paper aims to provide theoretical support for this approach by addressing two key issues: i) the benefits of cross-modal source coding (CMSC) in eliminating inter-modal redundancy, and ii) the benefits of cross-modal channel coding (CMCC) in alleviating inter-modal interference. Specifically, we first construct a cross-modal communication system model to illustrate how inter-modal correlations can be leveraged by incorporating common semantic information in both source coding and channel coding. Next, we analyze the reduction in error probability for a target modality’s symbols at the same coding rate and coding delay, both with and without assistance from other modalities and common semantics, to quantify the benefits of CMSC. Finally, we analyze the differences in achievable channel degrees of freedom and sum capacities under the same channel conditions, both with and without the utilization of common semantics, to quantify the advantages of CMCC. In the three case studies, numerical results validate the theoretical soundness of CMSC and CMCC for binary sources over a Gaussian channel, as well as their feasibility on a practical audio, video and haptic teleoperation platform.